Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-12 14:19:00 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,267 Commits 1,902 Branches 2,445 Tags
07f6e8ff75b3de5f02b603cb0b7e36885cbabaef
Commit Graph
2127 Commits
Author SHA1 Message Date
DottaandPaperclip 07f6e8ff75 perf(workspaces): group same-root cohorts with bounded workers (#20437) 
## Thinking Path

> - Paperclip preserves agent work when its control-plane server is
replaced.
> - Online migration imports local workspaces into authenticated
snapshots.
> - Several workspace records can point to one physical directory.
> - Shared child verification avoids repeated work inside one held
cohort fence.
> - The eight-member limit splits 30 identical-root aliases into four
cohorts and repeats the child passes.
> - This pull request admits up to 32 members only for one canonical
physical root, while keeping each work stage bounded to eight peers.
> - The benefit is fewer repeated child checks while capture and
publication guards stay in place.

## Linked Issues or Issue Description

Refs: #19873, #19882, #20418, #20424, #20429. Related PR search found
shared verification and restart fixes, but no larger same-root cohort
owner. The shared verifier and archived-source changes are merged into
this branch from current master.

**Current behavior**

Thirty fresh aliases of one physical root require cohorts of 8, 8, 8 and
6. Each cohort performs an initial and final local child-validation
pass. The same children are checked eight times in total.

**Proposed behavior**

Admit one cohort with nine to 32 members only when every canonical
resolved physical root is exactly equal. Keep the existing complete
source-scope equality and admission guards. Keep mixed-leaf cohorts
limited to eight members. Bound source reads, reservations, recovery
checks, final readers, replay and cleanup to eight joined peers. Capture
concurrency stays at most eight.

**Reason and benefit**

For N distinct canonical covered children, the 30-alias case changes 8N
local child checks to 2N, a 75% reduction in repeated child validation.
If N is 68, that is 544 checks versus 136. Thirty parent captures,
uploads and restores remain required. No total wall-clock improvement is
claimed.

**Breaking changes**

Saved cohort fields and exact membership remain unchanged. Older tools
that enforce the eight-member bound refuse larger cohorts. Use the newer
tool to resume them; never split or regroup their saved membership.

## What Changed

- Raise the canonical service, saved-cohort and maintenance-request
member bounds to 32. Keep the capture-worker bound at eight and
single-workspace modes at one.
- Require one exact canonical physical root for more than eight members.
Keep the existing full source-scope comparison, leaf rule and P/I/L/F
guards.
- Use one private stage mapper with batches of at most eight. Join every
started peer before an error can leave the stage. Stop future batches
after a failure or cancellation.
- Preserve the existing cancellation and pending-snapshot ordering for
stages of at most eight members. Check cancellation around extended
batches.
- Keep all saved cohort, member, operation and snapshot identities,
proof refresh and atomic publication checks.
- Bound cleanup without skipping later scratch cleanup after an earlier
cleanup error.
- Extend the existing PostgreSQL fixture with ten real aliases,
partial-upload resume, cold restoration, replay and mixed-root refusal.
Extend the request boundary test.
- Document old-tool refusal, work counts and the retained-scratch
capacity bound.

## Verification

- Canonical Node24 worktree preflight and Plugin SDK prerequisites
passed. After merging current master, preflight passed again with
unchanged cached build inputs.
- Genuine RED on the old bound: both real ten-alias import/resume cases
and the 32-member request reject the eight-member limit (3 failed; 47
not selected; 18.23 seconds).
- First focused GREEN: 4 passed in 24.20 seconds. The fresh ten-alias
case performs exactly four local child captures for two children: one
initial and one final pass. Actual reader, transaction and upload maxima
are eight. All ten archives cold-restore after the original source is
parked; replay retains the same cohort.
- The partial-upload case keeps all ten saved operation IDs and the
existing snapshot IDs. More than eight members across different
canonical leaf roots refuse before any journal or storage work. The
request parser accepts 32 members and eight workers, and refuses 33
members or nine workers.
- The first expanded corpus had 55 passes and one real compatibility
failure: a new eager cancellation check skipped the old pending snapshot
boundary. The mapper now retains existing cancellation ordering for at
most eight peers. That failure remains recorded as a nonpass.
- Final sources include current master at `18697b8d21`, with the
verifier, archived-source and UI repairs. The selected four-file corpus
passes 47 cases in 78.88 seconds, with 191 intentionally not selected.
The full maintenance suite separately passes 23 cases in 10.21 seconds.
Nine cases overlap; the final union is 61 distinct passing checks.
- Final coverage includes all 27 cohort cases, all 23 maintenance cases,
all six actual old-format compatibility cases and five archived
prepared-source controls. Existing source mutation, finalizer
contention, actual fence loss, cancellation peer joins, journal
compare-and-swap and outer-commit rollback controls pass.
- Final server TypeScript, module boundaries and whitespace checks pass.
Only the resource-risk documentation changed after that test run;
runtime and fixture bytes stayed unchanged.
- Commands: `pnpm exec vitest run
server/src/services/remote-workspace-cohort-migration.test.ts
server/src/maintenance/online-workspace-migration.test.ts
server/src/__tests__/online-workspace-migration-compatibility.test.ts`;
archived controls use the existing `remote-workspace-migration.test.ts`
cases named `unselected archived recipe` and `archived prepared source`;
`pnpm --filter @paperclipai/server exec tsc --noEmit`; `node
scripts/check-module-boundaries.mjs`.
- Initial source review was 4/5 and requested a control for the extended
reader-batch stop boundary. The test-only follow-up
`ca80ae9cc8791a72dab632090488093c5d0916f3` adds actual ten-member
failure and cancellation cases. They hold a first-batch peer, verify
that the source fence stays held, and prove members nine and ten never
start preflight. Both pass in 10.87 seconds with no new journal,
snapshot or upload effects. Runtime source hashes are unchanged; the
local union is now 63 distinct checks.
- The first stop-test attempt observed the two controlled readers before
all SQL-backed peers had entered the inspection callback. Its assertion
was too early. The corrected test checks the held first-batch readers at
that point, then requires all eight to join before completion. Both
cases pass; the earlier attempt remains a nonpass.
- Final head `ca80ae9cc8791a72dab632090488093c5d0916f3`: [required
CI](https://github.com/paperclipai/paperclip/actions/runs/42737676664)
and [aggregate
verify](https://github.com/paperclipai/paperclip/actions/runs/42737676664/job/129575813765)
passed. Full rollup: 56 successes, four intentional skips and one
advisory neutral result. Classic Snyk passed.
[Greptile](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756419093)
and
[Ponytail](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756388693)
both reviewed that exact head at 5/5. The one review finding is
addressed and resolved; no open findings remain. Older partial CI and
the initial 4/5 review remain nonfinal evidence. Hosted acceptance and
measured wall-clock savings remain pending.

## Risks

- Completed parent captures retain their scratch until the atomic
publication completes. A 32-member cohort can retain up to four times
the parent scratch of an eight-member cohort. Check actual parent
capture, verification and restore sizes against current target
filesystem capacity before selecting a larger cohort. Keep the existing
resource guards.
- Larger member arrays must not raise execution fanout. Every
reader/reservation/final stage is bounded to eight and joins current
peers before failure. Capture workers remain at most eight;
child-verification workers remain two.
- More than eight members require the exact same canonical physical root
and complete source scope. A different root, new writer, changed source,
recovery owner or journal still refuses through the existing owners.
- Shared verification is limited to the held invocation fence. The
separate fresh final pass remains required after all captures join.
Different child sets and restart attempts do not share a local-byte
proof.
- The old saved membership and archive identities remain authoritative.
Older tools fail closed on the larger bound. No schema, archive or
provider format changes are included.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The runtime does not expose a more specific deployment identifier or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 20:03:25 -06:00
Nicky LeachandClaude Opus 5.5 232480d18f perf(runner): write preparation span rows to the run log in the background (#20426) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a native run starts its provider, the executor copies the
preparation spans (queue, prepare phases, environment) into the run
trace and the run log
> - It records about 25 spans and waits for one run-log insert per span,
one after another
> - On a hosted test deployment, that step
(`native.controller.span_persistence`, added in #20396) took a median of
0.28 s, and every task waited for it before the Runner started
> - The run trace already has a way to write diagnostics without
blocking the caller: `defer`, which `finish` drains before the root span
closes
> - This pull request writes those run-log rows through that path, in
order, while the spans themselves are still recorded at once
> - The benefit is about 0.25 s less before the provider starts, on
every native run, cold or warm

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The time between the end of heartbeat preparation and
`native.session.execute` for native Runner tasks.

**Subsystem affected**
Server: native runtime tracing
(`server/src/services/native-runtime/native-run-trace.ts`) and the
native session executor.

**Current behavior**
`executeNativeSession` calls `trace.record` for each preparation span
and awaits it. `trace.end` awaits `emitCompletedSpanEvent`, which
inserts one `run.performance.span` row through `onEvent`. On a hosted
test deployment, measured with the #20396 spans across 10 runs,
`native.controller.span_persistence` took a median of 0.28 s:
- 0.30 to 0.36 s for runs right after a computer wake
- 0.17 to 0.25 s for runs on an awake computer

It is the second largest step between preparation and the provider start
(the median of that whole gap is 1.0 s).

**Proposed behavior**
The spans are still started and ended at once, with the same parents.
The speed-budget bookkeeping still runs at once too. Only the run-log
row is written later: one row at a time, in the order the spans ended,
before the trace finishes.

**Reason and benefit**
The rows are diagnostics, and they already carry their own start time
(`wallStartedAtMs`, `startOffsetMs`). Readers do not depend on the row
order: the only reader in the repository, `summarizeStartupReuse`, looks
for `runner.process.launch`, which this change does not defer. The
provider can start without waiting for these inserts.

**Breaking changes**
None.

## What Changed

- `native-run-trace.ts`: `end` and `record` take an optional
`deferRunLog`. With it, `end` queues the run-log row on a per-trace
chain registered with `defer`, so rows are written one at a time in
order and `finish` waits for them. A row requested after the trace
finished is dropped, as for other deferred diagnostics.
- `native-session-executor.ts`: the preparation spans and the
`environment.startup` scope use `deferRunLog: true`.
- `doc/run-log-events.md`: note that native preparation span rows are
written late, in order.
- `native-run-trace.test.ts`: two tests.
- Spans are recorded under their parent while every row write is
blocked, and the rows are written in order before
`task.provider_session.measured`.
  - A deferred row requested after `finish` is not written.

## Verification

- `npx vitest run src/services/native-runtime/native-run-trace.test.ts
src/services/native-runtime/native-run-trace-context.test.ts` (34
passed). Both new tests fail without the source change: the first times
out waiting on the blocked write, and the second sees the row written.
- `npx vitest run src/__tests__/heartbeat-preparation-gap-spans.test.ts
src/services/native-runtime/native-session-executor.test.ts` (1321
passed)
- `pnpm --filter @paperclipai/server typecheck`

## Risks

- Low risk. The change is diagnostic only: OpenTelemetry spans and the
speed-budget checks are unchanged.
- The preparation rows can now land in the run log after the first
Runner events. Readers already use the payload times, not the row order.
- If the server process dies between the provider start and `finish`,
the preparation rows of that run can be missing from the run log. The
OpenTelemetry spans are not affected.
- The deferred rows are written while the run continues, as
budget-breach rows already are, so the sink sees concurrent writes from
one run. This was already the case for deferred diagnostics.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, Honeycomb queries) and
extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 17:50:15 -08:00
DottaandPaperclip 18697b8d21 perf(workspaces): share cohort child-source verification (#20418) 
## Thinking Path

> - Paperclip must preserve agent workspace bytes during online
migration.
> - A cohort can contain several aliases of the same physical source.
> - The importer verifies imported children before capture and again
before publication.
> - The immutable-object cache avoids repeated downloads but still
recaptures local child bytes for every alias.
> - This pull request shares the initial proof within the held cohort
fence and checks fresh bytes once more after every capture joins.
> - The benefit is less repeated local archive work while child
mutations still block the whole cohort.
> - Required browser CI twice reproduced an independent stale
composer-mode bug. A focused regression proves that refreshed persisted
task mode was ignored after mount. The state repair follows task data
when there is no local override. Explicit local mode choices retain
their own generation so a completed send cannot clear the next draft’s
choice.

## Linked Issues or Issue Description

Refs: #20311, #20387, #20401, #20404.

**Current behavior**

Two aliases recapture the same imported child's local bytes four times,
even with one immutable object read. A larger alias cohort repeats this
multi-GB work for each member.

**Proposed behavior**

Share the successful initial verification for the same physical root,
Git roots and exact recorded child set or discovery mode. After all
captures join, run a separate fresh final verification for each root and
recorded child set. Keep every member's database and publication checks.

**Reason and benefit**

The local PostgreSQL fixture proves that two aliases now perform two
child recaptures instead of four. Hosted timing improvement has not been
measured.

## What Changed

- Add invocation-local initial and final proof maps to the existing
cohort child verifier.
- Retain the existing canonical source-owner check before and after
shared results.
- Defer cohort final nested checks until all captures finish, and join
every final reader before publication or fence release.
- Keep singleton verification, saved journal schemas, snapshot IDs,
archive bytes and per-member publication checks unchanged.
- Extend the existing real PostgreSQL alias fixture with a child
mutation during the last parent upload.
- Repair the required CI planning-mode race with a nullable local
override: absent override follows task data; a user selection belongs to
the draft. A successful send or confirmed resend/receipt clears it only
if no newer choice exists. Failed saves preserve the mode and restored
message. Queued-edit backup retains whether the mode was explicit. Leave
the browser assertion and timeouts unchanged.

## Verification

- Canonical Node24 worktree preflight and plugin SDK prerequisites
passed. Server TypeScript passed.
- Genuine unchanged-source PostgreSQL RED: two aliases performed four
local child recaptures instead of two (one failed, 22 intentionally not
selected). GREEN: five source-sharing, last-upload child mutation,
capture/cancellation join and actual outer-commit rollback controls
passed (18 intentionally not selected; 66.53s). The actual fence-loss
control also passed (one passed, 22 intentionally not selected; 38.13s).
Both wrappers completed cleanup.
- Migration source and tree bytes remain identical to locally qualified
`466837cfe997924c3794fd5d0e96134cab1cd04c`; later commits only repair
the composer and its tests.
- Required planning browser CI originally failed, then failed its one
unchanged-head retry, because the planning chip remained absent after
opening-question dismissal. Those failures remain recorded
([original](https://github.com/paperclipai/paperclip/actions/runs/42723622165/job/129523785985),
[retry](https://github.com/paperclipai/paperclip/actions/runs/42723622165/job/129534415590)).
The final-head identical case passes in 30.5s, with all 34 shard cases
passing ([final
CI](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129550549775)).
Assertions and timeouts were not relaxed.
- Composer regressions reproduced stale loaded mode, optimistic-save
rollback, overwritten next-draft choices, and stale mode after confirmed
resend/receipt. All 145 composer tests now pass (2.88s). Canonical UI
TypeScript and token gates pass. The original planning browser case also
passed locally (15.5s; 26.9s including setup).
- The first final-head serialized-suite attempt was automatically
cancelled at its unchanged 20-minute job limit. Grouped Turbo output did
not expose an individual failing assertion; cause remains unknown
([retained
attempt](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129550550373)).
One same-head diagnostic retry was a cache miss and executed all 419
heartbeat recovery tests successfully in 376.94s on the identical CI
checkout
([retry](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129561196792)).
It is not classified as infrastructure or a known flake.
- Final exact head `ba1f85f18085dd0e84c594ed0cb68d5bf817c93c`: all 61
checks terminal (56 success, four skipped, one neutral); required CI is
green
([workflow](https://github.com/paperclipai/paperclip/actions/runs/42731309796)).
Fresh Greptile is 5/5 with every review thread resolved
([review](https://github.com/paperclipai/paperclip/pull/20418#issuecomment-6754705323)).

Focused commands:

```sh
node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts -t "shares the merged child verifier|joins active capture peers|keeps every authority and activity unpublished"
pnpm exec vitest run ui/src/components/task-chat/TaskChatComposer.test.tsx
pnpm --filter @paperclipai/server exec tsc --noEmit
pnpm --filter @paperclipai/ui typecheck
pnpm check:token-gates
```

## Risks

- Reuse is limited to one invocation under the same complete source and
finalizer fences. Different physical roots, Git roots, recorded child
sets and discovery mode have different keys. No local-byte proof is
stored durably or reused after restart.
- Initial and final maps are separate. A child change during any parent
upload must fail the fresh final verification before the atomic
publication transaction starts.
- Final readers all finish before an error can release the source fence.
Per-member database child rows, source fingerprints, journals, keys and
publication locks retain their existing checks.
- This reduces repeated verification for aliases. It does not change
capture concurrency, upload policy or immutable checkpoint validation.
Hosted speedup remains unverified.
- The CI repair must distinguish task data from explicit draft intent.
Only a successful send with the same choice generation clears its
override; a later draft selection and failed-save rollback retain it.
Regression tests cover these transitions. It adds no styling, request,
or persistence changes.

## Model Used

OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does
not expose a more specific model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 19:28:31 -06:00
DottaandPaperclip 3040448da5 fix(workspaces): refresh saved cohort source proofs on resume (#20429) 
## Thinking Path

> - Paperclip preserves agent work when a control-plane server is
replaced.
> - Online workspace imports save exact cohort, operation and snapshot
identities for restart.
> - A cohort proof also includes the complete current companion source
scope.
> - An independent companion import removes that companion from the
local scope.
> - The saved cohort then fails resume even when every saved member is
unchanged.
> - This pull request rechecks the current scope under fresh locks and
refreshes its proof for all members in one transaction.
> - The benefit is restartable online migration while independent work
progresses.

## Linked Issues or Issue Description

Refs: #19873, #20394, #20418. Related PR search found the original
cohort owner, recovery-proof refresh and separate child-verification
reuse. This fix addresses cohort scope freshness.

**What happened?**

An interrupted cohort rejects resume with
`workspace_cohort_membership_changed` after a nonmember companion is
migrated. The saved source proof includes companion workspace and task
IDs. The exact saved members and their fingerprints can remain
unchanged.

**Expected behavior**

The same saved cohort must resume after fresh admission. It must keep
all member, operation and snapshot IDs. Changed members, active owners,
changed journals and lost source locks must still refuse publication.

**Steps to reproduce**

1. Start a two-member cohort and interrupt the second upload after the
first archive is saved.
2. Import an archived nonmember companion through the existing canonical
migration owner.
3. Resume the same two-member cohort. The original implementation
refuses before it acquires the source fence.

**Paperclip version or commit**

Reproduced on master `3a5572c9e5`.

**Deployment mode**

Online administrative workspace migration with PostgreSQL and
authenticated snapshot storage.

## What Changed

- Admit a changed aggregate scope proof only when every saved member has
a running journal and its current fingerprint matches both saved
fingerprints.
- Keep partial reservations on the original strict aggregate-proof path.
Keep exact cohort and operation membership checks.
- Run the existing source, lifecycle, recovery and finalizer admission
before refresh. Lock member rows, inspect the current scope again, and
compare every journal in one transaction.
- Change only `cohortMigration.sourceProof`. Preserve all other
metadata, identities, archive and recovery records. Abort the complete
refresh on cancellation, a changed journal or a lost fence.
- Extend the existing real PostgreSQL fixture with partial upload,
canonical companion migration, refusal controls, transaction rollback,
cold archive restore and replay.
- Document the proof lifetime and resume boundary.

## Verification

- Canonical Node24 worktree preflight passed. Plugin SDK build
prerequisites passed.
- Genuine RED on unchanged product: the real partial-upload and
companion-import sequence fails with
`workspace_cohort_membership_changed` (1 failed; 22 intentionally not
selected; 10.44 seconds).
- The identical positive passes after the fix (1 passed; 22
intentionally not selected; 9.87 seconds).
- The complete focused cohort file passes: 23 tests in 33.33 seconds.
Existing controls include changed member fingerprints, foreign and
completed journals, active operations, peer cancellation, actual
source-fence connection loss and outer-commit failure.
- The new sequence also checks a held source lock, a held finalizer
lock, cancellation before refresh, and an injected second-member update
failure. The failed update rolls back every proof change. Both saved
archives restore tracked and untracked work, and replay uses the same
cohort.
- After one final read-only preflight assertion, that complete new
sequence passes again in 14.52 seconds with 22 existing cases
intentionally not selected. Server TypeScript passes on the final files.
Module boundaries and `git diff --check` pass.
- Commands: `pnpm exec vitest run
server/src/services/remote-workspace-cohort-migration.test.ts`; final
case selection: `-t 'same uploaded cohort'`; `pnpm --filter
@paperclipai/server exec tsc --noEmit`; `node
scripts/check-module-boundaries.mjs`.
- Initial local setup attempts remain nonpasses: sandbox PostgreSQL
could not bind localhost, then the new archived companion fixture used a
noncanonical repository folder. The corrected fixture uses the existing
managed repository path. Neither setup failure is counted as the product
RED.
- Fresh [Greptile
review](https://github.com/paperclipai/paperclip/pull/20429#issuecomment-6755862789)
is 5/5 on exact head `6c640c4667b136e5416c1375f491729cf4bb9b49`, with
zero inline threads. Required final-head CI is running. Hosted
acceptance is not claimed by this PR.

## Risks

- The old aggregate proof contains only a hash. Refresh therefore
requires durable fingerprints for every saved member and repeats all
current admission guards. A partially reserved cohort cannot use this
path.
- A stale proof must never hide a changed member or a new writer.
Current source inspection, P/I/L/F locks, locked member rows, complete
journal compare-and-swap and the existing capture and publication checks
remain authoritative.
- The all-member update must stay atomic. A database fault or
cancellation rolls it back. The original cohort, operation and snapshot
IDs remain fixed.
- The change adds one bounded transaction only when a fully saved
cohort's scope proof changes. It adds no recurring work, schema,
provider, archive format, concurrency or deployment changes.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The runtime does not expose a more specific deployment identifier or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 19:16:34 -06:00
DottaandClaude Opus 5.5 347d85fef3 perf(chat): index GitHub review lookups and share review-wait reads (#20400) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - GitHub chat endpoints let agents review pull requests. The server
reads review state when it lists issues and when it updates GitHub
checks.
> - On a hosted instance, two GitHub review queries used the most total
database time. One query ran about 660,000 times at about 33 ms. The
other query ran about 10,000 times at about 1 s.
> - The first query finds a source comment's chat link. No index covers
`chat_message_links.comment_id`, so each call reads the full table.
Issue lists also read the same review waits two times per request.
> - The second query joins deliveries, links and runs on a JSON key. The
planner walks the agent's run history in creation order and reads each
run's context.
> - This pull request adds four concurrent indexes, splits the run
lookup into two small reads, and shares review-wait reads in one
issue-list request.
> - The benefit is less background database load, so request reads wait
less. The results stay the same.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
GitHub review state reads. These are the inherited review-wait lookup by
source comment, and the review-check lookup of the run that a delivery's
comment woke.

**Subsystem affected**
Server chat channels (GitHub review waits and checks), issue list
attention maps, and the database schema.

**Current behavior**
- `readInheritedGitHubReviewWait` filters `chat_message_links` by
`company_id`, `comment_id` and `direction`. No index has `comment_id`,
so PostgreSQL reads the full table on each call. The run-origin check
does the same lookup first.
- An issue list calls the external review-wait read from the review
attention map and again from the blocked-inbox attention map. The two
maps read the same in-review issues.
- `githubReviewCheckState` joins deliveries, links and runs with
`context_snapshot->>'wakeCommentId' = comment_id::text`, sorts by
`created_at desc` and takes one row. The planner reads the agent's runs
in creation order and reads each run context. If no run matches yet, it
reads all of them. The delivery filter on `normalized_event` and the
link join on `delivery_id` also have no index.

**Proposed behavior**
The same rows come back from small index reads. Each issue's review wait
is read one time per issue-list request.

**Reason and benefit**
These two statement families used about 33,000 s of database time in one
cumulative statistics window. The database had about 50 concurrent
active queries, and request reads waited behind this background work.

**Breaking changes**
None. The migration adds indexes only.

## What Changed

- Migration `0456_chat_review_lookup_indexes` builds four indexes with
`CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It runs through the existing
top-level statement path, which repairs an invalid index on retry:
- `chat_message_links_company_comment_idx` on `(company_id, comment_id)`
where `comment_id is not null`.
- `chat_message_links_delivery_idx` on `(delivery_id)` where
`delivery_id is not null`.
- `chat_deliveries_github_automatic_delivery_idx` on `(endpoint_id,
normalized_event->'githubAutomatic'->'context'->>'deliveryId')` where
that value is not null.
- `heartbeat_runs_company_wake_comment_idx` on `(company_id,
context_snapshot->>'wakeCommentId')`. This index is not partial. The
planner does not use expression statistics from a partial index. Without
these statistics, it estimates many matches and keeps the creation-order
walk.
- `githubReviewCheckState` reads the delivery's comment IDs first. If
there is a comment, it reads the newest same-agent run with one of these
`wakeCommentId` values. If there is no comment, it skips the run read.
The comparison stays text equality on `comment_id::text`.
- `createExternalReviewWaitReader` shares one read per issue between the
review and blocked-inbox attention maps of one issue-list request. The
reader exists only for that request.
- `doc/DATABASE.md` describes the indexes and the new read order.
- `scripts/benchmark-review-lookup-indexes.ts` compares the pre-fix and
new code with and without the indexes.
`scripts/benchmark-review-lookup-index-migration.ts` times the migration
on a large synthetic database.

## Verification

Local SQL measurements on a synthetic fixture. They are not production
timings. Environment: embedded PostgreSQL 18.1, baseline revision
`02bfe71a6b`. Fixture: 50,000 links, 5,000 deliveries with about 2 KiB
events, and 4,000 runs with about 12 KiB incompressible contexts. The
wake run is in the middle of the run history. Each value is the median
server execution time from `EXPLAIN ANALYZE` (2 warmups, 9 samples). The
order was: no indexes, indexes, then no indexes again.

| Caller and statement | Before (no indexes, two passes) | After |
Shared buffers before → after |
| --- | --- | --- | --- |
| Inherited review wait, link by source comment (same SQL) | 2.72 / 2.36
ms | 0.076 ms | 929 → 12 |
| Review check, wake run found | 31.9 / 21.9 ms | 0.29 ms (two
statements) | 25,776 → 62 |
| Review check, no wake run yet | 71.0 / 47.5 ms | 0.06 ms (two
statements) | 34,861 → 12 |

- Indexes alone do not fix the review check. The old SQL with the new
indexes took 8.6 ms (run found) and 29.7 ms (no run), because it still
walked the run history. The new SQL without the new indexes was neutral
(23.5 and 35.1 ms).
- Service wall times on this shared machine were noisy, so the table
reports server execution time.
- Correctness: the benchmark fails unless the inherited statement
returns the same positional rows in each index state. The check service
result and the final wake-run rows of the new code must also match the
pre-fix module for both cases.
- Migration timing, normal migration runner, synthetic data: 200,000
runs (1.8 GB with 8 KiB incompressible contexts), 500,000 links and
100,000 deliveries. First run 4.2 s; rerun 0.38 s. Separate builds:
links by comment 1.5 s, run wake comment 1.1 s, deliveries 0.7 s, links
by delivery 0.3 s. All indexes were valid. The run index was 8.3 MB.
- New tests:
- `server/src/__tests__/github-review-check-state.test.ts` checks that
the newest same-agent run is selected. It also checks that other agents,
other companies, uppercase IDs, non-string values, list-only
`wakeCommentIds`, comment-less links and unknown deliveries do not
match. It checks that the generated statements can use the new indexes.
- `server/src/services/native-runtime/external-review-wait.test.ts`
checks that the reader reads each issue one time, keeps concurrent
callers correct, and rejects all waiting callers when a read fails.
- Red/green: with the pre-fix `review-check-state.ts`, the behavior test
passes and the index test fails. With this change, both pass.
- Focused runs pass: `issue-blocker-attention`,
`heartbeat-issue-liveness-escalation`, review-wait, review-check-state,
external-review-wait, review-repair (106 tests), and
`external-chat-wait.integration` (285 tests). `pnpm --filter
@paperclipai/server typecheck` and `pnpm --filter @paperclipai/db
check:migrations` pass.

Commands:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-review-lookup-indexes.ts --baseline-ref=02bfe71a6b
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-review-lookup-index-migration.ts
```

## Risks

- The migration builds four indexes concurrently. It does not block
writes. On a large database, the run index reads every run context one
time. The local run took 1.1 s for 1.8 GB from a warm cache. Cold
storage on smaller compute can be slower.
- The run index adds one entry for each run insert and for each run
update that is not a heap-only update. Many runs have no
`wakeCommentId`, so these entries hold a null key.
- The review check now sends two statements instead of one when a
delivery has a comment. Each statement reads only a few index entries.
- When two matching runs have the same `created_at`, either can be
selected. The previous query had the same tie behavior.
- This change does not change the number of in-review issues that one
issue list reads. The blocked-inbox map still reads all in-review issues
of the company. A smaller scope needs a separate change to the liveness
classifier.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with
tool use (shell, file edits, local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 18:47:34 -06:00
DottaandClaude Opus 5.5 0116e45958 perf(db): index pending budget Stops instead of scanning all run results (#20391) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Budget enforcement stops agent runs when a budget hard limit is
reached. The stop intent is saved in each run's `result_json` until run
control delivers it.
> - The budget worker, run control, and the idle-sleep check find these
pending stops with a predicate on `result_json`. No index supports that
predicate.
> - Each check therefore reads every heartbeat run and its large result
JSON from TOAST. On a hosted database, this statement family was among
the most expensive: about 8,000 calls at about 1 s each, about 9,800 s
in total.
> - Pending stops are rare. The predicate is constant, so a partial
index can contain only those rows.
> - This pull request adds that partial index. The SQL that the server
sends does not change.
> - The benefit is that each check reads a few index entries instead of
the full table. The database has more capacity for request work.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The check for committed budget Stops that are not delivered yet. The
budget worker runs it each time it looks for work
(`hasPendingBudgetEnforcement`), and again for each delivery pass
(`listPendingBudgetCancellationScopes`). Run control and the idle-sleep
check use the same predicate.

**Subsystem affected**
Budget enforcement in the server
(`server/src/services/heartbeat/budget-cancellation.ts`,
`server/src/services/budgets.ts`,
`server/src/services/heartbeat/run-control.ts`,
`server/src/services/idle-sleep-safety.ts`) and the `heartbeat_runs`
schema in `packages/db`.

**Current behavior**
The query groups all rows of `heartbeat_runs` and reads
`result_json->'startupCancellation'->'budgetEnforcement'` for each row.
A limit of 1 does not help, because the group and sort occur before the
limit. Cost grows with the run history, not with the number of pending
stops.

**Proposed behavior**
A partial index, `heartbeat_runs_pending_budget_cancellation_idx`,
contains only runs with a pending budget Stop. The planner uses it for
all four callers. The results are the same.

**Reason and benefit**
Local SQL measurements show a 300x to 1,000x decrease in time and buffer
reads (see Verification).

**Breaking changes**
None.

## What Changed

- Added the partial index
`heartbeat_runs_pending_budget_cancellation_idx` on `heartbeat_runs
(company_id)`. Its predicate is identical to
`pendingBudgetRunCancellationPredicate`.
- Added migration `0455_pending_budget_cancellation_index.sql`. It uses
`CREATE INDEX CONCURRENTLY IF NOT EXISTS`, and it is in the list of
migrations that run outside a transaction, as `0436` is.
- Moved the scoped run-control read and the idle-sleep check into
exported builders in `budget-cancellation.ts`, next to the scope list
query. The SQL does not change. The callers and the test now share the
same statements.
- Added a real PostgreSQL regression test. It checks that delivered,
malformed, and missing intents are not pending. It also EXPLAINs the
statements that the callers send (scope list with limit 1 and 100,
scoped read, `EXISTS` check) and requires the new index in each plan. If
a person changes a query shape, the server predicate, or the index
predicate, the test fails.
- Added `scripts/benchmark-budget-cancellation-scan.ts`. It captures the
SQL that the real service sends, then compares the plans, timings, and
complete rows with and without the index.
- Updated `doc/DATABASE.md`.

## Verification

Red/green regression test:

- Before the migration, the new test failed: the plan was `Seq Scan`
with the predicate as a filter.
- After the migration: `npx vitest run
src/services/heartbeat/budget-cancellation.test.ts` passes (17 tests).
- Also passed: `src/__tests__/budgets-service.test.ts`,
`src/__tests__/idle-sleep-diagnostics-postgres.test.ts`,
`src/services/budget-resume-continuation.test.ts`.
- `pnpm --filter @paperclipai/db check:migrations`, and typecheck of
`@paperclipai/db` and `@paperclipai/server` pass.

Performance and result equality (local SQL measurements only):

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-budget-cancellation-scan.ts --runs=20000 --payload-kib=16 [--reverse]
```

- Fixture: embedded PostgreSQL 18.1. 20,241 runs in 2 companies, with
incompressible result JSON of about 16 KiB (table 379 MB). 40 pending
stops in 6 scopes, 200 delivered or malformed decoys, and 1
cross-company intent.
- The SQL is the same in both treatments. The script toggles only the
new index. 2 warmups and 7 samples per query. Rows are compared outside
the timed part. Ordered queries must match in order. The scoped
run-control read has no `ORDER BY`, so it is compared as a multiset.

| Query (caller) | Baseline median | With index median | Shared buffers
|
| --- | --- | --- | --- |
| `listPendingBudgetCancellationScopes(db, 1)` (worker `hasPending`) |
688 ms | 2.1 ms | 243,500 → 1,158 |
| `listPendingBudgetCancellationScopes(db)` (delivery) | 799 ms | 1.4 ms
| 243,500 → 1,158 |
| Scoped pending runs (run control) | 357 ms | 1.1 ms | 121,835 → 322 |
| `EXISTS` pending (idle-sleep check) | 440 ms | 0.37 ms | 240,560 → 2 |

- Reverse order (index first) gave the same result: 524 ms → 1.6 ms, 464
ms → 1.4 ms, 306 ms → 1.0 ms, 306 ms → 0.42 ms.
- Plans change from `Seq Scan` (or a broad bitmap scan) to `Index Scan`
or `Index Only Scan` on the new index.
- All result rows are equal in every run.

Migration duration:

- 20,241 runs (379 MB): index build 2.0 s to 3.9 s.
- 200,241 runs with 8 KiB result JSON (2.1 GB): index build 5.6 s. With
this table size, the four queries change from 1,148 / 879 / 514 / 4,392
ms to 1.1 / 1.2 / 0.85 / 8.3 ms.
- The build is concurrent, so it does not block run writes.

## Risks

- Migration: the build reads every run's result JSON one time. It is
concurrent and resumable (`IF NOT EXISTS`). The executor repairs an
invalid concurrent index on retry.
- Write cost: run writes evaluate the index predicate. The index
contains very few rows, so the extra cost is small.
- Predicate drift: if a later change edits only one of the two
predicates, the planner cannot use the index. The new regression test
detects this.
- The hosted timings are not measured yet. The local numbers come from a
synthetic fixture.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 18:14:52 -06:00
DottaandClaude Opus 5.5 ab80fea62f perf(recovery): probe run bindings per comment in the lost-wake check (#20412) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The ownership reconciler finds user comments whose wake a restart
lost, and it saves a wake for them (plan invariant 4)
> - Its lost-wake check matched a task's runs with one OR across the
legacy context ID and the native issue column
> - With that OR, PostgreSQL can join runs only by company. It then
tests each recent comment against every run of its company and reads
each run's context
> - On a hosted staging database, this statement used about 23 s per
call on average (384 calls, about 8,900 s in total). One call ran for 35
s. Other requests waited behind it
> - This pull request reads the check in three stages and probes the two
run bindings separately with existing indexes
> - The benefit is the same result with bounded, index-driven work for
each recent comment

## Linked Issues or Issue Description

**What happened?**

The reconciler's user-comment lost-wake check was one of the most
expensive statements on a hosted staging database. Query statistics
showed 384 calls with a mean of about 23 s. One active call was seen at
35 s. The database compute was busy, and short authentication reads
waited behind long statements.

The statement matched runs with `(context_snapshot->>'issueId' = issue
OR native_issue_id = issue)` in three subqueries. The OR gives
PostgreSQL no join key except `company_id`. A local plan on synthetic
data showed the cost: a hash join of 273 comments against all 38,002
runs, which took 1.0 s of a 1.3 s statement. The work grows with recent
comments multiplied by the company's run count, and each test reads a
run context.

**Expected behavior**

Each recent comment uses a small number of indexed probes. The check
returns the same comments in the same order, with the same limit and
grace logic.

**Steps to reproduce**

1. Seed a company with a deep run history and a few hundred recent user
comments.
2. Let some recent comments have no wake (for example, after a restart
lost the wake of a steered comment).
3. Run the check. PostgreSQL can pick a hash join on `company_id` with
the OR as a join filter. With nested loops off, the old statement takes
seconds. The local benchmark below shows this.

**Paperclip version or commit**

Baseline: `02bfe71a6b`.

**Deployment mode**

Server with PostgreSQL. Local proof uses disposable PostgreSQL 18.1.
Production and staging use hosted PostgreSQL.

Related: the earlier change that split native and legacy bindings for
other ownership reads (see the "Execution recovery query performance"
section in `doc/DATABASE.md`). A search found no open pull request for
this check.

## What Changed

- `checkUserCommentsWithoutWake` reads in three materialized stages:
1. Recent user comments on assigned, non-conversation tasks that no wake
request carries. This stage keeps the same issue, status and
agent-closed rules.
2. Of those, comments with no live or later run and with at least one
earlier run.
3. Of those, comments that no live run acknowledged in an item event
(the steer check). Only this stage reads event payloads.
- Each run lookup is two probes: one on the legacy context ID and one on
the native issue column. Both use existing indexes. `exists (A or B)`
becomes `exists (A) or exists (B)`, which is logically the same.
- Each probe ends with `offset 0`. This keeps it a per-row index lookup.
Without it, PostgreSQL can build a hashed subplan over all runs, or a
hash join over all item events of the company. The repository already
uses this fence in `execution-blocker.ts`.
- The final `order by c.created_at limit N`, the projection, the grace
window and the reported reason do not change.
- Add `scripts/benchmark-comment-wake-check.ts`. It loads the check from
an explicit baseline commit and from the working tree, and it compares
the complete service result and the complete positional SQL rows before
it records time.
- Add a regression test for each run binding on its own: native only,
legacy only, live run, later run, steered run, an acknowledgement before
the comment, a log-only mention, a run that ended before the comment,
and a run in another company.
- Document the check and the benchmark command in `doc/DATABASE.md`.
- No schema change, index, migration, or new startup work.

## Verification

- `pnpm exec vitest run
server/src/__tests__/ownership-reconciliation-report.test.ts
--maxWorkers=1`: 34 passed. This includes the existing lost-wake,
restart, steered and agent-closed tests and the new binding test.
- `pnpm --filter @paperclipai/server typecheck`: passed.
- `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-comment-wake-check.ts --baseline-ref=02bfe71a6b
--samples=7`

Benchmark setup: PostgreSQL 18.1 (embedded, local), `shared_buffers` 128
MB, `work_mem` 4 MB. All data is synthetic, with md5-hex filler so
contexts and payloads do not compress well. There are two companies per
fixture. Two warmup rounds, then 7 timed samples for each arm, in
alternating order. Each sample runs the real check function. Before any
timing, each fixture asserts that the complete check result and every
positional SQL row are equal for baseline and treatment, and that the
fixture reports at least one owed comment. Every timed sample also
asserts the same result.

| Fixture | Runs / recent comments / owed | Baseline median | Treatment
median | Speedup | Hash-join plan, baseline → treatment |
| --- | --- | --- | --- | --- | --- |
| small | 2,541 / 50 / 6 | 8.5 ms | 2.6 ms | 3.2x | 19 ms → 3 ms |
| representative | 75,502 / 750 / 75 | 26.4 ms | 23.3 ms | 1.1x | 2,262
ms → 17 ms |
| lost steer wakes | 38,002 / 750 / 75 | 454.7 ms | 66.9 ms | 6.8x |
2,717 ms → 68 ms |
| deep history | 150,750 / 1,125 / 114 | 26.9 ms | 18.8 ms | 1.4x |
4,006 ms → 34 ms |
| busy hour | 128,750 / 5,625 / 200 (limit) | 79.2 ms | 47.4 ms | 1.7x |
20,047 ms → 75 ms |
| long steered runs (stress) | 19,100 / 600 / 60 | 626.4 ms | 457.4 ms |
1.4x | 5,871 ms → 502 ms |

- The table above used baseline `02bfe71a6b`, the branch base at that
time. After the rebase, a repeat run with baseline `8004ad069e` (the new
merge base, with the helper guard fixed to run from the repository root)
gave the same picture on a busier machine: small 15.4 → 9.1 ms, lost
steer wakes 1,733 → 139 ms, busy hour 103 → 65 ms, all with equal
results.
- The "lost steer wakes" fixture shows the slow plan with default
planner settings. PostgreSQL hashed all runs by company and tested 273
comments against 38,002 runs.
- The last column runs each statement once with `enable_nestloop = off`.
This makes the company-only hash join that the old statement allows
visible, and it shows that the new statement has no such plan. The
result is checked as equal in this mode too. Staging's 23 s mean matches
this growth: recent comments multiplied by the company's run count.
- Shared buffers fell in every fixture except "small" with default
settings (for example, busy hour 176,416 → 89,803).
- Neutral path: in the "long steered runs" stress fixture, every steered
comment lost its wake. These comments correctly reach the steer check.
That check must read the payload of each item event after the comment in
the run that was live, so it is the remaining cost. In normal operation
a comment sent to a live run has a deferred wake that carries it, so
stage one removes it.

## Risks

- Low risk. The predicate is the same conjunction of conditions. Only
the evaluation order and the probe shape change, and the benchmark
asserts complete-row equality on six fixtures and in both planner modes.
- Rows with the same `created_at` have no defined order, as before. The
benchmark uses distinct times so positional comparison is meaningful.
- `offset 0` stops PostgreSQL from turning these probes into joins. If a
task someday has a very large number of runs, each probe still uses the
company and issue indexes.
- Upgrade safety: this change reads saved run, wake and event state
only. It changes no persisted shape, run identity, or saved session, so
state saved by the previous release carries forward unchanged. A cutover
test is not applicable.
- Not verified on the hosted database. The local proof uses synthetic
data, and I did not capture a hosted plan. Confirm with a fresh query
statistics window after deployment.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file editing, local PostgreSQL) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 18:12:02 -06:00
DottaandPaperclip b2a7d322ed test: preserve runner commit stall acknowledgement invariants (#20383) 
Assert that stalled event commits do not advance acknowledgement or committed state. Clarify the limits of historical failure attribution. Preserve the production implementation already merged in #20385.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-11-09 18:04:00 -06:00
DottaandPaperclip b09f17fbfd fix(agents): repair queued setup through task connection cards (#20361) 
## Thinking Path

> - Paperclip manages AI agents doing work for people.
> - Agent verification must finish before an accepted task can start.
> - A missing AI account default can prevent verification before a
provider starts.
> - Lifecycle policy treated that owned configuration refusal as a
temporary failure and retried each minute.
> - The accepted turn remained Queued without explaining the
prerequisite.
> - Park account configuration failures and reuse the task's existing
inline connection card.
> - Successful repair rechecks verification and releases the original
queued turn under its original authority.

## Linked Issues or Issue Description

**What happened?** An OpenCode agent selected the responsible user's
OpenRouter default, but that default did not exist. Verification retried
unchanged configuration indefinitely and its accepted task stayed
Queued.

**Expected behavior:** Preserve accepted work, explain the setup
prerequisite, and let the appropriate person repair it using the
existing task connection UI. Completion must verify the agent before
starting the same queued turn. Genuine pause, approval and access gates
remain effective.

**Reproduction:** Create an OpenCode Runner agent with a
responsible-user OpenRouter binding and no personal default; assign a
task while verification is pending. The base retries verification and
shows an unexplained queue.

Related: #18702 handles pending task connection recovery after account
repair. This PR handles the earlier agent-verification boundary. #20330
addresses a separate Codex governed-wait timeout and is already merged.

## What Changed

- Share owned account-selection failure evidence through a pure library
contract. Lifecycle policy parks those configuration failures with
fixed, redacted guidance; temporary infrastructure errors retain their
retries.
- Batch setup-state reads for unstarted queued turns. Distinguish manual
repair from automatic setup retries and preserve Stop, started/history
state and lifecycle holds.
- Reuse the existing `connection_intent` card, account selection and
completion APIs. Its controller-only `agentSetup` marker accepts only
never-started legacy turns; runtime tokens still require a running run.
- Address the card to the lifecycle verifier's account owner, preserving
the task's execution principal and private-task visibility. Attach
missing cards on admission, verification, startup and full recovery,
including retained retries and continuations. Repeated delivery and
publication remain bounded. A new queued turn supersedes the setup card
of a stopped turn; generic connection cards retain cross-run reuse.
- Recheck lifecycle verification after usable-account completion, then
release the existing turn through the normal readiness path. Preserve
decline, Stop, reassignment and changed setup-owner gates. Fence an
in-flight verification failure from overwriting a repair; add no
provider wake or recurring worker.
- Park owned account refusals during resume as well as initial
verification. Bind setup cards to the lifecycle operation, settle
obsolete deliveries, and keep explicit Retry guidance accurate.
- Reuse the router’s existing lifecycle pin for setup repair without
rerouting the task, advancing the pool cursor, or replacing its saved
binding. Legacy direct-account cards remain readable.
- Hide the incorrect Resume banner during setup, retaining genuine
manual/budget pause controls. Document the execution and retry contract.

## Verification

- Red: eight original behavior assertions failed on the base, including
the real selector's retry decision and the real batched queue
projection. Six owned selection cases failed classification.
Temporary-error presentation, different task/verifier users, retained
retry/continuation recovery and the setup Resume banner also have
failing-before/passing-after regressions.
- Green: 430 affected backend, lifecycle, shared-contract and UI tests
passed before the final review corrections. Subsequent focused runs
cover one inline card, real account selection/completion, duplicate
delivery/publication, verification before readiness, unchanged run
authority, private tasks, no extra provider wake,
decline/Stop/reassignment, company isolation, invalid runtime tokens,
transient retries and redacted diagnostics.
- Final setup/card and generic connection-service suites: 49 tests
passed. The stopped-turn case failed before the correction, then passed
with real account completion, the stopped source remaining cancelled,
and the replacement source remaining queued.
- After rebase, all 129 focused assertions passed. The local queue
suite’s 88 execution assertions passed repeatedly, but its
database-fixture teardown exceeded both 30-second and 120-second hooks.
An isolated queue case completed, and the setup/connection suites fully
passed. The original queue test and cleanup remain unchanged; broad CI
supplies the full-suite result.
- Server/UI typechecks, module boundaries, token gates and diff checks
passed. `pnpm verify --explain` requested broad CI because the
execution-contract document changed; no redundant full local suite was
substituted.
- Real source acceptance: macOS arm64, Node 26.4.0, isolated embedded
PostgreSQL and local board identity. Created an OpenCode/OpenRouter
agent and task through the API with no default. Admission retained one
unstarted turn and produced the existing inline Connect OpenRouter card.
The browser opened its credential form and closed it without entering
credentials. The same card/run persisted after server restart and
reload, with the setup label and no incorrect Resume. The browser also
reproduced Stop followed by a new queued turn retaining the stale card
on the prior head. On `12f609c98372fd1e50644028db75cfcc8deb210d`,
startup superseded the stale card and retained the same replacement
turn, with the original turn still cancelled and no provider execution.
- The earlier CI case-search failure also reproduces on exact base
`0e3fdebea8257bdff11573081c76b306f12db230` without this PR. Its repair
is now in master via #20371; this branch is rebased onto it and contains
no API-search changes.
- Staging's actual Runtime test returned “Connect an account and choose
your personal default” before provider startup. The original turn later
started independently on deployed code, then hit a separate
workspace-export failure. Automatic file recovery subsequently marked
that run successful; its task remains Blocked and the trial’s useful
output is unverified. This PR is not deployed; provider authentication
through its card and downstream publication remain unverified. No
account, credential, default, permission or binding was changed during
diagnosis.
- Previous revision review: Greptile 5/5 on
`12f609c98372fd1e50644028db75cfcc8deb210d`, with all four inline
findings addressed and resolved. [Full CI
passed](https://github.com/paperclipai/paperclip/actions/runs/42713494508)
on that revision: 59 successful checks, one neutral check and three
intentional skips, with no pending or failed checks.

- Latest revision `976bcae993805a8a2222395bbff9394f6791b424`: resume,
router and Retry-copy regressions failed before repair. Green: 92
lifecycle/setup/shared tests, 83 setup/generic-connection/router tests,
and 137 UI/shared tests. Server/UI typechecks, module boundaries and
token gates passed. [Full CI
passed](https://github.com/paperclipai/paperclip/actions/runs/42720731137)
on this exact revision: 59 successful checks and three intentional
skips, with no pending or failed checks. Greptile reviewed this head at
5/5; all six review findings are addressed and resolved. Real Stop/retry
acceptance retained the same replacement turn and current-operation card
through two actual server restarts, with no provider execution.

## Risks

- Known account configuration failures now require repair or explicit
Retry. Temporary failures keep automatic retries; accepted work and
authority remain intact.
- The projection adds one company-scoped batch read only for unstarted
queued turns. Card attachment examines at most 100 active candidates and
adds no timer, worker, schema or migration.
- The optional stored setup-card marker requires an aware reader during
rollback. No run/session/checkpoint identity format changes. Setup
presentation grants no access and cannot bypass verification or other
gates.

## Model Used

- OpenAI GPT-6 (Codex), with reasoning, tool use, code editing and test
execution. The exact serving model identifier and context-window size
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 17:49:34 -06:00
DottaandClaude Opus 5.5 c985541a57 perf(db): index retry-exhaustion receipts for the attention scan (#20386) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The attention feed shows runs that failed after automatic retries
stopped
> - To find these runs, the feed reads every `heartbeat_run_events` row
of the company and filters the rows by a message prefix
> - Run output makes this table very large, so each feed request does a
large scan
> - On a hosted test database, this one statement used the most total
database time: 923 calls at a mean of 5 s (4,709 s in total)
> - Request authentication waits behind this background work, so all
users feel the load
> - This pull request adds a small partial index that holds only the
retry-exhaustion receipts
> - The benefit is that the same query reads a few index entries instead
of all company events

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The attention feed query that finds retry-exhausted runs
(`listAttentionExhaustedRuns`).

**Subsystem affected**
Server attention feed and the `heartbeat_run_events` table schema.

**Current behavior**
The query reads all company rows in `heartbeat_run_events`, filters them
by `event_type = 'lifecycle'` and the `Bounded retry exhausted` message
prefix, and then joins the few matches to failed or timed-out runs. The
cost grows with all run output. It does not grow with the number of
exhausted runs.

**Proposed behavior**
A partial index holds only the matching receipts. The query plan reads
the index and joins a few rows. The query text and the result do not
change.

**Reason and benefit**
The scan was the most expensive statement by total time on a loaded
hosted database. A local reproduction shows a 54x faster query and about
116x fewer buffer reads.

**Breaking changes**
None. The API, the feed rows, and their order do not change.

## What Changed

- Add the partial index `heartbeat_run_events_retry_exhausted_idx` on
`(company_id, run_id, id DESC)`. The index covers only rows where
`event_type = 'lifecycle'` and `message like 'Bounded retry
exhausted%'`.
- Add migration `0454`. It uses `CREATE INDEX CONCURRENTLY IF NOT
EXISTS`. The migration runner runs it as a top-level statement, the same
as migrations 0432, 0436 and 0450.
- Add a regression test. It pins the row order (newest run first, then
the newest receipt on a tie) and the latest receipt per run. It also
checks that the query plan can use the new index. If the query predicate
changes so that the index no longer applies, the test fails.
- Add `scripts/benchmark-exhausted-run-attention.ts`. It measures the
same query with and without the index, and the time to build the index.
- Document the index in `doc/DATABASE.md`.

## Verification

Cause and thesis. The scan cost comes from reading all company events,
not from the few exhausted runs. Thesis: an index that holds only the
receipts makes the query cost depend on the number of receipts, and
keeps the result identical. The benchmark below confirms the thesis.

Rejected alternative without a migration. I first tried a LATERAL lookup
that starts from the failed and timed-out runs. On the same fixture it
was about 2x slower (median 379 ms against 194 ms). It reads the full
event history of each failed run with random I/O, and that costs more
than one sequential scan.

Local SQL benchmark (the SQL that the service generates; same SQL with
and without the index):

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-exhausted-run-attention.ts --runs=30000 --events-per-run=100 --samples=9
```

Fixture (synthetic): one company with 30,000 runs (3,819 failed or timed
out) and 3.0 M events, plus a decoy company with 0.5 M events. The table
is 1.4 GB. About 5% of events are `lifecycle` events. Events of 20
concurrent runs are interleaved. 1 in 23 runs has a receipt, some runs
have two receipts, and output decoys contain the prefix. Two agents are
terminated. Embedded PostgreSQL 18.1 on an Apple laptop. The hosted
database is Neon with different compute, so the absolute times are local
SQL measurements only.

The script runs two cycles (without index, then with index; then with
index, then without index) to expose cache bias. Each measurement has 2
warmups and 9 samples.

| Query | Without index | With index | Samples and spread | Output check
|
| --- | --- | --- | --- | --- |
| `listAttentionExhaustedRuns` | median 157 ms (96–1,271 ms) | median
3.3 ms (2.2–5.7 ms) | 18 each | 133 rows, positional rows equal in all 4
measurements |
| Buffers (EXPLAIN ANALYZE) | 127,743 blocks | 5,580 blocks | | plan
uses the new index (asserted) |

Speedup: 48x (median), on head `ae6037b2a0` (the same SQL and index as
the final head). The machine was under heavy load from other work, so
the spread without the index is wide. An earlier run on the first head
gave 248 ms against 4.6 ms (54x). Index build (`CREATE INDEX
CONCURRENTLY` from the migration file): 1.7 s and 2.5 s for the 1.4 GB
table. Index size: 152 KiB.

Review fix: the query now writes `event_type = 'lifecycle'` as a SQL
literal, not a bound parameter. With prepared statements, PostgreSQL can
reuse a generic plan. A generic plan cannot prove that a bound parameter
matches the index predicate, so it would not use the index.

Tests:

- `cd server && npx vitest run src/__tests__/attention-service.test.ts`:
44 passed. The new test checks a fresh plan and a forced generic plan
(`plan_cache_mode = force_generic_plan` with `PREPARE`). It failed with
the bound `event_type` parameter and with a changed message prefix
(confirmed red), and it passes with the fix. Locally, the file's
`afterAll` database cleanup hook timed out on a heavily loaded machine
(load average about 28). The timeout also occurs when the new test is
excluded.
- `pnpm --filter @paperclipai/db check:migrations`: passed.
- `pnpm --filter @paperclipai/db typecheck` and `pnpm --filter
@paperclipai/server typecheck`: passed.
- `pnpm verify`: see the CI results for this head.

## Risks

- Migration time. A concurrent index build reads the full
`heartbeat_run_events` table and does not block writes. Locally, the
build took 1.2–2.5 s for 1.4 GB. The time grows linearly with the table
size. I did not time the build on a production-size clone. On a database
with a 5 s full company scan, I estimate the build at tens of seconds,
which is in the 1–2 minute budget. This estimate is not proven.
- If the build stops, PostgreSQL can leave an invalid index. The
migration runner repairs invalid concurrent indexes when it runs again,
and `IF NOT EXISTS` makes a rerun safe.
- The index predicate must stay textually equal to the query predicate.
If someone changes the message prefix or binds a predicate value as a
parameter, the planner cannot use the index. The new test catches both
cases, for fresh and generic plans.
- Each new receipt writes one more small index entry. Receipts are rare,
so the write cost is very small.
- The hosted improvement is not verified until deployment. After
deployment, compare a new query-statistics window for this statement.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL). Standard
context window and reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 17:37:20 -06:00
DottaandClaude Opus 5.5 c57cd1bdb9 perf(server): read plugin lifecycle inbox heads in one pass (#20405) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins get agent, project and agent runtime changes from the
`resource_lifecycle_events` journal. Each plugin acknowledges events one
resource at a time.
> - The inbox page query checked each unacknowledged event against all
older events of the same resource. Each check walked the acknowledged
history of that resource.
> - Agent runtime streams get a new event for each configuration change,
so one resource can have thousands of acknowledged events. The cost grew
as pending events times acknowledged history.
> - On a hosted instance, three copies of this read ran for 601 seconds
each, and one earlier copy ran for 19 minutes. The delivery worker runs
it again after each acknowledgment.
> - This pull request selects the head of each resource in one pass and
keeps the same result.
> - The benefit is that the page read is linear in the company journal
size. In local tests it is 250x to 660x faster, and the result rows are
equal.

## Linked Issues or Issue Description

**What happened?**
The plugin lifecycle inbox page query (`select "id", "company_id",
"resource_type", "resource_id", "action", "created_at" from
"resource_lifecycle_events" where ...`) ran for 601 seconds on a hosted
database. Three copies were active at the same time.

**Expected behavior**
A lifecycle page read must stay fast when a resource has a long
acknowledged history and a pending backlog.

**Steps to reproduce**
1. Give a company 40 agents with 1,500 `agent_runtime` events each.
2. Acknowledge the first 1,200 events of each agent for one plugin.
3. Call `pluginLifecycleInbox(db, pluginId).list(companyId, 100,
undefined, ["agent", "project", "agent_runtime"])`.
4. The read takes about 10 seconds for 60,000 events of that company. At
200,000 events it takes about 6 minutes.

**Paperclip version or commit**
`02bfe71a6b` (master).

**Deployment mode**
Hosted (PostgreSQL). Local repro uses embedded PostgreSQL 18.

## What Changed

- `server/src/services/plugin-lifecycle-inbox.ts`: `list` now selects
one head per resource with `DISTINCT ON (resource_type, resource_id)`.
The head is the unacknowledged creation, else the oldest unacknowledged
event. Then the query applies `afterId`, id order and the limit. This is
the same set that the old `firstForResource` predicate admits.
`acknowledge` keeps the old predicate for its single-event check.
- `plugin-lifecycle-inbox.ts`: new `hasPending(companyId, types)`. It
returns whether any unacknowledged event exists. Each resource with an
unacknowledged event has a head, so this gives the same answer as
`list(companyId, 1).length > 0`.
- `server/src/services/plugin-lifecycle-delivery.ts`: the idle-sleep
journal check uses `hasPending`. This keeps its early exit. A one-row
`list` page would read the full journal.
- `server/src/__tests__/resource-lifecycle-events.test.ts`: new database
test for heads across acknowledged history, a creation recorded after
transitions, acknowledged gaps, shared ids across resource types,
per-plugin progress, another company, `afterId` pages, limits,
`hasPending`, and the next head after an acknowledgment. The test also
passes on the old query, so it locks the contract.
- `scripts/benchmark-plugin-lifecycle-inbox.ts`: new benchmark. It loads
the baseline service from git, runs both on the same fixture, alternates
order, asserts equal rows, and prints plans.
- `doc/DATABASE.md`: describes the read cost and the benchmark command.
- No schema change, index or migration.

## Verification

Thesis: the old read cost is (unacknowledged candidates) x (acknowledged
older events per resource), because each candidate walks the
acknowledged prefix of its resource. One pass that keeps the
per-resource head removes that product and returns equal rows.

Baseline plan at 220,080 events: 40,020 candidates each walked about
3,999 older rows, which gave 160,039,980 acknowledgment probes and
562,140,833 buffer hits. The treatment plan reads the journal once with
a hash anti join and sorts the 40,020 unacknowledged rows (3,925 buffer
hits).

Local SQL timings (embedded PostgreSQL 18, Apple Silicon; event counts
include a 20,000-event second company, synthetic data, warm cache, one
warmup, order alternates):

| Fixture (events / acknowledged) | Case | Before median | After median
| Samples (after) | Speedup |
| --- | --- | --- | --- | --- | --- |
| 40,060 / 16,040 | page of 100, all types | 3,263 ms | 13.1 ms | 3
(12.0-14.1) | 250x |
| 80,080 / 48,060 | page of 100, all types | 10,137 ms | 23.9 ms | 5
(22.7-32.6) | 424x |
| 220,080 / 160,060 | page of 100, all types | 355,718 ms | 538 ms | 3
(72-785) | 662x |
| 1,020,080 / 960,060 | page of 100, all types | not run (est. > 30 min)
| 1,682 ms | 7 (1,203-2,384) | - |
| 80,080 / 48,060 | page of 100, agent and project only | 1.5 ms | 1.6
ms | 5 | neutral |
| 80,080 / 48,060 | idle-sleep check | 14.5 ms | 13.8 ms | 5 | neutral |
| 1,020,080 / 960,060 | idle-sleep check | not run | 406 ms | 7 | - |

- Result rows are equal (`assert.deepEqual` on all mapped rows, in
order) for each case where both versions ran.
- The agent runtime revision read in `plugin-agent-runtime-state.ts` and
`agent-file-store.ts` (`select id ... order by id desc limit 1`) uses a
backward index-only scan and reads 5 buffers at 1M events. It is not
changed.

Commands:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=02bfe71a6b --agents=40 --per-agent=1500 --acked=1200 --samples=5
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-plugin-lifecycle-inbox.ts --baseline-ref=02bfe71a6b --agents=40 --per-agent=25000 --acked=24000 --samples=7 --treatment-only
cd server && npx vitest run src/__tests__/resource-lifecycle-events.test.ts src/__tests__/plugin-lifecycle-delivery.test.ts src/__tests__/agent-runtime-hooks.test.ts
```

- `resource-lifecycle-events.test.ts`: 28 passed.
`plugin-lifecycle-delivery.test.ts`: 40 passed (includes idle-sleep
"none" and "present"). `agent-runtime-hooks.test.ts`: 22 passed.
- `tsc --noEmit -p server`: pass.

## Risks

- The page read is still linear in the company journal size, because
acknowledgments are per plugin and the journal has no retention yet. At
1M events a page takes about 1.7 s locally. Each acknowledgment wakes
another full pass. A per-plugin cursor or retention can remove this
later.
- The planner can choose a different plan on the hosted database. The
local plans use a hash anti join and one sort. The hosted improvement is
not yet measured.
- `acknowledge` still checks one event against older events of its
resource. That check is linear in that resource's history per call, as
before.
- Low behavior risk: the new test passes on both the old and new
queries.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local PostgreSQL). Extended
reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 17:37:14 -06:00
DottaandPaperclip 199670ba99 fix(accounting): prevent usage receipt stalls during streamed run updates (#20385) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner commits usage receipts before it acknowledges retained
events.
> - Usage receipts and streamed run logs update the same run row.
> - A repeatable-read snapshot can become stale before the receipt
writer locks that row.
> - Continuous log writes can exhaust all accounting retries and stall
the retained event.
> - This pull request uses the existing run lock with READ COMMITTED for
standalone receipt writes.
> - Other accounting operations keep their consistent snapshot. Stall
diagnostics retain safe nested error codes.

## Linked Issues or Issue Description

Related: #20235.

**What happened?**

A hosted native run stopped on a retained usage event. Its diagnostic
reported `Error at source seq N`. The saved outbox and accounting
records locate the failing delivery boundary. Traces show repeated
accounting transactions while streamed events still updated the run row.
The historical inner exception is unavailable.

A controlled PostgreSQL reproduction proves a matching defect: a run-log
append after company validation makes the receipt writer's
repeatable-read snapshot stale. Repeated appends exhaust all four
retries with SQLSTATE `40001`. This is a demonstrated failure mechanism;
attribution of the historical incident remains an inference.

**Expected behavior**

Commit the receipt under its existing run-row lock despite concurrent
run-log writes. Observe concurrent finalization and preserve
late-receipt debt. Keep duplicate receipts idempotent and reject
conflicting or cross-company receipts.

**Steps to reproduce**

Run `server/src/services/usage-receipts-read.test.ts`. The concurrency
regression uses real PostgreSQL and the production run-log append
function. It commits a log append after company validation and before
the receipt locks the run. Before the repair, the receipt rejects with a
wrapped `40001` after four attempts. After the repair, it commits once.
The finalization variant verifies that a newly settled run receives
late-receipt debt without replacing settled state.

## What Changed

- Allow aggregate-lock-protected accounting writers to select READ
COMMITTED. Keep repeatable read as the default.
- Use READ COMMITTED only for standalone usage receipt writes. Keep the
existing run lock, immutable receipt journal, replay checks and
transactional budget-work publication.
- Add PostgreSQL regressions for concurrent log appends, concurrent
finalization, duplicate delivery and the unchanged default snapshot.
- Read the outer event-commit error and at most three nested causes for
an existing safe error code. Exclude exception messages and SQL.
- Document receipt isolation and bounded stall diagnostics.

## Verification

- Red: the production receipt writer exhausted four retries with
PostgreSQL `40001` under controlled concurrent run-log appends.
- Red: wrapped database and transport stall diagnostics returned `Error`
instead of `40001` and `ECONNRESET`.
- Green: Runner control-plane tests passed all 207 cases. All 68 server
assertions passed across receipt reads, native usage delivery and cost
accounting. Cost-accounting teardown exceeded its 30-second limit twice
on this loaded host, after all 28 assertions passed. The workspace
project retained its configured 30-second hook limit despite the CLI
override. This is a local suite cleanup failure, not a passing suite;
the same revision passed the complete server suites in CI.
- Server layered TypeScript checking passed, including source and test
layers. Runner TypeScript checking passed. Server checks cover
`ccfe3bdeb1ab88537a03fd6af9044f66863939a5`; the unchanged Runner checks
cover its parent.
- Local environment: macOS arm64, Node 26.4.0, disposable embedded
PostgreSQL. Native usage-delivery tests use the real Rust receiver.
- CI at `ccfe3bdeb1ab88537a03fd6af9044f66863939a5`:
[build](https://github.com/paperclipai/paperclip/actions/runs/42714760702/job/129493568428),
[typecheck](https://github.com/paperclipai/paperclip/actions/runs/42714760702/job/129493568367),
native Runner integration, browser shards and all nine serialized server
groups passed. All 12 general server shards passed in the [full
run](https://github.com/paperclipai/paperclip/actions/runs/42714760702).
Final rollup: 59 successful checks/statuses, three intentionally skipped
checks, no pending or failed checks.
- [Greptile reviewed this revision at 5/5 with no actionable
findings](https://github.com/paperclipai/paperclip/pull/20385#issuecomment-6753163397).
- Hosted evidence came from the task, run, saved sandbox outbox,
accounting receipts and traces. The run started fresh and did not cross
a deployment. No original agent turn was replayed.
- The hosted repair has not been deployed. The original task has not
been resumed. A fixed-revision remote-provider user journey remains
unverified.

## Risks

- READ COMMITTED changes snapshot timing for standalone receipt writes.
The run-row lock protects all mutable receipt/accounting reads and
serializes competing receipts and finalization. Other accounting
operations retain repeatable read.
- Receipt identity, replay conflict checks, company scoping, budget
publication, retry bounds and acknowledgement rules remain intact.
- No persisted format, provider identity or database schema changes.
Existing saved state needs no migration.
- Diagnostics inspect at most four objects and retain only the existing
identifier format, with an 80-character code limit.
- The reproduced race matches the incident boundary and timing, but the
original inner exception was not recorded. Rollout and live acceptance
remain necessary before incident resolution.

## Model Used

- OpenAI Codex, GPT-6 family, with code and shell tools. The exact model
ID and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 17:35:13 -06:00
DottaandPaperclip bf818280f2 perf(workspaces): reserve cohort members concurrently (#20401) 
## Thinking Path

> - Paperclip controls agent work and preserves its workspaces.
> - The online importer moves up to eight shared-source leaves under one
source fence.
> - Each member must have a durable reservation before capture starts.
> - Serial reservations repeat the same validation cost before any
archive work.
> - This pull request reserves members in parallel through their
existing independent transactions.
> - Every reservation finishes before an error can release the fence or
start capture.

## Linked Issues or Issue Description

Refs: #19873, #20387, #20394.

**Current behavior**

A measured six-member cohort spent about 110 seconds in serial
reservations. Each reservation keeps its existing source, task, recovery
and journal checks.

**Proposed behavior**

Start the bounded member reservations together. Join all outcomes,
retain member order, and refuse capture if any member fails. Keep the
exact partial-journal resume contract.

**Reason and benefit**

This removes serial metadata work before capture. Hosted speedup has not
been measured for this change.

## What Changed

- Replace only the cohort reservation caller loop with
`Promise.allSettled`.
- Preserve the existing per-member transaction, preassigned operation
ID, source fingerprint and all source fences.
- Extend the real PostgreSQL failure fixture to require overlapping
reservations and complete peer closure before fence release.
- Document the unchanged partial-journal and atomic publication
contract.

## Verification

- Canonical cached worktree preflight and plugin SDK prerequisites
passed on Node 24.22.2.
- The unchanged production baseline failed the new reservation overlap
assertion: one reader entered instead of two. The canonical PostgreSQL
wrapper cleaned up. The other 21 tests were intentionally not selected.
- The first baseline command failed before PostgreSQL because root-level
`tsx` resolution was absent. The corrected command uses the checkout's
explicit `cli/node_modules/tsx/dist/loader.mjs`.
- The four focused PostgreSQL controls passed before the final rebase:
overlap/join failure, exact partial-journal resume, parallel capture
with both cold restores, and atomic outer-commit rollback. Eighteen
tests were intentionally not selected. Server TypeScript passed.
- The final branch is based on merged #20394 at
`302ada90dfda0e8fbcd8a514d506d310c9843bcb`. The two directly affected
PostgreSQL reservation/resume controls passed again on final head
`06d436e45705da3b616a2b91725960f6e0ea52c0` (2 passed, 20 intentionally
not selected; 100.56 seconds). Server TypeScript passed again. The
canonical wrapper completed cleanup.

- The separate Ponytail review remains 4/5. Its nonblocking fan-out
warning was resolved as the documented phase contract; no source change
was required. Optional multi-error aggregation is outside this change.
- Fresh Greptile review is 5/5 on
`06d436e45705da3b616a2b91725960f6e0ea52c0`, with no actionable findings
or unresolved threads:
https://github.com/paperclipai/paperclip/pull/20401#issuecomment-6754025480.

- Exact-head CI is complete: 56 success, 4 skipped and 1 neutral, with
no failed or pending checks. Aggregate `ci / verify` passed:
https://github.com/paperclipai/paperclip/actions/runs/42718439355.
- Reproduce the final focused cases on Node 24: `node --import
./cli/node_modules/tsx/dist/loader.mjs
packages/db/src/test-postgres-command.ts pnpm exec vitest run
server/src/services/remote-workspace-cohort-migration.test.ts -t "joins
concurrent reservation validation readers|resumes the exact membership
after reservation stops halfway"`. Run `pnpm --filter
@paperclipai/server exec tsc --noEmit` for TypeScript.

## Risks

- API phase contract: `concurrency` limits capture workers. Preflight,
reservation and post-finalizer validation fan out over the exact
selected member set, with the canonical maximum of eight. The original
DB pool bounds connections. `concurrency: 1` does not promise one
metadata query or transaction at a time.
- Up to eight reservations can use independent pool connections at once.
Members that pass can leave durable journals when a peer fails. The
existing exact-member resume path retains those journals; this change
does not retry automatically.
- Capture and publication remain blocked until every reservation
settles. Publication stays serial in one transaction. Saved journal
schemas, singleton behavior and recovery proof checks do not change.
- This does not change concurrency inside the publication transaction.
One PostgreSQL connection serializes that SQL, and changing its row-lock
order has no measured benefit here.

## Model Used

OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does
not expose a more specific model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 17:28:23 -06:00
Nicky LeachandClaude Opus 5.5 d95fdd8d79 feat(observability): span the unnamed gaps in run preparation (#20396) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a task reaches the agent, the server prepares the run: it
reads the task, the workspace and the configuration, and then hands the
run to the native controller
> - With an OTLP endpoint, each preparation step emits a
`heartbeat.prepare.<phase>` or `native.controller.<phase>` span
> - On a hosted test deployment, 0.3 to 0.5 s of each `task.prepare`
sits between those spans, and about 0.5 s more sits between
`execution.dispatch_admission` and `native.coordinator.claim`
> - Nobody can say which step is slow in that time, so latency work in
this area is guesswork
> - This pull request adds spans around the existing awaited work in
those gaps, with the existing timing helpers and naming conventions
> - The benefit is that a trace of a slow preparation now names the step
that used the time

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The OpenTelemetry trace of task preparation. Today it has unnamed gaps
between the named preparation spans.

**Subsystem affected**
Server: heartbeat run preparation and the native controller
(`server/src/services/heartbeat.ts`,
`server/src/services/native-runtime/native-session-executor.ts`).

**Current behavior**
On a hosted test deployment (8 runs per group), `task.prepare` takes 2.6
to 3.4 s. Between `heartbeat.prepare.runtime_state` and
`heartbeat.prepare.run_identity`, about 0.25 s has no span. Between
`execution.dispatch_admission` and `native.coordinator.claim`, about 0.5
s has no span, and in one pair of runs it was 1.4 s. Nobody can tell
which step used that time.

**Proposed behavior**
Spans name the existing awaited reads and writes in those gaps. The
change adds no reads and records no content. The operator endpoint gate
and the no-operation behaviour stay.

**Reason and benefit**
Latency work in this area needs a measurement. With these spans, a trace
of a slow preparation names the step that used the time, so a fix can be
confirmed or rejected after deployment.

**Breaking changes**
None. The change adds span names to the closed list. It changes no API,
schema or behaviour.

**Additional context**
The server uses postgres.js, which the auto-instrumentation does not
cover, so database spans do not fill these gaps.

## What Changed

- `server/src/services/execution-preparation-timing.ts`: add the new
closed phase names.
- `server/src/services/heartbeat.ts`: add `heartbeat.prepare.*` spans
for `chat_retry_admission`, `workspace_selection`,
`dependency_readiness`, `issue_checkout`, `instance_settings`,
`workspace_context`, `project_context`, `plan_wake_routing`,
`routine_environment`, `responsible_user` and `execution_start`. The
re-reads of the issue after a checkout use the existing `issue_context`
name.
- `server/src/services/native-runtime/native-session-executor.ts`: add
`native.controller.*` spans for `attachment_staging`, `warm_restore`,
`span_persistence` (persistence of the preparation spans to the run log)
and `state_migration` before the coordinator claim.
- `doc/observability.md`: list the new spans.
- `server/src/__tests__/heartbeat-preparation-gap-spans.test.ts`: run a
real heartbeat through native preparation with an in-memory span
exporter. Check that each new span is emitted and that the new
preparation spans are children of `task.execution_attempt`.

## Verification

- `pnpm --filter @paperclipai/server typecheck`
- `npx vitest run src/__tests__/heartbeat-preparation-gap-spans.test.ts`
(fails without the source change, passes with it)
- `npx vitest run
src/services/heartbeat/preparation-timing-context.test.ts
src/services/execution-preparation-timing.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/__tests__/heartbeat-context-compaction-execution.test.ts`

## Risks

- Low risk. The change only wraps existing awaited calls. Values and
errors pass through the timing helper unchanged, and a checkout conflict
still reaches the same `catch`.
- Phases of 50 ms or more inside `heartbeat.prepare.*` are also written
to the run log, as for the existing phases. This adds a few run-log rows
on slow runs.
- Without an OTLP endpoint, the helpers keep their current no-operation
behaviour.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 15:27:58 -08:00
Nicky LeachandPaperclip 1838de8c6c Pause agents automatically when moving to a plugin environment (#20375) 
## Thinking Path

> - Paperclip manages AI agents and their execution environments.
> - Core owns agent lifecycle state and controls when work can run.
> - A plugin environment can need a computer that belongs to the agent.
> - Moving a ready agent to that environment currently requires manual
pause and resume actions.
> - Saving the environment is sufficient intent to perform those routine
steps.
> - This change records the pause with the environment update and uses
the existing lifecycle worker.
> - Work can resume only after execution stops, the plugin starts its
resources, and verification passes.

## Linked Issues or Issue Description

Related: #19173 introduced environment-scoped plugin selection and the
manual-pause requirement. I searched for related pull requests and found
no duplicate fix. No separate issue is linked.

**What happened?**

Saving a ready agent with an environment whose lifecycle plugin is not
selected returns a conflict. The error tells the user to pause the
agent, change its environment, and resume it.

**Expected behavior**

Saving the environment change starts the required pause, resource setup,
and verification automatically. Existing manual, budget, and company
holds still block resume.

**Steps to reproduce**

1. Use an agent in `ready` on an environment without an agent-owned
plugin.
2. Select an agent-owned plugin environment in the agent configuration.
3. Save without first pausing the agent.

Before this fix, the save fails. With this fix, the agent moves through
`pausing`, `resuming`, and `verifying` before `ready`.

## What Changed

- Save the new environment and automatic pause in the same transaction.
Keep the existing plugin selection during stop.
- Use the existing completion of a pause without holds to resume
automatically. Set the saved resume target to `verifying` so the new
environment must pass its test.
- Preserve independent pause holds, lifecycle version checks,
notifications, resource status events, and worker recovery.
- Add migration tests for idle and running agents, real process-stop
proof across a worker restart, added holds, failed verification, and
route authorization.
- Update the lifecycle and development docs. Keep the feature maps
unchanged, as required for a small behavior fix.

## Verification

- Current revision: `300f57539bce9b198406c3793ab3c2de46492659`.
- Lifecycle and case API discovery suites passed after updating the
branch to master. The final environment-plugin suite passed all 22
tests, including four real-route authorization cases for users and
agents. Coverage includes real process-stop proof across worker
replacement, independent holds, and failed migration verification.
- Server typecheck: passed on the current base. All workspace typechecks
and `pnpm build` passed before the base update; fresh CI covers the
updated base.
- `node scripts/check-module-boundaries.mjs`, `node
scripts/check-maps.mjs`, and `git diff --check`: passed.
- Reviewed the diff for secrets, personal data, private addresses, and
run artifacts before each push.
- [Current
CI](https://github.com/paperclipai/paperclip/actions/runs/42714733533):
passed on `300f57539bce9b198406c3793ab3c2de46492659` (52 successful
jobs; the optional test-selection optimization was skipped). Standard
Greptile: 5/5 on the current revision, with no unresolved findings. The
security scanner accepted the documented configuration-authority policy.
- Earlier CI failed the `case-history-api-discovery` assertion for the
query `case for task`. It also failed locally. The branch now includes
master's fix #20382, and that test passes. The migration implementation
did not change. The superseded CI attempt was cancelled after recording
the failure.
- Full local `pnpm test:run`: incomplete; stopped after two passing chat
partitions. The focused suites passed, and required CI supplies broad
coverage, as specified in the repository instructions.
- Local target: isolated source checkout on macOS ARM64, Node 26,
disposable PostgreSQL, fixture lifecycle plugin. Initial database
startup failed because the bundled macOS PostgreSQL package lacked dylib
aliases. Temporary dependency aliases repaired startup; the focused
database tests then passed. The installed Rust toolchain also needed to
be added to PATH.
- The pending-stop test retains a live process ID in the cancelled run.
The real host stop check blocks plugin setup. After stop acknowledgment
and worker replacement, the saved migration reaches readiness.

## Risks

- Saving this environment change can stop current execution. The system
performs that pause as part of the requested migration.
- A failed stop, plugin call, or verification keeps the agent blocked
and uses existing retry behavior.
- Editing a shared environment or the instance default still uses its
existing conflict checks. This change covers the individual agent
environment update.
- Configuration permission authorizes the temporary system pause needed
for the resource move. Separate manual-pause authority is not required
for this configuration operation. Real route tests cover authorized and
denied users and agents.
- No schema migration or deployment is included. A fixture plugin does
not prove live computer provisioning or model execution.

## Model Used

OpenAI Codex, based on GPT-6, with code execution and tool use. The
exact runtime model ID and context limit are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 15:02:54 -08:00
Nicky LeachandPaperclip 2550f399d9 fix(lifecycle): align stop proof for pre-dispatch resource waits (#20384) 
## Thinking Path

> - Paperclip manages AI agents and controls when their work can run.
> - An agent pause waits until every earlier run has stopped.
> - Recovery already treats verified workspace and AI-capacity waits as
pre-dispatch outcomes.
> - The stop proof did not accept those same receipts. Historical waits
could keep an agent pausing forever.
> - This pull request shares the resource-wait classification and the
existing execution-event check.
> - The benefit is consistent stop evidence without a new lifecycle
command, database backfill, or host-boot assumption.

## Linked Issues or Issue Description

Refs #18147 and #17718.

This fixes the verified resource-wait case of #18147. It does not
certify ambiguous historical runs that have no positive non-execution
evidence.

An agent can have an old cancelled legacy run with `errorCode:
workspace_busy`, `executionStage: preparing`, and `executionRecovery: {
kind: workspace_wait, providerWorkStarted: false }`. No process, native
coordinator, or retained resource exists. Recovery recognizes this as a
resource wait, but pause previously required a stop acknowledgment from
an executor that never started.

### Credit and related work

Thanks to @br1an-c0des (#17692) and @crownlayerapp (#15383) for the
focused preparation and shared-admission analysis, @lucasweismann-ai
(#18072) for the durable dispatch-evidence and race analysis, and
@Pdesengrini (#18905) for the cancellation-receipt and process-identity
guard cases. We reviewed these alternatives before implementing this
smaller change. No contributor commits were cherry-picked.

The design uses receipts already written by core. It does not adopt a
reboot-time bypass, a new adapter-entry journal, or a pause-abort route.
#17236 and #20037 address the separate cost of history reads.

## What Changed

- Add one shared classifier for matching workspace and AI-capacity wait
receipts. Use it in legacy recovery and cancellation stop proof.
- Reuse the existing dispatch-event predicate instead of maintaining
another event list.
- Retain checks for native execution references, unresolved workspace
obligations, controller leases, and resource cleanup. A live controller
or cleanup still in flight cannot pass the proof.
- Add a focused PostgreSQL suite. It drives the real pause worker with
historical wait records and checks Retry eligibility against the same
evidence.
- Document the contract. No schema, API, UI, dependency, or scheduling
change is required.

## Verification

Source checkout on macOS arm64, Node 26.12.0, isolated local PostgreSQL
18, synthetic data only.

```sh
node node_modules/vitest/vitest.mjs run --root server \
  src/modules/agent-lifecycle/__tests__/resource-wait-stop.test.ts \
  src/services/legacy-execution-recovery.test.ts
```

- 61 tests passed across these two files.
- All eight positive regressions failed on unchanged master. The fix
restores them.
- After rebasing onto current master, the 61 focused tests passed again.
Another 53 selected tests passed in
`explicit-native-continuation.test.ts` with `-t 'pre-dispatch review
wait|unclaimed workspace retry|historical native preparation'`. The
other 361 cases in that file were not run.
- Guard cases cover wrong/missing receipts, dispatch events, process
identity, native ownership, live controller leases, workspace recovery,
cleanup in flight, unreceipted remote resources, and an unknown
historical HTTP dispatch.
- A remote lease passes only after its exact termination receipt is
stored.
- Secret/PII diff scan and `git diff --check` passed.
- Standard Greptile reviewed commit
`201a61d2bb63290ca9b9b9d524e6c6715a872310`: [5/5 with no actionable
findings](https://github.com/paperclipai/paperclip/pull/20384#issuecomment-6753165178).
No review threads are unresolved.
- Local testing is limited to affected cases. GitHub CI owns broad
typecheck, build, and test coverage.

Verification setup failures remain accounted for: worktree preflight
could not complete because Cargo is unavailable; the first bundled
PostgreSQL run failed because its macOS binary lacks a shared library.
The focused tests passed on a separate, temporary local PostgreSQL 18
cluster. An early local server typecheck could not resolve the unbuilt
Runner package; it is not claimed as a pass.

## Risks

- Stop proof now accepts existing positive resource-wait receipts.
Missing PIDs, terminal status, elapsed time, and a different boot ID
remain insufficient.
- The receipt must match its error code. The runtime must remain
unresolved, with a preparation or settled stage and no conflicting
native or workspace references.
- The shared cancellation proof is also used by explicit continuation
and Retry. Those paths keep their existing user-intent, ownership,
pause, budget, and authorization gates. A stop proof does not itself
start another run.
- Runs that lack usable evidence remain held. This PR does not claim to
repair every run shape reported on #18147.
- No migration or live data edit is needed. No hosted deployment is
included.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code execution,
PostgreSQL regression tests, and GitHub tools. The exact serving
revision, context window, and internal thinking configuration are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 14:40:14 -08:00
DottaandPaperclip 9c634e42e4 fix(workspace): bound queued source-lock admission for imports (#20370) 
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Existing local workspaces need durable imports before a volume-free
deployment.
> - Imports must acquire the same source locks as workspace writers.
> - Try-only admission can repeatedly lose to background maintenance,
even when each writer finishes.
> - This pull request adds an optional bounded wait to the canonical
migration APIs.
> - Operators can join the source-lock queue while all existing
preservation checks remain active.

## Linked Issues or Issue Description

**What happened?**

An online import can repeatedly return `workspace_source_busy` while
background workspace maintenance reacquires its project source lock. The
import does not join the PostgreSQL lock queue.

**Expected behavior**

An administrative import can wait for a finite writer to finish. One
wait budget covers the complete source scope. Timeout or cancellation
must release acquired locks before the import reserves a journal.

**Steps to reproduce**

1. Hold the source lock for an eligible local workspace.
2. Start an online import while the writer holds that lock.
3. Release the writer. Before this change, the import has already
refused admission. With `sourceLockWaitMs`, the import acquires the
source scope and commits normally.

**Paperclip version or commit**

Reproduced on master `0e3fdebea8257bdff11573081c76b306f12db230`.

**Deployment mode**

External PostgreSQL with an online workspace import. Related
preservation work: https://github.com/paperclipai/paperclip/pull/20311.

## What Changed

- Add optional `sourceLockWaitMs` to singleton and cohort migration
APIs. Omitted options keep current admission behavior.
- Queue the sorted source keys with one decreasing wait budget. Abort
cancels only the dedicated backend identified by PID and backend start
time.
- Use an independent source transaction for queued singleton imports.
Keep lifecycle checks, fresh source checks, journal ownership, and
publication after commit.
- Document the canonical API option. Add real PostgreSQL importer
regressions for release, timeout, abort, and partial-key cleanup.
- Correct the inherited task-case lookup summary so the existing
search-discovery regression passes without changing its assertion or
search algorithm.

## Verification

- Baseline RED: the new singleton regression fails on the unchanged
product because the import returns busy without queueing.
- `pnpm exec vitest run
server/src/services/remote-workspace-migration.test.ts -t 'queued|bounds
the whole source queue'`: 20 passed, including five new cases.
- `pnpm exec vitest run
server/src/services/isolated-remote-workspace-source-lock.test.ts`: 4
passed.
- Server layer typecheck, `node scripts/check-module-boundaries.mjs`,
and `git diff --check` passed.
- Tests used Node 24.22.2 and the existing embedded PostgreSQL fixture.
Hosted rollout verification remains pending.
- CI build, typecheck, browser suites, all serialized suites, and the
other general shards passed. The final server shard failed only the
inherited `case-history-api-discovery.test.ts:14` search-ranking
assertion. Its unchanged local regression reproduces on the base and
passes 3/3 with a separate one-line route-summary correction. The shared
one-line summary correction is included in follow-up commit
`8163ffa9d4728134676b84c8c242a5bf62c802b5`; the unchanged case-discovery
suite passes 3/3 locally. The original CI attempt remains failed:
https://github.com/paperclipai/paperclip/actions/runs/42704666536/job/129458605783.

## Risks

Queued admission can hold earlier source keys while it waits for a later
key. The caller controls the finite budget. The abort path opens a
short-lived dedicated cancellation connection. If cancellation cannot
connect, PostgreSQL's remaining lock timeout still bounds the wait. This
option must be called at the root of administrative work; an external
scope that already holds the same source locks can cause
self-contention. The existing pre-held journal-resume path must keep its
omitted-option behavior.

## Model Used

OpenAI Codex, based on GPT-6, with tool use and code execution. The
exact deployment identifier and context-window size are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 16:29:53 -06:00
DottaandPaperclip 3cc1032f6a fix: deliver saved answers to dependency-blocked tasks (#20363) 
## Thinking Path

> - Paperclip lets people direct agent work through tasks and structured
answers.
> - Dependencies stop task execution until prerequisite work finishes.
> - A person must still be able to answer or redirect the assignee of a
blocked task.
> - Ordinary comments already get a bounded answer/triage turn. A saved
question response has no comment ID, so the same gates reject it.
> - Later dispatch also requires an active task and tries to check it
out. That conflicts with bounded interaction on a blocked task.
> - This pull request verifies the stored human response at each gate
and keeps the turn in bounded interaction mode.
> - The assignee can receive the answer without starting dependency work
or removing its blockers.

## Linked Issues or Issue Description

Refs #19923. That change covers dependency waits and comment retries.
This change covers stored interaction responses that have no comment ID.

**What happened?**

A parent task waits on a parked child. The assignee asked a setup
question. After a person answers, the response stays queued. Sending
that saved response is refused as `issue_dependencies_blocked`. Ordinary
new messages can reach the same assignee.

Passing admission alone is insufficient. The dispatch status check
rejects a blocked task, and automatic checkout then tries to start work
whose prerequisites are unfinished.

**Expected behavior**

Deliver the stored human answer or confirmation to its requesting
assignee as a bounded answer/triage turn. Keep the task blocked and keep
its dependency relationships. Preserve current-owner, pause, approval,
budget, recovery, and execution gates.

**Steps to reproduce**

1. Assign a parent task and make a backlog child block it.
2. Save a question on the parent and answer it as a person.
3. Dispatch its resolved-interaction wake, or send the saved answer from
the task queue.
4. Observe that the answer does not reach the provider while an ordinary
comment can run in answer/triage mode.

**Paperclip version or commit**

Reproduced on master at `653ab8b18f76`. This branch is rebased onto
master at `0e3fdebea8`.

## What Changed

- Verify the stored response by company, task, requesting agent, status,
kind, human resolver, and resolution time. A context flag alone is
insufficient.
- Allow a verified response through dependency admission and claim in
the existing bounded interaction mode.
- Recheck that response during scheduled retry promotion and final
provider dispatch.
- Keep bounded responses from checking out the blocked task or setting
it to `in_progress`, including when the last blocker finishes during
preparation.
- Preserve continuation classification through workspace and AI-account
resource waits, so the final response and owner checks still run.
- Correct the task-case API summary so the existing `case for task`
discovery regression passes on the first search page. This fixes a
pre-existing failure exposed by full CI, without changing search ranking
or assertions.
- Ignore obsolete Retry notices after newer conversation work, without
acknowledging saved input or changing execution authority.
- Document the response contract. Add real PostgreSQL and
assembled-heartbeat regressions.

## Verification

- Final commit `ba4b337cf1bb0236b7c0a742665b222ee894ff51`: [required CI
passed, attempt
2](https://github.com/paperclipai/paperclip/actions/runs/42706363120/attempts/2),
including typecheck, production build, server/chat tests, Runner checks,
and all 8 browser shards. PR status: 59 passing checks and 3 normal
skips.
- Red: answered questions and accepted/rejected confirmations return no
admitted run before the fix. All three new positive cases fail at the
dependency gate. A resource-wait retry also reproduces as refused before
its gate is fixed.
- Green: at delivery revision `9375a603e4`, focused heartbeat
integration and classification passed 14 tests; retry scheduling and
dispatch safety passed 241 tests. These use real PostgreSQL and
controlled adapter and resource-availability boundaries. The wider
initial dependency, native-continuation, and queue suites also passed
before the final refinements.
- Negative cases keep pending, missing, wrong-task, and agent-resolved
responses blocked. Changing the response proof immediately before
dispatch cancels the run without calling the provider. Policy controls
retain the current assignee, task status, and pending approval checks.
- Positive cases execute once, leave the parent blocked, preserve the
dependency edge, and leave the child in backlog with no child execution.
The same saved response survives a resource wait; removing its human
proof refuses promotion.
- The resource-wait provider-boundary and blocker-completion regressions
also failed before their fixes. All 14 focused integration and
classifier cases passed at delivery revision `9375a603e4`, including a
changed response during retry preparation.
- Red: a retained Retry for an older run makes a newer successful
conversation display as failed. Green: all 78 database-backed query and
interruption-presentation tests pass, including a later real failure and
independent queued input controls.
- Server layered typecheck and module-boundary checks passed at
`11992aad0c7fbaf13e7548d71ee633560bd8bb6f`, before the one-line API
summary correction. That CI passed build, typecheck and every browser
shard, but exposed the pre-existing case-search assertion and a
report-assets test timeout. Final-head CI passed as linked above.
- Fresh Greptile review is 5/5 on final commit
`ba4b337cf1bb0236b7c0a742665b222ee894ff51`, with no unresolved threads.
- The existing API discovery suite reproduced its first-page assertion
before the summary correction; all 3 cases pass afterward. The timed-out
report-assets privacy test passed locally in isolation and the full
support job passed on the final-head rerun. Runner shutdowns interrupted
the first attempt; affected jobs were rerun once without code changes. A
separate onboarding timeout also passed on that rerun.
- `pnpm verify --explain` requests broad CI because the execution
contract is a global input. The final CI run supplied that coverage.
- Separately recovered the reported staging conversation through an
ordinary message and the existing steering control. The agent
acknowledged the message and the saved-answer queue drained. The fresh
Notebook action subsequently succeeded. This was on the previously
deployed revision; it does not claim this code is deployed.

## Risks

- This changes admission and dispatch. Each gate checks the persisted
response; no caller-supplied marker grants the exception.
- The bounded turn does not check out the task, resolve a dependency, or
approve another action. Existing pause, ownership, budget, approval, and
stopped-execution controls still apply.
- Each additional lookup is one primary-key read and runs only for a
response on a dependency-blocked path. There is no new polling worker.
- No migration is required. Older queued contexts get the exception only
when their stored human response still verifies. No prior run or action
is replayed by this change.
- This PR does not roll out the code to staging.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools, code execution, and
browser inspection. The runtime does not expose the exact model ID,
context-window limit, or reasoning-mode label; those values are not
asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 16:27:34 -06:00
DottaandClaude Opus 5.5 7f55323abb fix(recovery): end a remote workspace epoch whose writer died during preparation (#20346) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task with a remote workspace prepares that workspace before the
run selects its runtime, and preparation marks the workspace epoch dirty
with the run as its writer
> - If the server is killed during preparation, the run never selected a
runtime and never started provider work, but the dirty epoch remains
> - Failed-workspace preservation, the path that resolves a dead
writer's dirty epoch, applies to native runs only, so nothing owned this
epoch
> - The dirty epoch kept the dead run's sandbox lease as a pending
export and held the workspace against the task's retry, so the task
never recovered
> - This pull request lets the ownership reconciler end such an epoch at
the last committed save
> - The benefit is that a restart during workspace preparation no longer
wedges the task

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos wave on Daytona (a remote workspace that clones a
real repository), the controller was killed while each run's first
workspace save was in flight. That save is now settled by #20326. Each
run ended `process_lost` with a bounded retry, and both runs were still
in `execution_stage: preparing` with `runtime_mode_resolved_at: null`.
The workspace authority stayed `dirty: true`, `snapshotId: null`, with
the dead run as writer. The dead run's environment lease was proven
owner-gone but was refused by its settle as `workspace_export_pending`,
and the reconciler reported `I2_settled` violations. The retries stayed
`queued` until the test harness deleted the environment 21 minutes
later. Then they failed `remote_workspace_execution_required`, because
the environment was gone.

**Expected behavior**
When a writer dies before runtime selection, its unsaved files are only
the restored source and preparation output, because no agent turn ran.
Recovery must end that epoch at the last committed save, so the lease
settles and the retry restores the workspace into a new directory. A
writer that reached runtime selection must keep its existing recovery.

**Steps to reproduce**
1. Run a task whose agent uses Paperclip Runner with a remote sandbox
workspace.
2. Kill the server while the run is still preparing the workspace (for
example during its first save).
3. After the restart, the run ends `process_lost` and its retry is
queued, but the workspace stays dirty and owned by the dead run, the
lease stays active, and the retry never starts.

**Paperclip version or commit**
`master` at 251c6d5c69 (after #20326).

Refs #20326 (settles the save that was in flight when the controller
died).

## What Changed

- New `settlePreparationOnlyRemoteWorkspaceEpoch`
(`recovery/remote-workspace-unsaved-files-settlement.ts`), wired as the
settle of the existing `remote_workspace_unsaved_files` ownership kind.
Once the writer run is proven gone, it ends the epoch with the existing
`abandonUnverifiedRemoteWorkspaceEpoch` only when all of these are true:
  - the run is terminal;
  - its runtime was never resolved (`runtime_mode_resolved_at` is null);
  - it ended in `execution_stage: preparing`;
  - it has no native finalization;
- durable records show no work began: no `adapter.invoke` or
`adapter.progress` events, no `turn.*`, `provider.*` or
`native.process_*` events, and no recorded process id, process group,
native session or runner instance on the run;
  - the workspace authority is still dirty, with this run as writer;
  - no save for that epoch is still `running` or `pending`.
The epoch then advances with no writer and no location, and the
abandoned epoch is recorded in `remoteWorkspaceAbandonedEpochs` with an
activity entry.
- After that, the lease no longer matches a prepared source, so the
existing environment lease settle releases it and the workspace writer
hold is free for the retry.
- The ownership inventory row for this kind names the new settle.

## Verification

- Red before the fix, green after: `ends a dirty remote epoch whose
writer died during preparation, and keeps every other writer's files`
(`ownership-reconciliation-report.test.ts`). On master it settles 0
epochs. With the fix it ends only the preparation-only epoch, and keeps
the dirty epoch of a writer whose runtime was resolved, a writer that
reached execution, a live writer, and a writer with a save still in
flight.
- Red before the fix, green after, end to end: `lets a retry prepare the
workspace again after its first writer was killed during preparation`
(`remote-workspace-save-concurrency.test.ts`). It runs the real reserve
and prepare. The first writer's initial save is cut off, which leaves a
first epoch with no committed save (`snapshotId: null`). The run is
reaped as `process_lost` while still preparing, with its lease left
active. Two reconciliation passes then end the epoch and release the
lease. The retry reserves the workspace and prepares it again from the
project base in a new directory, then commits its own initial save. On
master, the epoch stays dirty.
- The safety cases also keep a preparation-stage writer with an
`adapter.invoke` event, and one with a recorded process id.
- Red before the fix, green after: `releases the lease and frees the
workspace for the retry once a preparation-only epoch ends`. Before the
passes, the lease counts as a prepared source and the workspace writer
hold is `held` for a retry. After two reconciliation passes, the lease
is no longer active, the prepared source is gone, and the writer hold is
`free`. On master the prepared source remains.
- The reconciler, retained-source fallback, remote workspace lifecycle,
save-concurrency, warm-handoff recovery, and retained orphan sandbox
suites pass (139 tests, 1 environment-gated file skipped). After
rebasing onto #20326, the reconciler, save-concurrency, retained-source
fallback, and controller instance lease suites pass (91 tests).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.
- Evidence: the runs in the wave's kept database all had
`runtime_mode_resolved_at: null` and `execution_stage: preparing`. Their
`runtime_mode: legacy` is only the column default, not a resolved
runtime.

## Risks

- This ends unsaved files without saving them. That is safe only because
no provider turn can have started before runtime selection, and the
guard requires the writer never to have resolved a runtime. A writer
that did resolve one keeps its existing preservation path.
- The abandoned root stays in the sandbox and is not reused. The next
preparation restores the last committed save, or the source, into a new
directory.
- No schema or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, local database inspection).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 14:12:51 -06:00
DottaandPaperclip c67909cb0f fix: keep tool approvals valid through OAuth refresh and delayed review (#20340) 
## Thinking Path

> - Paperclip lets people manage agent work and approve app actions.
> - An approval must bind the action that the person reviewed.
> - OAuth access tokens can expire while that person is away.
> - The old target snapshot treated each token refresh as a new
authorization. It also imposed a one-hour review deadline.
> - This pull request separates token renewal from authorization
identity and keeps the human decision durable.
> - Current access, the reviewed arguments, meaningful target changes,
task closure, and execution receipts still control dispatch.
> - People can approve later without repeating a decision after routine
connection maintenance.

## Linked Issues or Issue Description

**What happened?**

A pending app action returned `approved_tool_target_changed` after its
OAuth tokens refreshed. The tool and connection settings were unchanged.
The same card also expired after one hour. A sign-in problem could
discard an accepted decision before provider dispatch.

**Expected behavior**

The original card remains reviewable after routine OAuth refresh and a
server restart. A later approval runs the exact reviewed action once. A
connection repair preserves the decision while Paperclip verifies the
same authorization. Meaningful changes require a new review.

**Steps to reproduce**

1. Connect an OAuth MCP app and configure an action to require approval.
2. Request the action, then refresh its access and refresh tokens before
approving.
3. Approve the original card. Before this change, it returns HTTP 409
even when the authorization is unchanged.
4. Wait more than one hour. Before this change, the card expires without
a task-specific deadline.

**Paperclip version or commit**

Reproduced from master at `76254ead2229`. The final branch is rebased
onto current master.

## What Changed

- Add versioned approval target comparison. New snapshots bind
credential identity epochs, consent, permissions, ownership, connection
configuration, tool contracts, and credential references. Authenticated
OAuth refresh preserves identity. Manual credential replacement does
not.
- Remove the implicit one-hour deadline from ordinary app action
reviews. Keep explicit deadlines and terminal-task expiry. Synchronize
the nullable interaction deadline in shared contracts and UI.
- Keep accepted, undispatched actions when sign-in is required. Show a
saved-approval repair state. Resume on committed connection changes
after target and access revalidation, without an idle retry loop.
- Preserve the existing execution claim, result receipt, and
uncertain-outcome no-replay rules.
- Explain meaningful target changes without inviting the person to retry
the stale card.
- Add a production task-thread Storybook example and document
compatibility and recovery behavior.

## Verification

- Red: production-refresher invalid-grant handling and formal-approval
withdrawal during a sign-in wait also reproduced as failing regressions
before their fixes.
- Red: before implementation, the local OAuth HTTP fixture failed for
both public and confidential clients with `approved_tool_target_changed`
after token refresh.
- Green: the extended fixture passes for public and confidential clients
with personal and shared grants. It covers actual local HTTP token
exchanges, multiple refreshes, a rebuilt gateway, an ended originating
run, approval after 25 hours, duplicate approval, preserved sharing
arguments, sign-in waiting, and event-driven repair. The gateways use
the production OAuth refresher; an invalid refresh preserves the
accepted decision before provider dispatch. Omitted-scope responses
cover equivalent scope-reporting changes. The provider is an isolated
local fixture, not a deployed Notebook service.
- Negative controls refuse changed consent, scopes, tool definition,
destination, and manual credential replacement before provider
execution.
- Existing gateway service, connected-tool acceptance, secret owner,
secret service, unsigned-review queue, and connection-health suites
passed. Explicit-deadline fixtures still prove expiry behavior,
including deadlines on accepted sign-in waits. A withdrawn formal
approval cancels a waiting action before connection recovery.
- Approval card, task-chat, error-copy and target-comparison tests
passed. Server and UI typechecks passed. Token gates and Storybook
catalog checks passed.
- Inspected the new saved-approval repair card in local Storybook using
the production task thread. It shows a reconnect link and no second
approval button or false running state.
- `pnpm verify --explain` returned exit 2 and requested broad CI
coverage. [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/42677855507)
passed on `eb4f1c7caa9ee6e802c314ffc5eadec475a5de0c`, including build,
typecheck/release registry, general and serialized tests, and all eight
browser shards.
- Greptile reviewed the same final head at 5/5. Both findings from the
first review are fixed and their threads resolved; no open follow-ups
remain.

## Risks

- No database migration is needed. The new comparison reuses the vault
identity epoch. Only authenticated OAuth refresh code can preserve that
epoch; HTTP credential-write schemas cannot select that behavior.
- Earlier signed requests keep their original exact-version checks and
saved deadlines. Already-expired cards are not revived. A historical
card whose credentials already changed needs a fresh review.
- A new consent, account, permission set, destination, or manually
replaced credential must not inherit the old authorization. Reconnection
continues automatically only when revalidation proves the same saved
authorization.
- The interaction enrichment now permits a null deadline. Existing
string deadlines remain valid. The web client and shared validator
change together.
- This PR does not deploy to staging or perform a live Notebook
publication.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools, code execution, and
browser inspection. The runtime does not expose the exact model ID,
context-window limit, or reasoning-mode label; those values are not
asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 13:46:28 -06:00
DottaandPaperclip 0e325e8d52 fix(runner): settle saved handoffs after verified shutdown (#20336) 
## Thinking Path

> - Paperclip manages AI agents and their work.
> - An agent can save a question or approval handoff before its provider
stops.
> - A missing provider terminal notification does not prove that the
saved work failed.
> - Runner already verifies provider exit, event drain, tool settlement
and suspension.
> - This change uses that proof to settle the same saved handoff.
> - Unproven shutdown holds continuation and explains that the work is
saved.
> - Recovery must never repeat the completed work or allow overlapping
execution.

## Linked Issues or Issue Description

Refs #20330. That PR extends the Codex terminal drain window. This
change handles the separate case where no terminal arrives but the owned
shutdown can prove stopped execution.

**What happened?**

A native run saved a human handoff, accepted cancellation, and later
stopped its provider. The missing terminal notification still made the
attempt fail and show “Couldn’t finish this attempt.”

**Expected behavior**

Keep the saved handoff. Accept its yielded result after either an
authoritative provider terminal or verified shutdown of the exact
execution. If shutdown remains uncertain, retain the handoff and explain
the hold without offering a duplicate Retry.

**Steps to reproduce**

Save a governed wait, acknowledge interruption, and omit the provider
terminal. The new regression uses the real Runner binary and a
controlled Codex process. The old runtime rejects the handoff instead of
asking the existing shutdown owner to settle it.

**Deployment mode**

Native Runner with Codex. Local verification used macOS arm64 and Node
26.4.0. Hosted sandbox recovery has not been qualified.

## What Changed

- Add an optional driver shutdown-settlement operation. Runner returns
proof only after its existing stop, drain, tool settlement and
suspension barrier passes.
- Bind the receipt to the run, normalized session and turn. Persist it
with the saved wait and validate it during recovery.
- Complete the same yielded handoff after verified shutdown. Retire the
stopped session from warm reuse. Do not invent a provider terminal or
complete usage.
- Retain a named shutdown-pending condition under the existing bounded
same-run recovery policy. Preserve actual provider failures and
integrity failures.
- Show saved-work shutdown guidance in task and inbox presentation. Keep
genuine approvals authoritative and disable duplicate Retry.
- Document the completion and restart contract. The checkpoint field is
additive; older checkpoints retain the terminal-or-shutdown path.

## Verification

- Red: the new runtime regression failed against the original
implementation because verified shutdown was never requested. Server
presentation and error-classification regressions also failed before the
fix.
- Red: both inbox shutdown states showed generic text before the inbox
change.
- Green: Runner runtime, shutdown and harness-backend suites pass 301
tests. Tests cover missing terminals, failed stop proof, wrong identity,
cancellation, late proof, genuine provider failure, and restart after
receipt persistence. Recovery completes once without another model turn.
- Green: the native executor and task-presentation suites pass 1,371
tests. Execution projection, inbox and blocker presentation pass 125
tests. Two test-authoring mistakes were corrected: an integrity-error
text expectation and an assertion for a nonexistent projection field.
Their initial failures remain in local logs.
- Runner TypeScript and full server layered type checks pass. Token
gates and diff whitespace checks pass.
- Live provider acceptance: real Runner and Codex with gpt-6.1-sol, one
disposable local workspace, one tool call, and one deliberately dropped
terminal notification. The saved file, bound settlement receipt, one
accepted yielded result, one provider turn, confirmed provider exit and
suspended Runner all passed. The tool handler and persistence port were
controlled. This does not claim a full server or hosted journey.
- The full browser question-and-answer cell stopped before execution
because OPENAI_API_KEY was absent. No model work ran in that attempt.
Hosted sandbox recovery and the full browser continuation journey remain
unverified.
- `pnpm verify` returned exit 2 because the execution-spec change
requires broad CI. Required CI will supply that coverage; a full local
repository suite was not repeated.

- Review correction in `8553da244e`: a failed trailing-event write could
hide a provider failure. A new regression reproduced red, then passed
after all ordinary drain errors retained the named pending condition. No
settlement receipt or result can be accepted after that error. The live
missing-terminal check passed again on the corrected runtime.
- The real PostgreSQL control-plane regression and three existing
governed-wait controls pass. The new case persists one accepted yielded
result, a bound shutdown receipt, and the original pending human
question without a provider terminal.
- First CI revision `7c2cb9b1`: test-program typecheck found a missing
`PrpEvent` annotation in the new fixture. The annotation is fixed and
the Runner test-program typecheck passes. Runner support also timed out
in the unchanged `registered roots remain protected even when equal to
genuine public static bytes` case after 30 seconds. One unchanged local
diagnostic passed in 3.69 seconds overall. The hosted timeout cause
remains unknown; no timeout, fixture, or assertion was weakened.
[Original failed support
job](https://github.com/paperclipai/paperclip/actions/runs/42670851821/job/129346900133).

- Rebased onto the merged #20330 and current recovery changes. The
combined branch passes 1,402 executor/presentation tests. Both prompt
and 18-second provider terminals still pass. A warm session with no
terminal passes after the full 30-second drain and verified
stop/suspension. Its initial overly broad assertion rejected a
nonterminal turn event; the corrected assertion rejects every terminal
type. The rebased CI gate was cancelled before any runner or step
started; its cause is unconfirmed and it supplies no validation credit.
Final revision `b49ec5a144` starts fresh CI and review.

- Final-head CI attempt 1 failed before creating jobs. GitHub reports an
internal server error (correlation
`2c576b07-b67f-4179-a69f-74f7d99cd926`), followed by empty
runner-selection outputs for the aggregate jobs. One unchanged-commit
retry was requested; the initial admission failure remains recorded. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/42673159868).

- **Current blocker:** final head
`b49ec5a144e94e41e96e2508a79bd9aec91fe45a` has a fresh 5/5 Greptile
review and zero unresolved threads, but required CI is not complete.
GitHub reports an active [Actions and Checks API
incident](https://www.githubstatus.com/incidents/tp8cghg7tlc4). Attempt
2 ran only part of the full selected graph. Build, canary, Rust, sandbox
providers, admitted browser shards and other completed test jobs passed.
The E2E aggregate correctly failed because `policy` and `e2e_shards` had
`abandoned` results. Server shard 5/12 received a runner shutdown signal
at 18:36:55 UTC and its test step was cancelled; its job still reports
in progress. The shutdown cause is not established. The separate review
workflow had the same pre-execution internal-error class and passed one
bounded retry. No further retry was submitted during the incident.
Resume with the missing/cancelled coverage after GitHub recovers; do not
count the absent typecheck, Runner or support jobs as passes. [Failed
coverage
gate](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129354318236),
[interrupted server
shard](https://github.com/paperclipai/paperclip/actions/runs/42673159868/job/129350854174).

## Risks

- A missing terminal can use one additional bounded shutdown interval.
Unknown shutdown still blocks continuation.
- Provider terminal and usage accounting keep their separate meaning.
The receipt proves stopped execution and settled tool outcomes, not
model success.
- Recovery rejects a foreign receipt, a later turn, an active turn with
a settled receipt, and an actual failed provider terminal.
- There is no database migration. Older checkpoint shapes remain
readable. Drivers without this optional proof retain a named pending
state.
- This PR does not deploy the fix or repair an existing hosted task.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools and local
code execution. The exact serving model ID and context window are not
exposed in this session. Live test inference used gpt-6.1-sol.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 13:46:16 -06:00
DottaandClaude Opus 5.5 251c6d5c69 fix(recovery): settle a remote workspace save whose process died mid-save (#20326) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshot
operations (`workspace_operations`, phase `remote_workspace_snapshot`).
Each save invocation is listed in `saveWork.activeIds` while it runs
> - When a controller process is killed during a save, its invocation
stays listed and the operation stays `running`
> - Nothing recorded which process ran the invocation, so the shared
ownership reconciler could not prove the owner gone and never settled
the save
> - The task's bounded retry then waited behind that save forever
> - This pull request records the owning process for each save
invocation, and lets the reconciler settle a save whose owner is proven
gone
> - The benefit is that a restart during a save no longer wedges the
task

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos wave on Daytona with a real cloned repository, two
tasks each started their first remote snapshot save (`boundary:
initial`, epoch 1). A SIGKILL hit the controller during the save, before
any upload. Each run ended `process_lost` and got a bounded retry. The
retries stayed `queued` forever, the snapshot operations stayed
`running` with `saveWork.activeIds: [<dead invocation>]` and
`quiescedAt: null`, and the sandbox leases stayed active. The ownership
reconciler reported the saves as owner-gone but had no way to settle
them.

**Expected behavior**
After a restart, recovery must settle a remote snapshot save whose
owning process is gone. It is marked failed with provenance, so the
task's retry can continue. A save that is still running must stay
untouched.

**Steps to reproduce**
1. Run a native task with a remote workspace (for example Daytona with a
`git_repo` project source).
2. Kill the controller (SIGKILL) while the first workspace save is in
flight, before its upload.
3. After the restart, the run ends `process_lost` and its retry is
promoted to `queued`, but the save operation stays `running` and the
retry never starts.

**Paperclip version or commit**
`master` at 1b7fe33432.

Refs #20291 (the quiesced case: a save with no active invocation) and
#20233 (the same ownership pattern for local workspace commands).

## What Changed

- `saveRemoteWorkspace` records the controller process stamp of each
invocation in `saveWork.owners` when it registers the invocation. It
removes that entry when the invocation settles. A module-level set marks
invocations that are still running in this process
(`isRemoteWorkspaceSaveInvocationLive`).
- The ownership registry owns a `remote_snapshot_save` with active
invocations by that stamp. A save whose active invocations have no
stamps was written before this change and is owned by its run, as with
finalize attempts. A mix of stamps, or more than one process, is
unprovable.
- `settleAbandonedRemoteSnapshotSave` settles a save with active
invocations once its owner is proven gone:
- Process-owned: the proven stamp must still match every active
invocation, and none may be live in this process.
  - Legacy run-owned: the run must be terminal.
- Either way it clears the active invocations, records them as
`abandonedInvocationIds`, and marks the save `failed` with `lastFailure:
remote_workspace_save_abandoned`. The #20291 re-arm of blocked fallbacks
still applies.
- The controller instance lease counts running saves stamped with this
boot as held work, so a long save keeps its owner provably alive.
- The ownership inventory row for this kind describes the stamp.

## Verification

- Red before the fix, green after: `settles a remote snapshot save whose
process died mid-save, and keeps live, foreign or unprovable ones`
(`ownership-reconciliation-report.test.ts`). On master it settles 0
rows. With the fix it settles exactly two:
  - a save stamped by a dead process (proof `pid_dead`);
- the wave's shape: an unstamped active invocation on a `process_lost`
run.
It keeps five kinds of save running: a save stamped by this live
process, a stamp from another PID space, a legacy save on a live run, a
save with mixed stamps, and (in the older #20291 test) a stamped active
invocation.
- `stamps each save invocation with the process that runs it and clears
the stamp when it settles` (`remote-workspace-save-concurrency.test.ts`)
pauses a real save mid-flight. It checks the owner stamp and the
in-process live mark, then checks that both are cleared after the save
finishes.
- The reconciler, save-concurrency, remote workspace lifecycle, snapshot
retention, retained-source fallback, preparation gates, and controller
instance lease suites pass (153 tests, 1 environment-gated file
skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.
- Not yet verified: the full Daytona wave replay that shows the queued
retry starts after the save settles. I asked the chaos harness owner to
replay it on this build.

## Risks

- A legacy save (written before this change) with an active invocation
is settled when its run is terminal. The finalize-attempt kind uses the
same rule. During a rolling deploy, an old-release process could still
be running such a save for a terminal run. If that save later succeeds,
its publication still marks the operation `succeeded`.
- If the retry in the wave is also held by the dead writer's dirty
workspace authority, settling the save may not be enough to start it.
The replay will show whether a second change is needed.
- No schema or API change. `saveWork.owners` is new optional metadata.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 13:27:44 -06:00
Dotta 0eacad0d0c fix: refresh preserved harness receipt facts without copying archives (#20322) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Volume-free deployment must preserve historical controller files
before removing the old disk.
> - A WEB preservation receipt binds an archive and source files to the
selected session and database facts.
> - Later metadata changes can make those facts stale while the
preserved files and archive remain unchanged.
> - The existing owner refuses every stale receipt, so preservation
cannot converge.
> - This pull request refreshes only facts freshness after it verifies
the same source bytes, checkpoints and archive.
> - Existing locks and an old-receipt digest comparison prevent a
concurrent replacement from being overwritten.

## Linked Issues or Issue Description

Refs #19814 for WEB evidence preservation without settling remote
recovery. Refs #19832 for bounded larger inventories. Related harness
receipt PRs were searched; no existing change provides this guarded
refresh.

**What happened?**

A valid WEB preservation receipt becomes stale after later run metadata
changes. Its original-volume owner refuses another preservation attempt
because an invalid receipt already exists. This leaves an eligibility
blocker even when the source and saved archive are unchanged.

**Expected behavior**

The original-volume owner can refresh stale facts only after it proves
every other persisted binding unchanged. It retains the prior receipt
and remote recovery state.

**Steps to reproduce**

1. Preserve a settled session's WEB files and controller evidence.
2. Add later run metadata so its database facts digest changes.
3. Preserve the same scope again. The previous implementation refuses at
the existing-receipt check.

**Paperclip version or commit**

The failure was reproduced on master
`12523fb677585fcd6da929f868014514a4e96e87`. This branch includes master
`1b7fe33432016b3b27ccdfbf7b8944da4b40726d`.

**Deployment mode**

Managed Linux server with PostgreSQL and durable object storage.

## What Changed

- Reuse the exact prior archive for existing WEB receipts. Recheck
current source bytes, controller history, checkpoint material and
archive bindings. Read back ciphertext and verify its size and hash.
- Permit only database/local-observation freshness changes. Selected
session, source run, controller, archive, WEB entries, and assets must
still match the prior proof. Local inode and timestamp observations may
change across attempts; preserved bytes, paths and modes must match.
- Recheck current facts and source files under the existing finalizer,
task and run fences. Accept a matching concurrent winner or require the
exact previous receipt digest before replacement.
- Retain the prior database and local facts digests, witness
fingerprint, observation time and receipt digest in the same
company-scoped activity transaction. Require those four fields to
reconstruct the exact prior proof from unchanged current bindings before
replacement. Normal audit redaction stays enabled.
- Preserve current-key checks and archive reads on ordinary idempotent
calls. Keep active-source, changed-evidence and cancellation refusals.
- Document the refresh contract. No schema, saved-run format or public
API change.

## Verification

- Genuine RED: the new real PostgreSQL/encrypted-archive case failed at
the old previous-invalid gate.
- Before the audit review follow-up, the full local owner suite passed
**94/94 integration cases in 50.01 seconds**. It covers same-archive
refresh with no PUT, audit history, current-key failures, unreadable
archives, cancellation, changed files, replaced volume, active sources,
matching concurrent winners and conflicting receipt replacement.
- The cases also cover a source becoming active during an idempotent
readback. Earlier 91- and 93-case runs preceded these final review
guards and are not the final result.
- Server typecheck, module boundaries and diff checks passed on the
final production/test bytes before the clean master merge. Required CI
supplies final-head breadth. Four narrow post-merge
refresh/current-key/idempotent regressions passed in 5.70 seconds on
`82e270f29fff98442c4184b96082f72f18a88721`.
- The fixtures use a real disposable PostgreSQL database and actual
local encrypted archives/storage. Controlled faults affect the stated
storage or concurrency boundary. They do not prove a hosted refresh or
archive plaintext cold restore.
- The existing integration file now exceeds 1,000 lines. Keeping these
cases with its private original-volume, controller, archive and
PostgreSQL fixture avoids duplicating that setup.
- Initial worktree preflight stalled on the global package-manager
wrapper. Preflight completed with the existing pinned toolchain. No
setup failure is counted as a product test pass.
- Review found that activity redaction could change a full prior-receipt
copy. A new regression enabled the real username censor and reproduced
that mismatch. The fix stores only the four changed scalar fields and
their original proof digest; a prepublication canonical reconstruction
guard rejects dropped or unknown fields.
- On the audit follow-up, 16 affected real PostgreSQL cases passed in
11.98 seconds; 80 unchanged cases were intentionally skipped. They
include real redaction with a username-containing WEB path,
reconstruction across two refreshes, unknown prior fields, CAS, active
sources, current-key failures, unavailable archives and cancellation.
Final server typecheck and boundaries passed. This was an intermediate
revision; the final-head results appear below.
- A second review found that comparing old inode/timestamp observations
also broke ordinary valid retries. Three real cases reproduced
valid-receipt touch/inode failures and stale-facts touch failure. The
fix permits metadata freshness changes across attempts while keeping
exact byte/path/mode/checkpoint/archive coverage and the existing locked
within-attempt metadata fence.
- Final local owner suite on the metadata follow-up: **102/102
integration cases passed in 44.04 seconds**, including valid
touch/new-inode retries, stale refresh with the same archive and no PUT,
four-scalar audit reconstruction, changed bytes/mode/entries, and a
touch during archive readback that must refuse. Server typecheck, module
boundaries and diff checks passed. The final-head CI and review results
appear below.
- Final head `c81475c927d0862dcb8d2c2356dc41ab4a27e63f`: [PR workflow
attempt
5](https://github.com/paperclipai/paperclip/actions/runs/42667287803/attempts/5)
completed successfully at 19:24:28 UTC on 2026-11-09. All 51 jobs are
accounted for: 49 passed and 2 were skipped. The actual [final ci /
verify](https://github.com/paperclipai/paperclip/actions/runs/42667287803/job/129382061558)
and [E2E
aggregate](https://github.com/paperclipai/paperclip/actions/runs/42667287803/job/129381486342)
passed. Typecheck, build and the selected test lanes passed. The
exact-head rollup is successful with no pending or failed checks.
- Attempt 5 resolved both trusted workflows to
`3a2f71db31aa99633610dc241f55f34ac65124de`, after #20339 merged. Its
routing gate passed and real jobs started. This verifies successful
startup and completion of this rerun; it does not exercise the
empty-runner fallback because routing succeeded.
- Earlier CI failures on `82e270f29fff98442c4184b96082f72f18a88721` and
attempts 1–4 of this final-head workflow remain nonpasses. Attempt 1 had
interrupted leaf jobs and failed aggregates. Attempts 2 and 3 ended in
workflow failure without a final verify job. Attempt 4 failed startup
with zero jobs and empty-runner errors. Those failures are not counted
as passes or labeled unrelated.
-
[Greptile](https://github.com/paperclipai/paperclip/pull/20322#issuecomment-6746395090)
and
[Ponytail](https://github.com/paperclipai/paperclip/pull/20322#issuecomment-6746327827)
reviewed the unchanged final head at **5/5**. Both historical review
threads are resolved; no inline thread remains open.
- Hosted application to an actual stale scope remains pending normal
release delivery.

## Risks

- Refresh must not bless different preservation evidence. The
unchanged-binding checks, fresh file and checkpoint proof, exact archive
reuse, source capability and locked receipt comparison remain mandatory.
- A fresh capability has a new witness fingerprint and expiry. The audit
retains the old fingerprint, database/local facts digests and
observation time. All unchanged proof bindings remain on the current
receipt. Their canonical reconstruction must match the old digest. The
new capability is validated at entry and publication.
- Readback verifies ciphertext size/hash and current-key checkpoint
material. This change does not claim a new archive GCM/plaintext
restore.
- Archive identity and rows remain unchanged, so retention ownership
does not change. Failed refreshes retain the previous receipt, original
files and unresolved remote recovery obligations.
- Older receipt shapes are not rewritten automatically. Normal
validation and named preservation refusals remain in force.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository
tools and code execution. Context-window size is not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-11-09 13:26:27 -06:00
DottaandClaude Opus 5.5 bce7503a82 perf(workspaces): time a restore's sandbox upload and extraction separately (#20331) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote (sandbox) workspaces are restored from a durable snapshot
before a later turn, so restore time delays that turn's first token
> - #20280 added run-log timing for the storage steps of a save and a
restore
> - A Daytona wave with a Paperclip-sized workspace (~750 MB) spent 39.9
s preparing the workspace of a restored run. The storage steps
(download, decrypt, verify, publish) took 2.4 s, and
`workspace.snapshot.sandbox_restore` took 35.6 s
> - The archive helper unpacks the same archive locally in 3.6 s, so
most of that time is elsewhere, probably copying 750 MB into the
sandbox. One span cannot say which
> - This pull request splits that span into the upload into the sandbox
and the extraction in it
> - The benefit is that the next measurement shows which part to fix

## Linked Issues or Issue Description

Refs #20280

**What happened?**
A restored run on Daytona spent 35.6 s in
`workspace.snapshot.sandbox_restore` (wave `w-20261109T170632-s9921`, R1
workload, 754,838,327-byte snapshot). The run log records only that
total, so it cannot show whether the copy into the sandbox or the unpack
inside it is slow.

**Steps to reproduce**
1. Run a remote workspace task with a large repository on Daytona and
let a later run restore it.
2. Read the restored run's `run.performance.span` events.
3. Only `workspace.snapshot.sandbox_restore` covers the time after the
storage download.

**Expected behavior**
The run log separates the upload into the sandbox from the extraction
there.

## What Changed

- `remote-workspace-transport.ts` (`restoreRemoteWorkspaceDirectory`):
the archive copy into the sandbox runs inside
`workspace.snapshot.sandbox_upload`, and the in-sandbox restore command
inside `workspace.snapshot.sandbox_extract`. Both are children of the
existing `workspace.snapshot.sandbox_restore`.
- `execution-preparation-timing.ts`: the two new phase names. A traced
`measureRunPhase` scope also carries its preparation ancestor's run
(`recordRunId`). Without it, with an OTLP endpoint set, a step nested
inside another step (here, upload and extract inside `sandbox_restore`)
never reached the run log. Execution attempts still record nothing.
- `doc/run-log-events.md`: the two rows in the durable storage timing
table.
- No behavior change. This is a run-log change: the rows stay in the
instance database.

## Verification

- New test `times the restore's sandbox upload and extraction as
separate run-log steps` (`remote-workspace-transport.test.ts`). It
restores a real archive through the real helper, inside the production
`sandbox_restore` step with an OTLP endpoint set, with a 60 ms pause in
each step, then reads the drained run phases. It fails without the spans
or without the scope fix.
- New test `records a step nested inside another step when an OTLP
endpoint is set` (`execution-preparation-phases.test.ts`). It fails
without the scope fix.
- `npx vitest run` on `execution-preparation-phases`,
`execution-preparation-timing`, `execution-timing-context`,
`database-operation-attribution`, `native-workspace-finalization-work`,
`remote-workspace-transport` and `remote-workspace-snapshot-storage`:
256 passed.

## Risks

- Low risk. Two more measured steps on the restore path; spans under 50
ms are not recorded.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 12:56:25 -06:00
DottaandClaude Opus 5.5 f56b097fbd fix(recovery): move stranded legacy subtasks off a parent's held workspace (#20314) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents split work into subtasks. Since #19860, every subtask gets
its own execution workspace from the project base, and a subtask that an
older release bound to its parent's workspace moves to its own workspace
on its next admission
> - Some legacy subtasks never get a next admission. Another task held
the shared workspace with unsaved files, so their runs failed setup
before any provider work. Stranded-work recovery then blocked them and
handed them to the board
> - On staging, a parent and two of its children stayed blocked for four
days this way. The children could have run in their own workspace at any
time
> - This pull request makes the scheduled recovery-action sweep move
such a child to its own workspace and continue it, with no person
> - The benefit is that a legacy subtask never stays blocked on a
workspace that it should not share, and the parent keeps its own
workspace

## Linked Issues or Issue Description

Refs #19860 (subtasks get their own workspace; legacy children move at
admission)
Refs #20295 (the held workspace's exhausted save now settles
automatically)
Refs #20273 (stall wakes wait while a workspace hold refuses a run)

**What happened?**
A parent task and three children shared the parent's remote workspace,
because an older release created the children before #19860. One child's
ended run held the workspace with unsaved files. The other two children
were woken, but each run failed setup with "The workspace still has an
unfinished writer or save operation." (`setup_failed`,
`executionRecovery.kind=bootstrap`, `providerWorkStarted=false`).
Stranded-work recovery then set each child to Blocked with a board-owned
escalation (`board_escalation_no_takeover_v1`) that asked a board
operator to retry. The #19860 move happens only at the next admission,
and a blocked child gets no admission.

**Expected behavior**
A subtask that failed only because of a legacy shared-workspace binding
continues in its own workspace from the project base automatically. The
parent and the shared workspace stay unchanged.

**Steps to reproduce**
1. With a release before #19860, create a parent task and children that
share its remote workspace.
2. Let one child's run end with unsaved files whose save is not settled.
3. Wake another child. Its run fails setup on the workspace hold, and
stranded-work recovery blocks it with a board escalation.
4. See that the blocked child is never moved to its own workspace and
never runs again.

**Root cause**
`server/src/services/execution-workspaces.ts`
(`releaseAncestorWorkspaceBindingAtAdmission`) moves a legacy child only
when a run of the child is admitted.
`server/src/services/recovery/service.ts`
(`reconcileStrandedAssignedIssues`) ignores Blocked tasks, so nothing
admits a child that a setup failure on the shared hold blocked.

## What Changed

- `server/src/services/execution-workspaces.ts`: extract the binding
release into `releaseAncestorWorkspaceBinding(tx, …)`, which runs inside
the caller's transaction. Admission uses it unchanged (same guards: no
other running run of the task, no unsettled workspace recovery, only a
relative's workspace).
- New `server/src/services/recovery/legacy-workspace-rehome.ts`
(`rehomeStrandedLegacyWorkspaceTasks`). It selects a board-owned
stranded escalation whose latest run failed setup on the workspace
writer hold before any provider work, for a Blocked task bound to a
relative's workspace. In one transaction it releases the binding,
resolves the escalation with the outcome
`legacy_shared_workspace_rehomed`, sets the task to `todo`, records a
pending wake (`legacyWorkspaceRehome.wake: "pending"`), and records
`issue.legacy_workspace_rehomed` system activity. After commit it wakes
the assignee once (`issue_recovery_action_restored`, with a fresh
session) and marks the wake `delivered` only when a run is created. A
wake lost to a crash, refused (`null`), or failed stays pending and
backs off (5 minutes, doubling, at most an hour); the pass returns the
earliest owed wake as its retry deadline, so the recovery worker
schedules it. Candidates are paged by action id.
- Guards: the Blocked status must come from that escalation (set after
the failed run, and no status decision at or after the block), the
shared dependency-readiness rule must allow work (cancelled blockers and
done blockers that still save files keep it blocked), the failed run
must belong to the current assignee, and the binding release must be
allowed. Otherwise nothing changes.
- `reconcileActiveRecoveryActions` (the company-scoped sweep that the
scheduled issue-recovery worker runs for each company with an active
recovery action) runs this pass first. `recoveryActionSweepCondition()`
also selects a company whose moved task still owes its wake.
- `doc/execution-semantics.md`: documents the move and what a moved
child sees.

### What a moved child sees

A fresh workspace from the project base, exactly like a new top-level
task in the project. It does not get the parent's changes. It does not
get its own earlier files from the shared workspace either, because
nothing of its work ran there (its runs failed setup before any provider
work). The child that still holds the shared workspace with unsaved
files is not moved by this change: its save recovery owns the hold
(#20295 settles it automatically), its files stay in the shared
workspace and in that recovery's saved-work receipts, and it moves on
its first admission after that recovery settles (the #19860 rule).

### Previous-release story (`rules/upgrade-safe-run-state.mdc`)

- **Migrated, no schema change.** The change reads escalations and task
bindings that older releases saved, and it changes only the task's
binding row, the task status, and the escalation. It does not change
saved run identity (native execution input, leases, checkpoints,
sessions). The regression test seeds the exact staging shape (Nov 4–5
evidence fields, `retryReason: workspace_busy`, `latestRunErrorCode:
setup_failed`).
- **Mid-run deploy.** A child with a running run or an unsettled
workspace recovery is not moved (the same guards as the admission
release).

## Verification

- Red (the move disabled), `npx vitest run
src/__tests__/legacy-workspace-rehome.test.ts`: 1 failed, 2 passed. The
staging-shape case failed because the stranded child stayed Blocked and
bound to the parent's workspace.
- Green: 8 of 8 pass. All cases run the scheduled path: the company is
selected by `recoveryActionSweepCondition()` and the pass is
`recoveryService(...).reconcileActiveRecoveryActions({ companyId })`
with the real heartbeat wake.
- Staging shape: a parent and 3 children share the parent's workspace.
One child holds it with a dirty writer run. Two children are blocked by
board escalations after setup failed on that hold. After the
stranded-work sweep, both children are unbound from the shared
workspace, their escalations are resolved by the system with text that
asks no person to act, and each child is woken exactly once with a fresh
session. Each child runs once through the real heartbeat path outside
the shared workspace and completes. The parent and the writer child keep
their bindings, the shared workspace row is unchanged, and nothing runs
for them. A later pass changes nothing.
- A move whose wake was lost to a crash: the scheduled worker still
selects the company, delivers the wake once, and the child runs and
completes.
- A refused wake (`null`) or a thrown enqueue stays pending, backs off,
does not hold up other wakes, is reported to the worker as the next
deadline, and keeps the company selected after a restart; once accepted,
it settles.
- Thirty rejected escalations do not hide a later eligible one (paging).
- A child with a cancelled blocker stays blocked with its board
escalation.
- A child whose last status decision came before the block moves; one
decided after the block stays blocked.
- A stranded child whose setup failed for another reason stays blocked
with its board escalation.
- A task bound to the workspace it created (the parent) is never moved.
- Related suites pass: `subtask-workspace-independence`,
`heartbeat-workspace-busy`, `heartbeat-remote-workspace-busy`,
`heartbeat-held-comment-wake`, `recovery-deadline-acceptance`,
`heartbeat-retry-scheduling`, `task-workspace-binding` (278 tests), and
after the review changes `issue-blocker-stall-wakeups`,
`issue-recovery-deadlines`, `issue-recovery-work-lifecycle`,
`recovery-durable-wait-projection`, `owner-gone-attestation` (124 tests
with the two workspace suites).
- `pnpm verify`: exit 2 ("broader verification required", because
`doc/execution-semantics.md` is a global CI input). Required CI supplies
the broad coverage.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.

## Risks

- The pass changes a Blocked task to `todo` only under narrow guards,
and only for a task bound to a relative's workspace whose last run
failed setup on that workspace's writer hold. A Blocked status from
another decision, or an unresolved blocker, keeps the task blocked.
- The moved child starts again from the project base. This is the
product rule for subtasks. Nothing of its own work is dropped, because
its runs failed before any provider work.
- The pass runs inside the existing active recovery action sweep and
selects at most 25 escalations and 25 pending wakes per pass.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser read-only API checks) and
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 12:15:25 -06:00
DottaandPaperclip 76254ead22 fix(runner): allow Codex interruption to settle governed waits (#20330) 
## Thinking Path

> - Paperclip manages AI agents and their work.
> - A native run must stop provider work before it waits for a human
response.
> - The Codex transport allows 30 seconds for a control command.
> - The governed-wait loop used its 15-second default for Codex.
> - A valid interruption terminal at 18 seconds therefore failed the
run.
> - This change gives Codex the existing 30-second terminal drain
allowance.
> - The run still fails if it cannot prove that the provider turn ended.

## Linked Issues or Issue Description

**What happened?** A Codex run can save a human wait, accept an
interruption, and fail before the provider sends its terminal. Its saved
wait remains intact, but normal finalization does not occur.

**Expected behavior:** Allow terminal delivery through the transport's
bounded command window. Keep missing-terminal cases closed.

**Steps to reproduce:** Run the new protocol regression against the old
backend. The real Runner binary uses a controlled Codex process that
delays its interruption terminal for 18 seconds. The run fails at the
15-second drain deadline. The same process passes with this change.

**Paperclip version or commit:** Reproduced against `1b7fe33432`.

**Deployment mode:** Local test of the native Runner path. The incident
was observed in a hosted sandbox.

**Installation method:** Source checkout.

**Operating system:** macOS arm64 for local verification. Hosted
incident runtime was Linux.

**Additional context:** No matching repair was found. This addresses a
reproducible deadline mismatch. The hosted log ends terminal observation
at 15 seconds, so it cannot prove whether that provider would have
settled within 30 seconds. The approved local live-provider check
passed. This PR does not claim recovery of the hosted task.

## What Changed

- Give Codex a 30-second governed-wait terminal drain through the
existing backend option.
- Keep other providers and the caller's shorter execution bound
unchanged.
- Add real Runner protocol cases for prompt, delayed, and absent
terminal delivery. Check retained wait state and one provider turn.
- Document the bounded wait and terminal-proof requirement.

## Verification

- Red: the 18-second controlled-provider case failed with
`governed_wait_provider_terminal_unproven` on the old backend. The
prompt case passed.
- Green: 282 tests passed across the native runtime, harness backend,
and new Runner protocol suite, including a repeat on the rebased PR
revision. `pnpm verify --explain` returned exit 2 because the
execution-contract document requires broad CI; the [required CI
run](https://github.com/paperclipai/paperclip/actions/runs/42668235142)
passed on the final head. All 58 applicable checks passed; the three
skipped checks were the two unrelated Storybook jobs and the
selected-test planner replaced by broad coverage.
- The missing-terminal case still fails without `completeRun` or a
successful run terminal. The saved governed wait remains available. Only
one provider turn and one tool call occur.
- Existing runtime tests cover shorter deadlines, foreign or unproven
terminals, complete usage without a terminal, and checkpoint recovery
without another model turn.
- Live acceptance at `f3da2b87425d4fd45c649bfc98785c61f82abbe0`: real
Runner binary, Codex CLI 0.165.0, gpt-6.1-sol, Node 26.4.0, macOS arm64,
disposable local workspace. One `request_human_input` tool call returned
a pending interaction fixture. The real provider emitted
`turn.interrupted`; the controller accepted the yielded wait and its run
terminal. The check passed in 8.14 seconds. The tool callback and
persistence port were controlled; no hosted server permission or
Notebook publication claim is made.
- Initial local probe setup failed before model execution: first the
explicit launch lacked `app-server`, then it omitted the Runner
permission configuration. The successful check used the Runner's normal
launch arguments and isolated home with the approved existing account.
- Hosted sandbox acceptance and recovery remain unverified. The
18-second delayed shutdown is a controlled protocol regression; the live
model check proves normal shutdown, not that specific latency.

## Risks

- A Codex provider that never stops can occupy the settlement window for
up to 15 additional seconds. It still fails closed and is quarantined.
- No persisted schema, provider identity, permission, or recovery rule
changes. Previous-release checkpoints use the same format and existing
recovery checks.
- This change cannot fix a provider that never emits a terminal. It also
does not grant missing setup permissions or decide pending approvals.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and context
window are not exposed in this session. Used reasoning, tool calls, code
edits, and local execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 12:11:47 -06:00
Dotta a3e7e63330 Guard Native recovery with process-space proof and explicit containment (#19813) 
## Thinking Path

> - Paperclip manages agents and their work.
> - Recovery must preserve execution ownership and accepted results.
> - A saved PID proves nothing outside its recorded process namespace.
> - Uncertain unfinished work must stay held until its original owners
stop.
> - Operators need a source-bound containment action and a separate
ordinary Retry.
> - Accepted results must use the existing finalizer without another
provider turn.
> - This PR keeps those paths safe, recoverable, and quiet between real
obligations.

## Linked Issues or Issue Description

Refs: #19713

Refs: #19591

Recovery could use a local PID to release or replace a controller from
another process namespace. Connected-chat Retry also rejected successful
containment on a task in `todo`. The new containment action used a
broader permission than the existing owner-stop action. A later review
found that result-less guards also rejected the existing accepted-result
confirmation action.

## What Changed

- Record and verify the controller's kernel/PID namespace. Keep unknown
and foreign unfinished ownership held. Preserve independent durable stop
evidence and the existing remote lease fences.
- Use the shared accepted-result ownership reconciler. Bind settlement
to the observed lease token, generation, boot, PID, namespace and
process birth. Retain finalizer, cleanup, current-controller, live-owner
and remote exclusions.
- Share the remote-owner reader across recovery consumers. Scope runner
leases to the same company, task and run. Do not make local PID probes
for remote accepted-result ownership.
- Gate original-host containment on `runtime:confirm_owner_stop`. Bind
it to the current task, source, assignee, original expiry, lease and
Stop state. Audit separate controller/provider containment and reviewed
action outcomes.
- Keep containment separate from ordinary Retry. Use one canonical
committed receipt and the normal admission, budget, workspace and
lineage checks. Preserve the recorded external-action outcomes.
- Keep the controller-only confirmation actionable for both result-less
work and accepted results. Result-less work gets its next source-bound
attention in the same transaction. Accepted results keep their original
result and use normal shared settlement/finalization.
- Keep identity inspection quiet. Preserve the original foreign-owner
expiry and use the existing scoped finalizer in accepted-result recovery
controls.
- Update the recovery card, validators, execution projection, docs and
access maps. Reuse existing canonical columns; add no migration or
ownership backfill.

## Verification

Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`.

**The current fix has a causal red/green check.** On the test-only
revision, both foreign-local stopped/fenced controller cases failed
because recovery probed saved runner, process-group and provider PIDs in
this host's namespace. This measured the wrong probes; it did not run
duplicate providers. After the fix, the same two cases passed every
no-probe, no-claim, no-controller-stamp, no-stop-receipt and
unchanged-authority assertion. Three existing remote preparation, grace
and fenced-controller progress cases also passed. Total: **5 passed, 0
failed;170 filtered rows excluded**. All three normal affected App
TypeScript programs passed on this head (660,681 and428 roots).

The fix separates a controller wait from execution stop proof.
Independent live/grace evidence still schedules recovery. A
stopped/fenced foreign controller can pass this namespace guard only for
remote execution, which keeps its later checkpoint and never-launched
checks. Foreign local execution stays held before runner or provider PID
inspection.

Earlier source-scoped results remain evidence, not fresh executions on
the current head:

- On `c86dd741`, the complete failed-owner file passed **25/25**, with
no skips or filters. Empty scans wrote no controller stamps. Foreign
ownership kept its original expiry. Normal returned-claim settlement and
attention at the exact deadline passed.
- On `c86dd741`, **15 selected accepted-result/restart controls
passed**. The81 filtered rows are excluded. Both completion cases used
the existing scoped finalizer and kept their accepted result without
another provider turn.
- Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334`
plus `c86dd741`) passed the complete **17-case maintenance-server
file**. This includes the original90-real-second quiet observation and
two qualified hourly windows. The quiet case took118,918ms. The four
jobs that failed on the earlier publication all passed on that synthetic
merge.
- On `6243b32c`, the complete ownership-report file passed **29/29**.
Native NULL namespace stayed unprovable and generic compatibility
remained. Three selected recovery controls passed;170 filtered rows are
excluded.
- On `f81f2c3c`,356 business cases passed: whole ownership-route221,
recoveryUI70, owner-proof38 and controller-confirmation8; selected
accepted-result authority18; and one real connected-chat containment,
separate authorized Retry, useful Native result and downstream reply
case. The2,024 filtered rows are excluded. A separate earlier
accepted-result causal case kept the original result and completed after
restart without replay.

Normal Shared exports, App programs, UI typecheck, token gates and map
checks passed on their recorded revisions. Unchanged DB, adapter, Eval
and Runner inputs/outputs have explicit carry proofs. The clean normal
merge with fetched master `8fe524f7` preserves all31 exclusive owned
paths and14,827 unrelated master entries. The nine overlapping paths
were reviewed, including the upstream unidentified-run stop action.
Fresh CI must validate that actual merge composition.

PG cases used existing external PG17, fresh owned databases, pools at
most4, one worker and no SQL observer during cases. Registered clients
and recorded process groups closed. Independent final SQL audits found
zero campaign sessions before normal database cleanup. Original failing
fixtures remain retained.

Original failures are preserved. One new test first failed compilation
due to a missing getter import; an import-only correction passed its
full owning programs before the causal red/green check. A local quiet
attempt stopped in the normal prerequisite hook before any business case
and its collector rejected two normal build markers. The later17-case CI
pass establishes quiet recovery; it does not turn that failed local
collector into a pass. An earlier useful case passed before its private
cleanup collector failed on an owned read-only directory; normal cleanup
completed after all owners closed. Earlier runner-shutdown cancellation
and a separate private whole-suite failure remain unexplained.

Current-head checks are complete:57 success,1 neutral and3 skipped,
including the successful aggregate CI gate. The legacy status passed.
Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this
head. All seven review threads are resolved. Full recursive typecheck,
the full test suite and release build were not repeated locally. No
result here proves deployed suspension, a live provider or termination
on another host.

## Risks

- Unknown physical ownership can require an owner/admin action. A false
containment statement can be unsafe.
- New work, reassignment, Stop, renewed authority, accepted results or
pending cleanup can invalidate an action. The operator must refresh.
- Controller confirmation does not prove provider/tool containment.
Completed external actions must not be replayed.
- Generic legacy and child-process compatibility remains separate from
the strict Native controller policy.
- Existing saved state and canonical schema are preserved. Permanent
safety refusals remain held with their normal named recovery path.

## Model Used

OpenAI `gpt-6.1-sol` agents implemented and tested the changes with
reasoning, repository tools and code execution. A GPT-6-based Codex
managing agent reviewed and integrated them. The host does not expose
the manager's exact build or the context-window sizes.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-11-09 12:06:11 -06:00
Devin FoleyandPaperclip 0f84df91da fix(native): accept a completed child's file in the parent's completion report (#20325) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In onboarding, the agent plans the user's first task, creates one
child task for the work, and waits for it
> - The `first-task` skill tells the agent to publish a requested file
on the child task. When the parent wakes, the agent links the child's
file and does not publish a copy
> - Native `register_deliverable` is bound to its current task, so the
parent cannot register the child's file again
> - The native completion check required each cited attachment to be on
the task being completed
> - So when the parent cited the child's download link,
`paperclip_finish` failed, and the agent reported the finished
onboarding task as blocked
> - This pull request lets a completion report cite an attachment on a
direct child task that is done
> - The benefit is that onboarding closes after its child task delivers
a file, and the user does not see a false "blocked" status

## Linked Issues or Issue Description

Related: #18258 (onboarding artifact completion instructions).

**What happened?**
An onboarding run created a child task to build and publish a project.
The child task registered `project.zip` with `register_deliverable` and
was marked done. The parent woke with reason `issue_children_completed`.
The agent called `paperclip_finish` with `reportedWorkDisposition:
"done"` and cited the child's `/api/attachments/<id>/content` link. The
call failed two times with:

```
Completion cites no registered attachment on this task. Use register_deliverable for the requested file and cite deliverable:<attachmentId> from its receipt.
```

The skill does not let the agent register the file on the parent, so the
agent called `paperclip_block` with "Paperclip completion validation" as
the owner. The parent task stayed `blocked`. Nothing wakes it again.

**Expected behavior**
The parent completes and links the child's file.

**Steps to reproduce**
1. Create a task with a child task.
2. In a native run on the child, register a file with
`register_deliverable`, and mark the child done.
3. In a native run on the parent, call `paperclip_finish` with
`reportedWorkDisposition: "done"` and the child's
`deliverable:<attachmentId>` or
`/api/attachments/<attachmentId>/content` as evidence.
4. The call fails with "Completion cites no registered attachment on
this task".

## What Changed

- `server/src/services/native-runtime/native-deliverable-feedback.ts`:
when a cited attachment is not on the task,
`validateNativeDeliverableEvidence` also looks for it on a direct child
task with status `done`. If it is there, the citation is accepted.
- The child's file is context, the same as an attachment from an earlier
run on the same task. It does not count as this run's publication of a
file that the parent's own request names.
- With `lockSources`, the finalization path locks the attachment and
child task rows with `FOR SHARE NOWAIT`, as it does for other sources.
  - The error text now says "on this task or its completed child tasks".
- This applies to the `paperclip_finish` check and to the two
finalization checks, because they use the same function.
- `doc/AGENT-ARTIFACTS.md`: the native runner section states that a
completion report can cite a file on a completed direct child task.

## Verification

- Extended "prepares one verified same-run attachment and replays
without duplicates" in
`server/src/services/native-runtime/native-runner-file-handoff.test.ts`:
  - An attachment on an unrelated task is still rejected.
  - An attachment on a child task that is not done is still rejected.
- An attachment on a done child task is accepted, through
`paperclip_finish` and through the locked finalization path.
- When the parent's own request names a file, the child's file alone
does not satisfy it.
- Without the source change, the test fails with the same error as the
run above.
- `npx vitest run` on `chat-attachment-reuse`,
`native-completion-feedback`, `native-completion-format-feedback`,
`native-deliverable-feedback`, and `native-runner-file-handoff` tests in
`server/src/services/native-runtime`: 281 passed.
- `npx vitest run src/__tests__/native-result-commit-work.test.ts
src/__tests__/native-status-arbiter-corpus.test.ts` in `server`: 167
passed.
- `npx tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Only direct children with status `done` are accepted. A grandchild's
file, or a file on a child that is not done, is still rejected.
- A parent that names a requested file in its own request still needs
its own delivery evidence. This keeps the current rule for earlier
outputs.
- A task that is already blocked by this error does not change. The user
can mark it done, or comment so the agent runs again.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code reading, database queries, shell commands, and
test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 09:44:17 -08:00
DottaandClaude Opus 5.5 47dccbf3eb fix(recovery): re-prove an exhausted save's agent stop and free the workspace (#20295) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent run ends, Paperclip saves the files that the run
changed in its remote workspace. Until that save settles, the run holds
the workspace, so no other task can write it
> - When the save fails, a bounded system recovery proves that the agent
stopped (a provider stop receipt) and then saves the files again
> - If the stop cannot be proven before the 30-minute recovery window
ends, the recovery is exhausted with `agent_stop_unconfirmed`.
Exhaustion was terminal: Paperclip never proved the stop again, even
after the run ended and its sandbox had a stop receipt
> - The hold then stayed forever. On staging, a child task's save was
exhausted this way on Nov 5. Its parent and its two sibling tasks share
that workspace, and all of them stayed blocked for four days
> - This pull request re-arms such an exhausted save automatically, on a
bounded backoff, once the run is provably over. After the bounded
re-arms, the hold continues from the last save with the unsaved files
attached
> - The benefit is that a dirty workspace hold always settles without a
person, and no unsaved work is discarded silently

## Linked Issues or Issue Description

Refs #20273 (the stall sweep stops failing wakes against this kind of
hold)
Refs #20278 (a settled restore hold returns its warm sandbox)
Refs #19860 (subtasks get their own workspace)

**What happened?**
A child task's run wrote its remote workspace and failed to save it
(`native_workspace_sync_out_retry_exhausted`). The save recovery made
one attempt. That attempt failed with `agent_stop_unconfirmed`, and the
recovery window ended before a second attempt. The recovery became
`exhausted`. Ten minutes after the first attempt, the sandbox lease got
a provider stop receipt, so the agent was provably gone. But nothing
looked at the exhausted recovery again. The run kept the workspace's
writer hold. The parent task and every other task on that workspace
could not start ("The workspace still has an unfinished writer or save
operation."). The blocker told a person to press Retry.

**Expected behavior**
An unproven stop is a wait, not a refusal of the saved files. When the
run is over, recovery must prove the stop again and finish the save, so
the hold settles automatically. If the stop still cannot be proven,
Paperclip must keep the unsaved files and then release the hold. A
person must not need to act.

**Steps to reproduce**
1. Run a task in a remote (sandbox) workspace. Make its final save fail.
2. Keep the workspace recovery from proving the agent stop until its
30-minute window ends (for example, keep the finalization ownership
busy).
3. See that the recovery action is `exhausted` with
`agent_stop_unconfirmed`, and that it stays so after the lease gets a
stop receipt.
4. Wake another task bound to the same workspace. See that its setup
fails because the workspace is held.

**Root cause**
- `server/src/services/workspace-recovery.ts`
(`recoverRetainedWorkspace`): an `exhausted` state only goes to the
sibling settlement and to the retained-source fallback. The fallback
accepts only permanent source refusals (`isUnverifiableSourceFailure`).
Every other exhausted state is handed to the board and never attempted
again.
- `recheckStoppedWorkspaceRecovery` skips `exhausted` states, and it
only accepts acknowledged cancellations. A failed run whose lease got a
stop receipt later is never re-examined.
- `retainedWorkspaceCandidates(…, obligationsOnly)` and
`nextWorkspaceRecoveryAt` do not select exhausted
`agent_stop_unconfirmed` holds, so the recovery worker never wakes for
them.

## What Changed

- New `server/src/services/workspace-recovery-stop-proof.ts`
(`rearmExhaustedStopProof`). For an exhausted save whose last failure is
`agent_stop_unconfirmed` (`recheck_evidence`), it starts a new bounded
system recovery generation. It does this only when the source run is
provably over: the run is terminal, no local execution remains, and no
controller lease or finalizer lease is live. The new generation repeats
the strict stop proof (a provider stop receipt) and the save. It records
an `issue.workspace_recovery_rearmed` system activity.
- Re-arms are bounded: at most 4. The first is due at once. The next
ones are due 30 minutes, 2 hours and 8 hours after the previous re-arm.
The `stopProofRecovery` marker in the recovery action evidence records
the count and the next due time.
- `workspace-recovery-fallback-discovery.ts`: new discovery conditions
(`stopProofRearmPendingCondition`, `stopProofRearmDueAt`,
`stopProofRearmDueCondition`) and `retainedHoldFallbackEligible`.
`retainedWorkspaceCandidates` (worker pass) and
`nextWorkspaceRecoveryAt` use them, so the recovery worker wakes when a
re-arm is due.
- `workspace-recovery-fallback.ts`: when every re-arm is spent and the
stop is still unproven, the hold takes the existing retained-source
fallback. The unsaved files are attached to the task (unapplied, secrets
omitted), the unverified epoch is abandoned, and the task continues from
the last save.
- `system-recovery-state.ts`: the exhausted `agent_stop_unconfirmed`
text now says that Paperclip tries again automatically. It no longer
asks a person to press Retry. Retry stays available.
- Review follow-ups: the re-arm applies to every workspace recovery the
worker owns (restore and historical holds too). A due re-arm is
scheduled no earlier than the source run's controller or finalizer lease
expiry, so the worker is not woken every 15 seconds while a lease is
live. A fallback whose bounded checks are spent is reconsidered once
when an allocation of the source run changes, and
`settleStopOnlyCleanup` wakes the workspace recovery worker when it
records a stop receipt. The automatic-retry text is shown only while a
re-arm remains.
- `doc/execution-semantics.md`: documents the re-arm, its bounds, the
fallback, and the upgrade behavior.

### Previous-release story (`rules/upgrade-safe-run-state.mdc`)

- **Carried forward, no migration.** A hold exhausted by an older
release has no `stopProofRecovery` marker. The recovery worker treats it
as due at once and re-arms it on its first pass after deploy. The new
regression test saves the exact Nov 5 evidence shape (`attempts: 1`,
`retriesUsed: 0`, `state: exhausted`, no marker) and resumes it with the
new code.
- **Mid-run deploy.** A run that is still running, or whose controller
lease or finalizer lease is live, is not re-armed. The test covers a
live controller lease. The re-arm uses the same CAS (generation and
revision) and the same row locks as the other recovery writers, so an
old worker's late write cannot overwrite the new generation.
- **No new refusal codes.** No saved run identity changes. The re-arm
writes only recovery-action evidence.

## Verification

- Red (before the fix), `npx vitest run
src/services/workspace-recovery-stop-proof.test.ts`: 3 failed, 1 passed.
The staging-shape test failed because the worker did not wake
(`nextWorkspaceRecoveryAt` returned `null`) and the hold stayed. The
backoff and fallback tests failed for the same discovery gap.
- Green (after the fix): 4 of 4 pass.
- Staging shape: a parent and 3 children that share one remote
workspace. One child's save is exhausted with `agent_stop_unconfirmed`,
and its run ended days ago with a stop receipt. Before recovery, the
hold refuses the parent (`remoteWorkspaceHoldRefusesStart` is `true`).
The recovery worker discovers the hold, re-arms it, proves the stop,
saves the files and completes. The workspace is not dirty, it has a new
snapshot, and the hold is `free` for the parent. The action stays
system-owned and its text asks no person to act.
- A live controller lease keeps the hold. No re-arm happens and no lease
is resumed.
- A re-arm that is not yet due is not discovered. The worker's next wake
is in the future, and the text says that Paperclip tries again
automatically.
- After 4 re-arms, the hold continues from the last save. The unsaved
files are captured as a task attachment, and the hold is `free`.
- A live controller lease defers the worker's next wake to the lease
expiry, and the hold is not selected before it.
- A spent fallback is not selected, and a later lease change (a late
stop receipt) makes it selected again; the worker then settles it.
- `system-recovery-state.test.ts`: the automatic-retry text appears only
while a re-arm remains.
- Related suites, all pass: `src/services/workspace-recovery*`,
`src/services/remote-workspace*`,
`src/services/system-recovery-state.test.ts`,
`src/services/task-retry.test.ts`,
`src/__tests__/heartbeat-observed-workspace-preservation.test.ts` (34
files, 1510 tests), plus the stop-cleanup suites
(`environment-retained-orphan-sandbox`, `live-recovery-work`,
`retained-native-session-cleanup-work`,
`legacy-workspace-restore-recovery`, native workspace warm-reuse,
cancelled preservation, reconstruction and failed recovery; 11 files,
622 tests).
- `pnpm verify`: exit 2 ("broader verification required", because
`doc/execution-semantics.md` is a global CI input). The focused suites
above cover the change, and required CI supplies the broad coverage.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.

## Risks

- A re-armed generation resumes the stopped sandbox to save its files.
This is the same physical path as a manual Retry. It runs only when the
source run is provably over.
- Re-arms are bounded (4) and spaced, so a stop that can never be proven
does not loop. After that, the existing retained-source fallback settles
the hold. If the retained files cannot be read, the fallback records
why. This is the same behavior as for a refused source.
- The parent's workspace receives the child's saved files, because the
child (bound by an older release) wrote that workspace. This is the
existing #19860 behavior: the files that a legacy child left in the
shared workspace stay there. #20314 moves the stranded legacy children
to their own workspace.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser read-only API checks) and
extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 11:44:15 -06:00
DottaandPaperclip ff2d463eed feat(cli): manage agent grants and keep remote login on the public origin (#20323) 
## Thinking Path

> - Paperclip lets people control their AI agents and permissions.
> - Remote operators need a browser-approved CLI login.
> - Proxies can send that login's approval link to an internal host.
> - Human member commands cannot grant an agent an explicit permission
such as commentary:view.
> - This change keeps approval on the canonical public origin and adds
Board-only single-grant commands.
> - The API checks company access and permission-management authority
and records each change.
> - An operator can sign in, grant one agent permission, and preserve
the agent's other grants.

## Linked Issues or Issue Description

**What happened?**

A proxied CLI login returned an approval URL on the upstream host. The
signed-in browser was on the public host. The CLI also had no command to
add an arbitrary explicit agent grant: member commands manage human
members, while agent permissions:update exposes fixed configuration
fields.

**Expected behavior**

Approval uses the configured canonical public origin. A Board operator
with users:manage_permissions can grant or revoke one explicit agent
permission without replacing other grants or activating an inactive
membership.

**Steps to reproduce**

Configure a public auth origin. Create a CLI challenge through a proxy
that replaces forwarded-host headers. Inspect the approval URL. Then try
to grant commentary:view to an agent through the CLI.

## What Changed

- Use the existing public-origin resolver for CLI approval URLs.
- Add agent permission:grant and permission:revoke commands.
- Add PATCH /api/agents/{id}/permission-grants with strict shared
validation and Board-only OpenAPI metadata.
- Serialize each mutation on the agent row. Preserve unrelated grants
and membership state. Commit the grant and its activity record together
and publish activity after commit.
- Document remote login and single-grant management.

## Verification

- The new approval-link regression failed before the fix and passes
after it.
- All 25 CLI auth route tests pass.
- All 72 affected CLI, login and OpenAPI tests pass.
- All 5 PostgreSQL grant tests pass, including actual commentary reads
(403 before grant, 200 after grant, 403 after revoke), preservation,
audit attribution, concurrent additions, invalid inputs, inactive or
missing membership, agent self-grants, cross-company denial, and
rollback on audit failure.
- Shared and CLI typechecks and server declaration-layer checks pass.
- pnpm verify requests broad CI coverage because doc/CLI.md is a global
verification input. Greptile reviewed final head
338c58fb24 at 5/5 with no findings. All 58
non-skipped checks pass on that head. CI run:
https://github.com/paperclipai/paperclip/actions/runs/42663640985. Live
target acceptance is in progress.

## Risks

This adds an explicit permission-management API. It requires Board
authentication, company access, and users:manage_permissions.
Foreign-company agents return 404. Granting a key intentionally makes
that key unscoped in its company; revocation removes that key. Other
keys and agent configuration remain unchanged. Inactive memberships and
unapproved or terminated agents cannot gain grants. No schema change is
needed. Revert this change to remove the new management surface;
existing grants retain their normal authorization rules.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code editing, shell tools,
and browser testing. The runtime does not expose a more specific model
ID or context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 11:35:52 -06:00
Dotta 1b7fe33432 fix(workspaces): preserve import verification across process restarts (#20311) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Online import preserves old local files before the web server can
use a disposable filesystem.
> - A parent import must authenticate saved child archives before it can
omit their directories.
> - The operation journal preserves import identity, but completed child
checks existed only in process memory.
> - A container restart therefore repeated downloads, decryption and
restoration that had already succeeded.
> - This pull request saves a versioned verification receipt on each
immutable child snapshot.
> - A replacement process can reuse those checks while it reacquires
locks and verifies current local files.

## Linked Issues or Issue Description

Refs #19854 for verification reuse within a live source fence. Refs
#19873 for cohort import. Refs #20298 for the separate remote-hold
admission fix. Searched related workspace verification and restart PRs;
none provides durable child verification.

**What happened?**

Replacing a server container interrupts a long online import. The
original operation and snapshot survive, but a later invocation
downloads, decrypts and restores completed child archives again. The
current cache ends with the source-lock owner.

**Expected behavior**

Completed immutable checks survive process and scratch loss. Resume
keeps the same operation and snapshot, reuses only correctly bound
verification, and publishes once. Current local files and Git state must
still be checked under fresh source locks.

**Steps to reproduce**

1. Import two nested child workspaces, then start the parent import.
2. Kill the importer after a child verification finishes. Remove its
scratch directory.
3. Resume in a new process. Before this fix, the completed child is
downloaded and restored again.

**Paperclip version or commit**

The regression fails on the previous migration verifier. The final
branch includes master at `189fefb99183953be37f0d1084275fbb0f35d6e2`.

**Deployment mode**

Managed Linux server with PostgreSQL and explicit S3 storage
configuration.

## What Changed

- Add optional opaque storage namespace identity. S3 binds it to the
explicit endpoint, region, bucket, normalized prefix and addressing
style. Implicit endpoints retain full verification.
- Save a versioned child-snapshot receipt only after canonical crypto,
manifest and file restoration succeed and a second HEAD confirms the
same strong ETag and size.
- Bind reuse to the complete snapshot reference, key material, child
lineage, source operation and storage namespace. Require a fresh strong
HEAD and canonical validation that the current secret provider can
unwrap the saved snapshot key.
- Keep current directory/Git capture and source-owner checks on every
invocation. Read-only verifier callers do not write checkpoints.
- Merge checkpoint metadata under a short row lock without changing
parent journal hashes, snapshot retention or update timestamps. Bound
reads and writes. Lock timeouts, statement timeouts, deadlocks and
serialization conflicts are safe cache misses. Other errors remain
fatal.
- Extend real PostgreSQL/Git/archive tests with process death, scratch
removal, resume, changed bindings and single publication.

## Verification

- Baseline RED: two import invocations read each child twice. With
durable receipts, each child is read once across both invocations.
- Real process regression passed on the final rebased head: SIGKILL the
owned importer after one checkpoint, delete all scratch, and resume in a
new Node process. It retains the same operation and snapshot, reads each
child once across both processes, and records one publication.
- 53 focused cases passed in 104.86 seconds on the final production
bytes before rebase. They cover existing nested/cohort behavior,
namespace identity, changed source/reference/key/lineage/ETag, unknown
proof versions, weak validators and bounded row contention.
- After clean rebase and stronger scratch/snapshot assertions, the real
process case passed again in 18.54 seconds. Server TypeScript, module
boundaries and diff checks passed on final head
`9da68ee7ce4a778201ef85e21a11c49eeea7bc16`. Required CI supplies
final-head broad coverage.
- The initial restricted-network dependency setup failed; unchanged
preflight passed with normal network access. Initial TypeScript failures
in test pipe assertions were corrected. No failed attempt is counted as
a pass.
- Review found a missing current-master-key check. A real restart
regression with all child checkpoints complete reproduced unsafe
publication after switching to a different valid master key. Follow-up
`7ea92c2abd9ffdb2e7dd81cecf62c8fc7a6e74a5` delegates to the canonical
snapshot-key owner before both durable and in-memory cache hits.
- The fixed process regressions passed 2/2. Ten key-authority and
existing identity/source guard cases passed. The wrong-key process now
refuses with zero additional child payload GETs, no publication, and
unchanged original journal/snapshot rows. Server typecheck, module
boundaries and diff checks passed on the follow-up.
- Review also found that transient database failures could make the
optional checkpoint cache stop a valid import. Read and save now fall
back only for SQLSTATE `55P03`, `57014`, `40P01`, and `40001`.
Connection loss, identity changes, crypto failures, programming errors
and cancellation remain fatal. Reads use a read-only transaction with
one-second lock and eight-second statement limits.
- The checkpoint database-fault regression failed two cases before the
fix. All four recoverable import cases then passed. The final selected
run passed 21 cases, including fatal-error and cancellation controls,
real row contention, both process-restart cases and key/identity guards.
Server typecheck and boundaries passed. Fault injection is limited to
the checkpoint database boundary; the importer, PostgreSQL, Git,
archives, storage and publication remain real.
- Latest master was merged without rewriting published history. Six real
import/process-restart cases passed again on final head
`8c8e4213a680ebb5c8d6864416bc3a4f271e1e76` in 38.03 seconds. [Required
CI](https://github.com/paperclipai/paperclip/actions/runs/42658248222)
finished with 54 successful checks and four intentional skips; Snyk
passed. [Fresh
review](https://github.com/paperclipai/paperclip/pull/20311#issuecomment-6745354381)
is 5/5 with all four threads addressed and resolved. Hosted verification
after normal release delivery remains pending.

## Risks

- A stale receipt could skip authentication of a different object.
Complete reference/lineage/namespace binding, fresh strong HEAD and
row-locked save validation prevent that reuse. Missing or unsupported
hints trigger full verification.
- Mutable local files can change between attempts. Their source locks
and fresh captures remain required; a checkpoint never certifies current
local bytes.
- Checkpoint persistence can contend with other snapshot owners. It has
one-second lock and eight-second statement limits; the four classified
transient SQL errors retain the normal fully verified path. Other
database errors stop the import so lost source ownership cannot be
hidden.
- No schema, archive format, public API or saved-run identity change.
Older snapshots without receipts remain readable and get full
verification. Existing data and recovery obligations stay intact.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository
tools and code execution. Context-window size is not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-11-09 11:03:48 -06:00
DottaandClaude Opus 5.5 099adb2728 fix(recovery): settle a remote workspace save that stopped without finishing (#20291) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshots.
Each save is a `workspace_operations` row with phase
`remote_workspace_snapshot`
> - The ownership registry lists that kind as in-progress work, but it
had no settle step
> - When a restart cut off a failing final save, the row stayed
`running` after its invocations had all stopped and its run had ended
> - The retained-source fallback refuses while any workspace operation
is running, so a task whose saved files could not be verified could
never continue from its last save
> - This pull request lets the ownership reconciler settle that row as
failed
> - The benefit is that the automatic fallback can continue such a task
without any user action

## Linked Issues or Issue Description

**What happened?**
A native run in a remote sandbox finished its turn. Its final workspace
save failed because the sandbox disk was full ("No space left on device"
in the snapshot copy). A graceful restart then interrupted the retry.
The run ended as `native_workspace_sync_out_retry_exhausted`. Its
`remote_workspace_snapshot` row kept `status: running` for six days,
with `saveWork: { activeIds: [], unknown: false, quiescedAt: <time> }`
and `exportCleanup: { state: "pending" }`. The retained-source fallback
checked the task and refused with `workspace_operation_running`, so the
task stayed blocked.

**Expected behavior**
A save with no active or unknown invocation, whose run is terminal, is
finished work. The reconciler must mark it failed, so the fallback can
continue from the last committed save. A save that might still be
running must stay untouched.

**Steps to reproduce**
1. Run a native task in a remote sandbox. Make its final save fail, then
restart the server during the retry, so the save's work is quiesced and
the operation stays `running`.
2. Let the task's recovery refuse with an unverifiable-source failure
(for example `allocation_ownership_ambiguous`).
3. The retained-source fallback refuses with
`workspace_operation_running` on every check, and the task never
continues.

**Paperclip version or commit**
`master` at f7c85bca18.

Refs #20282 (the other cause of the same stuck task: a false ownership
refusal).

## What Changed

- `settleAbandonedRemoteSnapshotSave`
(`recovery/workspace-finalize-attempt-settlement.ts`) settles a
`remote_workspace_snapshot` row in `running` or `pending` only when all
of these are true:
- its save work is quiesced (`activeIds` is empty and `unknown` is
false);
  - its run is terminal.
It sets `status: failed`, records `abandonedSave` provenance and
`lastFailure: remote_workspace_save_abandoned`, keeps `saveWork` and
`exportCleanup`, writes a `workspace_operation.reconciled` activity
entry, and wakes the workspace's terminal work.
- The ownership registry wires it as the settle of
`remote_snapshot_save`.
- In the same transaction it re-arms blocked fallbacks. For every open
hold whose fallback check last refused with
`workspace_operation_running` and whose source run used this save's
workspace, it clears the check and wakes the workspace-recovery queue. A
hold that had already spent its 12 checks (no next check) continues as
well.
- The settle uses the save path's own `readSaveWork`, now exported, so
it never settles save work that the save path would treat as invalid.
- Because the row now has a `lastFailure`, `retireQuiescedSaves` retires
it like any failed save when a later final save commits. A later retry
of the same save boundary reopens a failed row, as before.
- The ownership inventory row for this kind names the new settle.

## Verification

- Red before the fix, green after: `settles a remote snapshot save that
stopped without finishing after its run ended, and keeps open or live
ones` (`ownership-reconciliation-report.test.ts`). Before: `settled: 0`.
After: only the abandoned save settles. Saves with a live run, an active
invocation, or unknown save work stay `running`.
- Red before the fix, green after, end to end: `continues once the
reconciler settles a final save a restart left running after its run
ended` (`workspace-recovery-fallback.test.ts`). The fallback first
refuses with `workspace_operation_running`, and the test then spends its
check budget (12 checks, no next check), so the hold drops out of
discovery. After one ownership reconciliation pass, the save is `failed`
and the check is cleared. The worker then resolves the hold with
`retained_source_unverifiable_continued_from_last_saved`. Without the
re-arm, this test fails.
- The reconciler test also covers save work that is not quiesced
(`quiescedAt: null`), save work that is invalid (`quiescedAt: ""`), and
a `pending` save.
- `workspace-recovery-fallback.test.ts` (16 tests),
`ownership-reconciliation-report.test.ts`, and the remote workspace
lifecycle, save-concurrency, and snapshot-retention suites pass (80
tests, 1 environment-gated file skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.

## Risks

- A save in another process registers its invocation in `activeIds` in
the same row-locked transaction before it runs. The settle locks the row
and requires `activeIds` to be empty and `unknown` to be false, so it
cannot fail a save that is still working.
- A save whose process died with its invocation still in `activeIds`
stays `running`. That case needs an owner-death proof, which is out of
scope here.
- No schema or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only browser diagnostics).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 10:26:04 -06:00
DottaandClaude Opus 5.5 fd2e7482c0 fix(runner): stop a lost runnerd's provider and tools from a durable lineage journal (#20270) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A local native run uses one runner daemon (`paperclip-runnerd`).
runnerd starts the provider (for example OpenCode), and the provider
starts tool processes
> - When runnerd dies in the middle of a tool call, the provider and its
tools are re-parented. The only record of that process tree was in
runnerd's memory
> - Close then cannot prove that those processes stopped. It quarantines
the session, and the next attempt stops with
`native_session_cleanup_quarantined`: "Automatic recovery stopped /
operator recovery required"
> - Restarts and runner loss are product failures. Interrupted work must
recover by itself, and it must never get two writers in one workspace
> - This pull request makes runnerd journal its process lineage to disk,
and makes close stop and prove gone every journalled process after
runnerd is lost
> - The benefit is that a killed runnerd leads to an automatic, safe
retry within the normal attempt budget, instead of an operator stop

## Linked Issues or Issue Description

Refs #20259 (runner loss classification and the lineage sampler panic).
Refs #20260 (automatic continuation of an inert cleanup quarantine).

**What happened?**
A chat eval killed only the owned local `runnerd` process (SIGKILL)
during an OpenCode tool call
(`chat-features.runner-opencode.local.recovery.retry-exhausted`,
`recovery.explain-stall` and `recovery.source-action`). Attempt 1
recorded `native_runner_process_exited` with "Retry scheduled 1/3".
Attempt 2 stopped with `native_session_cleanup_quarantined`. The task
was blocked with no owner, and the next action was operator recovery.

**Expected behavior**
After runnerd is lost, the product stops the provider and tool processes
that runnerd started, proves them gone, and continues the run within its
retry budget. When it cannot prove that, it keeps the fail-closed
quarantine.

**Steps to reproduce**
1. Start a local native OpenCode (or Codex) run.
2. Let the agent run a tool that leaves the provider's session (for
example `nohup job &`).
3. Send SIGKILL to the `runnerd` PID only.
4. Close rejects with `runner did not durably suspend before
checkpoint`, and the next attempt fails with
`native_session_cleanup_quarantined`.

**Paperclip version or commit**
master at 4c15eea770

## What Changed

- `process_lineage.rs`: runnerd journals each supervised process's
lineage to `<runner state>/process-lineage/lineage-<runnerd
pid>-<n>.json`. The journal holds the root, every member, and every
group and session a member leads, each with its start time. The first
record is written when the provider is spawned, before any tool can run.
Later records are written by the existing 50 ms sampler only when the
record changes. A teardown that proves the lineage gone removes the
journal.
- `process_lineage/journal.rs` (new module): the on-disk format, the
write rules and the reaper `reap_journals`. The reaper refuses when
runnerd never journalled (no writer marker) or when runnerd is still
alive. It restores each journal into the same `LineageState` rules,
sends TERM and then KILL only to processes whose PID and start time
still match, and reports `verified` only when none remain. A journal
that it cannot read refuses the proof.
- Unsampled escapes: each record stores `sampled_through`, the newest
process start that its read listed. The value is refreshed at most once
a second while the membership is unchanged. The reaper refuses
(`unattributed_orphan`) when a same-user orphan (parent PID 1) that is
in no journal started after that point. Such a process can be a tool
that started after the last read, so the reaper neither signals it nor
ignores it.
- A failed journal write removes the writer marker. Removal needs no
free space. After that, no reap can verify from a record that may be
missing a member.
- `paperclip-runnerd`: durable mode enables the journal before any
provider starts. A new `reap-lineage --state-dir --runner-pid` mode
prints the report and exits 0 only when verified.
- `runnerd-codex-transport.ts`: close samples whether runnerd exited
unsuspended before close sent any signal. In that case, for a runnerd
that this transport spawned on this host, it runs the reaper from the
same executable. A verified report ends the session as a crash stop with
its durable record intact. Recovery then continues the run. Any other
result keeps the existing `NativeSessionCloseUnrecoverableError`
quarantine. Launcher-owned (remote) and adopted runners do not use this
path.
- `runnerd-codex-transport.ts`: close does not queue a durable
`runner.suspend` for a runnerd that it already knows is lost. Before,
that suspend stayed pending in the saved control-plane state. A later
runnerd on the same root then failed at startup with `terminal command
result acknowledgement changed its durable identity`. The shared
suspension barrier is unchanged for remote and adopted runners.
- `runnerd-codex-transport.ts`: a runnerd that exits during close (for
example because its provider snapshot is unreadable) is not a loss. It
keeps the quarantine.
- `runnerd-codex-transport.ts`: `#runnerHasExited` treated a
signal-killed runnerd as alive, because `exitCode` is `null` and only
`signalCode` is set. Close then spent its whole deadline trying to
suspend a dead runner. It now checks `signalCode` too.
- `fake-codex-app-server`: `--spawn-escaped-tool-job <pidfile>` starts a
tool job in its own session at turn start.
- `doc/execution-semantics.md`: describes the journal, the reaper and
the fallback.

## Verification

- Rust, `cargo test -p paperclip-runner-core --lib process_lineage` (14
pass) and `process_supervisor` (20 pass). New cases:
- `reaping_refuses_an_orphan_born_after_the_last_sample`: a tool starts
after the last read, starts a new session and exits, and leaves a
non-leader orphan job. The reap refuses and does not signal the job. It
fails without the guard.
- `a_failed_journal_write_withdraws_the_writer_marker` and
`an_unchanged_record_is_rewritten_only_to_advance_sampled_through`.
- `reaping_stops_a_dead_runners_provider_and_its_escaped_tool_job`: a
real provider and a real `setsid` tool job, with a dead writer. The reap
stops both, proves them gone and retires the journal and marker.
- `reaping_refuses_without_a_marker_or_while_the_writer_lives` and
`reaping_refuses_an_unreadable_journal`.
- `a_journalled_record_restores_the_same_lineage`: after a round trip,
the restored record still finds a re-parented job and its later child,
and ignores an unrelated process.
- TypeScript, live with the real debug runnerd: `stops a lost runnerd's
provider and escaped tool job from its lineage journal`.
- `journal=true`: SIGKILL runnerd mid-turn. The escaped tool job
outlives runnerd. Close resolves, the job is gone, and the report is
`verified`. Before the fix, close rejected with `runner did not durably
suspend before checkpoint` and the job stayed alive.
- `journal=false` (an older runnerd with no journal): close still
rejects, and the job is not signalled.
- `runner-lineage-reap.test.ts`: a report from an older runnerd, a
malformed report, or an inconsistent `verified` report never counts as
proof.
- Turn-start cost: the journal adds one single-process read and one
small file write when runnerd spawns its provider (about 1.5 ms in a
macOS debug build), once for each runnerd. Nothing is added for each
turn.
- Paid runner E2E,
`chat-features.runner-opencode.local.recovery.retry-exhausted`, on the
eval branch with this fix (runs 42637596571 and 42644400434). Before the
fix (run 42615758956), attempt 2 stopped with
`native_session_cleanup_quarantined`. Now each of the three runnerd
kills records `native_runner_process_exited`. Attempts 1 and 2 are
`retryable_failure`, and attempt 3 is `terminal_failure` with cause
`native_session_retry_exhausted` (attempt 3 of 3). The reaper ran after
the first kill and verified. The cell still fails one eval setup check.
The check waits for an *active* exhausted recovery action, but
`execution-recovery-resolution.ts` settles that action as resolved and
blocks the task (`preserve_without_replay_v1`). That is a separate
grader decision, outside this PR.

## Risks

- Medium. The change only affects a close where runnerd already exited
without suspending and without a signal from close. Every other close is
unchanged.
- The reaper signals processes by PID. Each signal re-checks the PID's
start time first, and a group or session is used only while its leader
PID still names the recorded process (the same rules as the existing
in-process teardown).
- A crash stop leaves the durable root unsuspended. The coordinator
already scheduled a retry for `native_runner_process_exited`. That retry
now runs instead of failing on the quarantine. The paid E2E shows that
attempts 2 and 3 start on the same run.
- The escape guard can refuse when an unrelated same-user process
becomes an orphan in the window of about one second after the last read.
The fallback is today's quarantine. An orphan of a subreaper other than
init is not seen by the guard.
- The journal adds at most one small write each second for each runnerd
while processes start on the host. This happens on the background
sampler, not before turn start.
- Previous-release story: no database or saved-session schema change.
The journal is a new file in the runner state directory. A runnerd from
an older release writes no journal and has no `reap-lineage` mode. Its
loss keeps today's quarantine, and the existing board Retry and
stopped-session proofs still apply.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 10:19:41 -06:00
Dotta 5c73633a8f fix(workspaces): preserve remote recovery holds during online import (#20298) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Online workspace import preserves old controller-local files while
unrelated remote work continues.
> - A task can have an old local checkout and a separate durable remote
workspace.
> - A terminal remote run can retain a file-recovery hold on that task.
> - The legacy source checker rejects remote references, so that hold
blocks independent local imports.
> - This pull request proves that the hold belongs only to the separate
remote source and fences its finalizer during import.
> - Old files can migrate while the remote recovery obligation stays
intact.

## Linked Issues or Issue Description

Refs #19873 and #19854 for the existing cohort and source-verification
paths. Searched related online migration and remote recovery PRs; no
duplicate fix was found.

**What happened?**

Online import rejects a legacy Git checkout with
`online_workspace_source_scope_unproven` when a same-task failed run has
a prepared reference to a separate durable remote workspace. The legacy
run reader rejects remote references before the hold can be classified.

**Expected behavior**

Independent local files can import under their source fences. The
separate remote workspace, dirty writer, allocation, accepted result and
recovery hold must stay intact.

**Steps to reproduce**

1. Create related local Git workspaces.
2. Record a terminal failed native run and an exhausted file-recovery
hold for a separate durable remote workspace on the same task.
3. Import the local leaf online. The old checker refuses before capture.

**Paperclip version or commit**

Reproduced on master at `052c638b519ebb15f81e4fcd1a96adf6cd3b2f10`.

**Deployment mode**

Managed server with PostgreSQL and remote Runner execution.

## What Changed

- Add a distinct remote-only preservation proof. Keep remote runs
excluded from legacy export sources.
- Use the canonical source inspector and prepared-reference matcher.
Bind the run, finalizer, action, task, accepted result, authority,
allocation and prior committed snapshot.
- Acquire the held run's finalizer fence through the existing sorted
lock path. Existing source-proof reinspection rejects changes before
publication.
- Preserve previous-release prepared references with an older baseline
snapshot. Do not require that snapshot's epoch or source run to equal
the current run.
- Add real PostgreSQL import, authenticated cold restore, preservation,
refusal, lock-contention and publication-race regressions. Document the
behavior.

## Verification

- Genuine baseline failure on unchanged product: the real importer
refuses with `online_workspace_source_scope_unproven`.
- Final focused regression file: 23 tests passed in 13.19 seconds
through the canonical isolated PostgreSQL wrapper on Node 24.22.2.
- 24 existing affected migration/recovery tests passed against the same
final production bytes. Another 83 tests were explicitly excluded by the
focused selection; no full-suite pass is claimed.
- The positive imports local bytes, confirms the remote finalizer fence
is held during upload, leaves all remote records unchanged, parks the
original repository and authenticates/restores the archive.
- Refusal and race cases cover malformed, foreign, same-local and mixed
references; active owners/writers; changed allocation, result, task,
action, authority and baseline; and finalizer contention.
- Server TypeScript check, module-boundary check and `git diff --check`
passed.
- An intermediate run had one incorrect assertion that expected an
explicit empty legacy-source array. The canonical journal omits that
empty field. The final assertion accepts absence and still forbids any
exported remote source. Final relevant results are 23 new plus 24
existing passes, rather than a claim that the earlier invocation passed
in full.
- Required CI passed on head `6a278e717db2b5f614b0a595f99bfd0fb5430227`:
55 successful checks and 4 intentional skips. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/42650025033).
Greptile reviewed that head at 5/5 with no actionable findings or open
inline/review comments. Hosted continuation remains pending normal
delivery.

## Risks

- A false disjointness proof could admit an unsafe local import. Strict
company/task/source binding, the canonical inspector, finalizer fencing
and publication revalidation retain refusal on uncertainty.
- This does not recover or settle remote files. Dirty authority and the
original recovery obligation remain unchanged.
- No schema, public API, archive format or saved-run identity changes.
Existing persisted state carries forward.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository
tools and code execution. The runtime does not expose the context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-11-09 10:13:41 -06:00
Nicky LeachandPaperclip 052c638b51 fix(environments): honor stopped-task proof after warm preparation (#20289) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A warm plugin Runner retains its task for later runs.
> - A follow-up can find that the retained task has stopped.
> - The resume path already asks settle to check status before it sends
stop.
> - A saved startup marker disabled this order, even after warm
preparation had finished.
> - This pull request applies the same stop-proof check to all
observed-stop releases.
> - The next run skips a repeated stop when status proves both execution
and required preparation have stopped.

## Linked Issues or Issue Description

Fixes: #20224
Refs #20203

## What Changed

- Remove the startup-marker exclusion from the existing status-first
path. Keep the existing shared confirmation check. It still requires
`preparationStopped: true` for startup work.
- Keep stop followed by status when the first status lacks the required
proof. Keep the startup drain, credential writeback, complete, and
cleanup failure handling.
- Add retained-task coverage with the startup metadata that warm
preparation saves. Cover a startup cleanup whose first status lacks
preparation proof.
- Update the plugin task lifecycle contract.

## Verification

- Local macOS, source base `3e8744fa91`: from `server`, `pnpm exec
vitest run src/__tests__/environment-task-runtime.test.ts
src/__tests__/plugin-warm-tasks.integration.test.ts
src/__tests__/plugin-task-credential-writeback.test.ts` passed: 3 files,
82 tests. The integration tests use real embedded PostgreSQL and
controlled plugin results.
- After restoring the fix, the two affected files passed again: 73
tests.
- Independent parent verification on commit `19f720ff68`: the two files
filtered to the changed startup-proof and retained-task cases passed (4
tests). Independent source review found no actionable issue.
- Baseline proof: with only the source condition restored to upstream,
the new retained-task test with saved startup metadata fails because it
sees an extra `stop`. The case without that metadata passes. The source
fix was then restored.
- The retained-task regression asserts exactly `environmentResumeLease`,
`status`, `complete`, and `environmentAcquireLease`, with both task
operations bound to the old task. The startup test also proves that
missing preparation proof sends `stop` and a second `status`.
- Test setup: `node scripts/agent-preflight.mjs --skip-rust-debug
--explain` installed dependencies, then stopped because Cargo is
unavailable on this Mac. Initial test collection failed because the
plugin SDK output was absent. `pnpm --filter @paperclipai/plugin-sdk
build` built that required dependency and the shared package; the
targeted test run then passed.
- Standard Greptile reviewed commit
`19f720ff686058b4dd6b34f27cdfb9c8ff60e50a` at 5/5 with no review
threads.
[Review](https://github.com/paperclipai/paperclip/pull/20289#issuecomment-6743829614).
- Full local checks were not run, as requested. [GitHub PR workflow
42644023868](https://github.com/paperclipai/paperclip/actions/runs/42644023868)
passed on final commit `19f720ff686058b4dd6b34f27cdfb9c8ff60e50a`. It
supplied broad typecheck, build, and test coverage. The separate Snyk
status passed. Optional Storybook checks were skipped.
- The change removes one provider stop call on the reported path. The
issue's measured eight calls would become seven if other calls stay the
same. The earlier five-call target is not claimed. No deployed latency
measurement is claimed.

## Risks

- A provider that cannot prove preparation stopped still gets stop and
status. That fallback can cost one extra status call, as the existing
status-first contract already allows.
- A status failure keeps cleanup pending and blocks replacement. No
termination proof or credential handoff is skipped.
- No schema, API, dependency, or timer change. No added work on
successful task reuse or ordinary task dispatch.
- Live provider acceptance remains unrun. The local evidence checks
orchestration and stop-proof handling.

## Model Used

- OpenAI Codex, GPT-6 family, with reasoning, repository inspection,
shell tools, and code execution. The exact serving model ID and context
window are not exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 07:44:19 -08:00
DottaandClaude Opus 5.5 dc9ad17515 perf(storage): time workspace snapshot save and restore steps and count storage retries (#20280) 
## Thinking Path

- Cloud durable storage is a major source of failures and slowness on
staging. For example, a staging task failed when saving files after
automatic retries.
- Before measuring or fixing it, we need to see where storage time goes.
Today a saved-workspace restore before the turn appears only inside the
coarse `workspace_environment_realize` phase.
- After the turn, the save's encrypt, put, verify and multipart throttle
backoff have no timing at all. Only `captureMs`, `transferMs`,
`verifyMs`, `uploadMs` and `publishMs` exist, in `saveTiming`.
- Storage retries happen in the SDK and in the multipart uploader,
invisibly.
- This adds the measurement first, so the storage-mode chaos waves
(MinIO behind a fault proxy) can report save and restore p50/p95 and
point at the dominant wait.

## Linked Issues or Issue Description

**What happened?**
Durable-storage saves and restores were opaque in the run log and save
timing. A slow or repeatedly retried storage call could not be told
apart from slow capture, encryption or sandbox work.

**Steps to reproduce**
1. Run a remote task with S3-compatible durable storage, so it saves
after the turn and restores on the next run.
2. Inspect the run log and the `remote_workspace_snapshot` operation's
`saveTiming`. The restore's download, decrypt and verify, and the save's
put, verify and throttle backoff, have no timing.

**Expected behavior**
Each storage step has a duration, and storage request retries and
backoff are counted.

## What Changed

- `server/src/storage/request-stats.ts` (new): an async-local collector
of storage request statistics. Counts only:
  - `requests`, `attempts` and `failedRequests`;
  - `retryDelayMs` (SDK waits);
- `parts`, `throttled` and `backoffMs` (the multipart uploader's waits).
- `server/src/storage/s3-failure.ts`: `withS3FailureDiagnostics`, which
wraps every S3 send, records each request's SDK `$metadata.attempts` and
`totalRetryDelay`, on success and on failure.
- `server/src/storage/s3-multipart.ts`: records each uploaded part and
each resend wait, flagged throttled when the backend returned 503, 429
or SlowDown.
- `server/src/services/remote-workspace-snapshot-storage.ts`:
  - Each step is a `storage.snapshot.*` run phase:
    - save: `prior_attempts`, `encrypt`, `digest`, `put`, `verify`;
    - restore: `download`, `decrypt`, `verify_plaintext`, `publish`.
- A caller can pass a `timing` record that receives the step durations,
the bytes and the request statistics.
- `server/src/services/remote-workspaces.ts`:
  - The save passes a timing record, kept in `saveTiming.storage`.
- The sandbox unpack during a restore is its own phase,
`workspace.snapshot.sandbox_restore`.
- `execution-preparation-timing.ts`: the new phase names.
- `doc/run-log-events.md`: a new "Durable storage timing" section.

Because the restore runs inside run preparation, its steps of 50 ms or
more become `run.performance.span` rows with no extra plumbing. This is
run-log and instance-database data only; there's no Telemetry change.

## Verification

- `server`: `storage-s3-multipart-backoff`, `storage-s3-multipart`,
`storage-s3-multipart-throttle`, `storage-s3-failure-diagnostics`,
`remote-workspace-snapshot-storage` and `execution-preparation-timing`
tests: 86 passed. They include:
- collected stats for a throttled multipart upload: 12 requests, 7
failed, 7 throttled, 35.5 s backoff and 3 parts;
  - a save and restore that fill every step duration and the byte count.
- `remote-workspaces.integration.test.ts`: 112 passed. Its `saveTiming`
assertion now includes `storage`.
- Server typecheck: no errors in the touched files.

## Risks

- **Overhead:** one async-local lookup per S3 request, plus timestamps
per snapshot step.
- **Collector scope:** the collector only exists inside a snapshot save
or restore. Other storage callers are unaffected; they record nothing.
- **`saveTiming`** gains a `storage` object. Readers that compare it
exactly (one test, updated) will see the new field.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 09:41:53 -06:00
DottaandClaude Opus 5.5 ac0d0374f2 perf(issue-detail): measure task opens on real instances (#20276) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People open a task from their issues list to see what an agent is
doing and to reply
> - On a busy staging instance, the composer and the latest messages
take 1.1–2.0 s to appear after a click. For one task with a slow run
log, they took 13.9 s
> - Local fixtures do not reproduce this. They do not have real network
round trips, real server contention, or real object-store latency
> - This pull request adds the measurement we need: a browser harness
that times a task open on a real instance, and a Server-Timing entry on
every API response that separates server time from network time
> - It also records the staging baseline and the improvement plan
> - The benefit is that each later performance change can be measured on
real data with the same tool, before and after deploy

## Linked Issues or Issue Description

**Problem or motivation**
Opening a task from the issues list is slow on instances with real data.
The composer and the latest chat messages can take seconds to appear. We
could not measure this on staging in a repeatable way. We also could not
tell server time from network and queueing time for most API requests.

**Proposed solution**
Add a reusable browser harness for task opens, and add a `paperclip_api`
Server-Timing entry to every API response.

**Alternatives considered**
Local Playwright fixtures only. They do not reproduce real network round
trips, server contention or object-store latency, so they did not show
the delays that people see. Per-endpoint Server-Timing only. It does not
cover the many other requests that run during a task open.

**Roadmap alignment**
This is measurement for existing task-loading behaviour. It does not add
product features. The staging baseline and the measured causes are in
`doc/plans/2026-11-09-task-loading-performance.md`. Follow-up pull
requests change the loading behaviour itself, starting with #20277.

## What Changed

- Added `server/src/middleware/api-server-timing.ts`. Every `/api`
response now has a `paperclip_api;dur=<ms>` Server-Timing entry. The
value is the server time from request arrival until headers are written,
including admission waits. The entry is appended, so the existing
`paperclip_issue` phases stay first.
- Registered the middleware first in `server/src/app.ts`.
- Added `tests/perf/issue-detail/staging-click-harness.js`. It clicks a
task row and records, frame by frame, when the composer and the latest
message become visible, layout shifts, movement of the latest message,
and the requests that run before the page is ready. The summary labels
each sample instead of using the task reference. It replaces query
values and every path segment that contains a digit. A fetch that is
still running when the sample ends is listed as pending.
- Documented the harness in `tests/perf/issue-detail/README.md`.
- Added the dated plan
`doc/plans/2026-11-09-task-loading-performance.md` with the staging
baseline, the causes, and the prioritized slices.

## Verification

- `cd server && npx vitest run src/__tests__/api-server-timing.test.ts`.
3 tests pass. They cover the JSON responses, the time spent in earlier
middleware, preserving route phases, and explicit `writeHead` calls.
- `tests/perf/issue-detail/issue-detail.perf.spec.ts` reads the first
`dur=` value. That value is still `paperclip_issue`.
- The harness was used on staging to capture the baseline in the plan
document.

## Risks

- Low risk. The header has only a duration, with no request, user or
company data.
- The middleware wraps `res.writeHead` once for each request. This is
the same pattern that `on-headers` uses.
- Server-Timing is visible to same-origin scripts only. No
`Timing-Allow-Origin` header is added.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, browser automation, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 09:06:37 -06:00
DottaandClaude Opus 5.5 1004e55180 fix(recovery): stop stall wakes from failing setup on a held workspace (#20273) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Recovery tells a waiting parent or dependent when a child or blocker
stalls on the board (`issue_blocker_stalled`)
> - A legacy child that an older release bound to its parent's remote
workspace failed its save, and its save recovery was exhausted. Its
ended run still holds the shared workspace, days past the one-hour
holder cutoff
> - Workspace reservation refuses a run on such a hold and fails setup
with "The workspace still has an unfinished writer or save operation."
Every stall wake of the parent created a run that could only fail this
way
> - The stall sweep treated each failed run as an undelivered wake and
retried it every ten minutes, with no backoff and no limit. On staging
this made 72 failed runs (58 of 60 failures since a deploy), in pairs
every ten minutes
> - This pull request makes the stall sweep hold a wake while the
waiter's workspace hold refuses a run, and bounds retries of a wake
whose run fails
> - The benefit is that a held workspace no longer produces a failed run
every ten minutes, and the wake still arrives once the hold settles

## Linked Issues or Issue Description

**What happened?**
A parent task was woken by `issue_blocker_stalled` twice every ten
minutes. Each run failed in setup (`setup_failed`,
`executionRecovery.kind=bootstrap`, `providerWorkStarted=false`) with
"The workspace still has an unfinished writer or save operation." The
parent's remote workspace was held by a child's ended writer run whose
save recovery was exhausted
(`native_workspace_sync_out_retry_exhausted`, last failure
`agent_stop_unconfirmed`). The child shared the parent's workspace
because an older release created it before subtasks got their own
workspace (#19860).

**Expected behavior**
A wake that cannot start because its workspace is held must not create a
failed run every time. It should wait until the hold settles. Automatic
retries must be bounded.

**Root cause**
- `server/src/services/heartbeat/workspace-preparation.ts` rethrows
`RemoteWorkspaceWriterActiveError` as a setup failure when the ended
writer is silent past `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS`
(documented, intended).
- `server/src/services/recovery/blocker-stall-wakeups.ts` retried every
non-delivered wake row (including `failed`) after
`REFUSED_WAKE_RETRY_MS` (10 minutes) with no backoff and no limit, and
it did not check whether the waiter could start at all.

## What Changed

- `remote-workspace-writer-hold.ts`: add
`remoteWorkspaceHoldRefusesStart`, the one rule for a hold that
reservation refuses outright (unattributed writer, or a writer silent
past the cutoff with no live owner fence).
- `workspace-preparation.ts`: use that rule instead of an inline copy,
so reservation and wake senders cannot drift.
- `blocker-stall-wakeups.ts`: before waking a waiter, check for proof
that its run would be refused. The proof is that the assignee's own
latest run on the task failed setup with the unfinished-writer error,
and the bound workspace hold still refuses a run. Then do not wake it;
count it as deferred and re-check at most hourly. A different assignee
(for example a controller-only agent that never uses the workspace), a
later run, or a settled hold lets the wake through. Reservation already
tried and failed the unchanged-writer release for that run, so the sweep
does not repeat that proof.
- `blocker-stall-wakeups.ts`: count failed runs that carry the stall
(the keyed wake's run and keyless resource-wait retries that keep its
context). Retry after 10, then 20 minutes, and stop after the third
failed run. Refusals (wake-on-demand off, agent paused) keep the
10-minute retry. A new stall (new recovery action) wakes again.
- `remote-workspace-writer-hold.ts` / `remote-workspaces.ts`: share the
unfinished-writer error text as one constant.
- `doc/execution-semantics.md`: document both rules.

## Verification

- Red before the fix
(`server/src/__tests__/issue-blocker-stall-wakeups.test.ts`): 3 new
tests failed. The held-workspace test got a delivered wake; the backoff
test got a third wake 11 minutes after the second failure.
- Green after the fix: `npx vitest run
src/__tests__/issue-blocker-stall-wakeups.test.ts` passes 22/22. Added
after review: a waiter with no refused run, or a reassigned waiter, is
still woken; failed resource-wait retries count toward the limit.
- The new tests use the staging shape: a legacy `shared_workspace`
remote workspace owned by the parent, a child bound to it, the child's
failed native writer run ended four days ago with
`native_workspace_sync_out_retry_exhausted`, a dirty authority, and the
child's board-owned exhausted recovery action. The parent's own earlier
run failed setup on the hold, as on staging. They also prove that a
writer inside the cutoff still gets its wake (the run parks in a
workspace wait), and that the wake is delivered once the hold settles.
- Related suites pass: `heartbeat-remote-workspace-busy.test.ts`,
`heartbeat-dependency-scheduling.test.ts`,
`native-cancelled-workspace-preservation.test.ts`,
`src/modules/run-dispatch`.
- `npx tsc --noEmit -p server` passes.
- `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global
CI input; required CI supplies the broad coverage.

## Risks

- Low. No schema or persisted-shape change. Saved state from the
previous release (existing wake rows, existing workspace authority) is
read as-is: earlier `failed` stall wake rows count toward the new
three-run limit, which stops the current loop at once after deploy.
- A parent whose workspace stays held is not woken for the stall until
the hold settles. The stall is still listed on its page and the child's
recovery action stays with the board.
- Not fixed here: the exhausted save hold itself is never settled
automatically (`agent_stop_unconfirmed` exhaustion in workspace
recovery), and legacy children that still share a parent's workspace are
not moved. Both touch workspace recovery code that is under concurrent
change and are left for a follow-up.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser read-only inspection of
staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 08:28:21 -06:00
DottaandClaude Opus 5.5 2dd7a98753 fix(recovery): return a settled restore hold's warm sandbox to reuse (#20278) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Recovery tells a waiting parent or dependent when a child or blocker
stalls on the board (`issue_blocker_stalled`)
> - A legacy child that an older release bound to its parent's remote
workspace failed its save, and its save recovery was exhausted. Its
ended run still holds the shared workspace, days past the one-hour
holder cutoff
> - Workspace reservation refuses a run on such a hold and fails setup
with "The workspace still has an unfinished writer or save operation."
Every stall wake of the parent created a run that could only fail this
way
> - The stall sweep treated each failed run as an undelivered wake and
retried it every ten minutes, with no backoff and no limit. On staging
this made 72 failed runs (58 of 60 failures since a deploy), in pairs
every ten minutes
> - This pull request makes the stall sweep hold a wake while the
waiter's workspace hold refuses a run, and bounds retries of a wake
whose run fails
> - The benefit is that a held workspace no longer produces a failed run
every ten minutes, and the wake still arrives once the hold settles

## Linked Issues or Issue Description

**What happened?**
A parent task was woken by `issue_blocker_stalled` twice every ten
minutes. Each run failed in setup (`setup_failed`,
`executionRecovery.kind=bootstrap`, `providerWorkStarted=false`) with
"The workspace still has an unfinished writer or save operation." The
parent's remote workspace was held by a child's ended writer run whose
save recovery was exhausted
(`native_workspace_sync_out_retry_exhausted`, last failure
`agent_stop_unconfirmed`). The child shared the parent's workspace
because an older release created it before subtasks got their own
workspace (#19860).

**Expected behavior**
A wake that cannot start because its workspace is held must not create a
failed run every time. It should wait until the hold settles. Automatic
retries must be bounded.

**Root cause**
- `server/src/services/heartbeat/workspace-preparation.ts` rethrows
`RemoteWorkspaceWriterActiveError` as a setup failure when the ended
writer is silent past `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS`
(documented, intended).
- `server/src/services/recovery/blocker-stall-wakeups.ts` retried every
non-delivered wake row (including `failed`) after
`REFUSED_WAKE_RETRY_MS` (10 minutes) with no backoff and no limit, and
it did not check whether the waiter could start at all.

## What Changed

- `remote-workspace-writer-hold.ts`: add
`remoteWorkspaceHoldRefusesStart`, the one rule for a hold that
reservation refuses outright (unattributed writer, or a writer silent
past the cutoff with no live owner fence).
- `workspace-preparation.ts`: use that rule instead of an inline copy,
so reservation and wake senders cannot drift.
- `blocker-stall-wakeups.ts`: before waking a waiter, read its bound
remote workspace hold. When the hold refuses a run, do not wake it;
count it as deferred and re-check at most hourly. A workspace with an
unchanged-release baseline is left to reservation, which can release it.
- `blocker-stall-wakeups.ts`: a stall wake whose run failed is retried
after 10, then 20 minutes, and not after the third failed run. Refusals
(wake-on-demand off, agent paused) keep the 10-minute retry. A new stall
(new recovery action) wakes again.
- `doc/execution-semantics.md`: document both rules.

## Verification

- Red before the fix
(`server/src/__tests__/issue-blocker-stall-wakeups.test.ts`): 3 new
tests failed. The held-workspace test got a delivered wake; the backoff
test got a third wake 11 minutes after the second failure.
- Green after the fix: `npx vitest run
src/__tests__/issue-blocker-stall-wakeups.test.ts` passes 20/20.
- The new tests use the staging shape: a legacy `shared_workspace`
remote workspace owned by the parent, a child bound to it, the child's
failed native writer run ended four days ago with
`native_workspace_sync_out_retry_exhausted`, a dirty authority, and the
child's board-owned exhausted recovery action. They also prove that a
writer inside the cutoff still gets its wake (the run parks in a
workspace wait), and that the wake is delivered once the hold settles.
- Related suites pass: `heartbeat-remote-workspace-busy.test.ts`,
`heartbeat-dependency-scheduling.test.ts`,
`native-cancelled-workspace-preservation.test.ts`,
`src/modules/run-dispatch`.
- `npx tsc --noEmit -p server` passes.
- `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global
CI input; required CI supplies the broad coverage.

## Risks

- Low. No schema or persisted-shape change. Saved state from the
previous release (existing wake rows, existing workspace authority) is
read as-is: earlier `failed` stall wake rows count toward the new
three-run limit, which stops the current loop at once after deploy.
- A parent whose workspace stays held is not woken for the stall until
the hold settles. The stall is still listed on its page and the child's
recovery action stays with the board.
- Not fixed here: the exhausted save hold itself is never settled
automatically (`agent_stop_unconfirmed` exhaustion in workspace
recovery), and legacy children that still share a parent's workspace are
not moved. Both touch workspace recovery code that is under concurrent
change and are left for a follow-up.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser read-only inspection of
staging).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 08:28:04 -06:00
DottaandClaude Opus 5.5 1b09920a64 fix(issues): refuse agent status changes that detach another run's live execution (#20271) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that an agent run executes holds a checkout lock and an
execution lock (`checkoutRunId`, `executionRunId`)
> - A status change away from `in_progress` clears these locks. It does
not stop the run that holds them
> - Agents cannot stop another agent's run: run cancel and the task-tree
pause are board-only
> - But an agent that can edit a task (for example a lead with the
manage-active-checkouts grant over its report) can change its status.
The run then keeps running and is no longer bound to its task
> - In chat evals, a lead "paused" a launch tree. It moved running
subtasks to backlog. The runs kept running, detached
> - This pull request refuses an agent's status change when a different
queued or running run holds the task's lock. The denial is final, links
the task, and names the person's controls
> - The benefit is that agents cannot interrupt another agent's run
through a status edit, and runs are not left orphaned

## Linked Issues or Issue Description

Refs #20261 (chat guidance sends a task-tree pause to the person's Pause
subtree control). Refs #19394 (resume restarts the work a subtree pause
interrupted).

**What happened**

An agent changed the status of a task while another agent's run executed
it. The request succeeded. The server cleared `checkoutRunId` and
`executionRunId`. The run continued, but it was no longer bound to the
task. In the chat evals `steering.stop-versus-pause` and
`task-coordination.pause-tree`, a lead moved running subtasks to backlog
to "pause" a launch. Their runs kept running, detached.

**Expected behavior**

An agent cannot get the effect of interrupting another agent's run. When
a different live run holds the task's lock, the agent's status change is
refused before any write. The refusal says it is final. It sends the
person to the task and names the controls that really stop or hold the
work: the run's Stop or Interrupt, or More task actions, then Pause
subtree on the parent task. The lock-holding run, board users, and paths
that stop the run themselves continue to work.

**Steps to reproduce**

1. Create agent A and agent B, with B reporting to A (A gets
`tasks:manage_active_checkouts` through the manager chain).
2. Give B an `in_progress` task. Set `checkoutRunId` and
`executionRunId` to a `running` heartbeat run of B.
3. As A (agent JWT with its own run), send `PATCH /api/issues/:id` with
`{ "status": "backlog" }`.
4. Before this change: `200`, the status is `backlog`, both lock fields
are `null`, and B's run is still `running`.

## What Changed

- `server/src/routes/issues.ts`: new
`assertAgentStatusChangeKeepsLiveRunBound`. For an agent actor, it
refuses a status change when `executionRunId` or `checkoutRunId` names a
different run in the same company that is `queued` or `running`. It runs
in `PATCH /issues/:id` after the final status and assignee are known
(execution-policy transitions included) and before any run stop or
write. It also runs in recovery resolution when the source status
changes.
- The guard applies to any status other than `in_progress`, also to the
current status, because the service clears the locks for those too.
- The write is conditional on the lock the guard checked. The route
re-reads the lock and passes it to `svc.update` as
`expectedExecutionLock`. The service compares it under the row lock and
returns `409 issue_execution_lock_changed` if a checkout landed in
between.
- These cases stay allowed: the lock-holding run itself, board users,
requests with no status, a change to `in_progress`, `cancelled` (the
route cancels the run), reassignment (the route stops the run first),
`done` through PATCH while a Runner goal is active (terminalization
pauses the goal and stops the run; recovery resolution does not get this
exception), and a lock holder that already finished.
- `packages/shared/src/issue-write-denial.ts`: new denial code
`issue_status_change_live_run` (409). The copy says the refusal is
final, links the task as `[identifier](/prefix/issues/identifier)`, and
names Stop or Interrupt and More task actions, then Pause subtree. A new
optional `issuePath` context field carries the link. The UI notice
renders it through the existing generic path.
- `doc/SPEC-implementation.md`: documents the rule in the checkout
contract section.
- Paths checked with no change needed: there is no bulk issue update
route; tree holds and run cancel are board-only; `POST
/issues/:id/release` already lets only the assignee's checkout run
release; API-catalog actions call the same PATCH route; MCP status
changes already refuse with `MCP_ACTIVE_EXECUTION`.

## Verification

- Red on master, green with this change:
- `server/src/__tests__/issue-stale-execution-lock-routes.test.ts`
(embedded Postgres, real issue service): a manager moves its report's
running task to backlog. On master: `200`, status `backlog`,
`checkoutRunId`/`executionRunId` changed to `null`. With the fix: `409
issue_status_change_live_run`, row unchanged, run still `running`.
- `server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts`:
refusals for `backlog`, `todo`, `blocked`, `in_review`, `done`, a queued
lock holder, and a repeated current status (red on master). Allowed
cases: field edits, the lock-holding run, board, `cancelled` (run is
cancelled), and finished lock holders; an allowed change carries
`expectedExecutionLock`.
- Same embedded-Postgres file: `svc.update` with an
`expectedExecutionLock` that differs from the row returns `409
issue_execution_lock_changed` and leaves the row unchanged.
- Also passed locally: `issue-closed-workspace-routes`,
`issue-comment-reopen-routes`, `issue-recovery-actions`,
`issue-scheduled-retry-routes`, `low-trust-red-team-routes`,
`issue-update-comment-wakeup-routes`, `external-object-routes`,
`packages/shared` `issue-write-denial`, `ui` `IssueWriteDenialNotice`.
- `tsc --noEmit` shows no errors in the changed files. Full typecheck,
build and test are left to CI.

## Risks

- Behavior change for agents: a lead or CEO that used status edits to
take over a report's running task now gets a 409. The supported paths
(comment, cancel, reassign, or the person's Stop or Pause subtree) still
work.
- A queued run that holds the lock also blocks the change. This is
deliberate: the queued run would otherwise start later with no task
bound to it.
- Agents can still cancel another agent's run by cancelling its task.
That is existing, documented behavior ("task cancellation cancels the
intent"), and this PR does not change it.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 08:21:40 -06:00
DottaandPaperclip a3a25ce6c9 fix(workspaces): stop running Paperclip's worktree setup before every agent turn (#20269) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before an agent turn starts, the harness prepares the task's git
worktree
> - The harness runs a repository's `scripts/provision-worktree.sh` when
the project sets no `provisionCommand` and the file exists
> - The harness runs it on every turn, both when it creates the worktree
and when it reuses it
> - In the Paperclip repository, that script writes the worktree's
isolated instance config and runs `pnpm install`
> - Its "install needed" check compares against every `node_modules`
directory in the base checkout, including other worktrees, so it
reinstalls on every turn
> - This pull request renames the Paperclip script so that the harness
default does not find it in this repository
> - The benefit is that Paperclip agent turns no longer wait on a 6–9
second install. Other repositories keep the harness feature.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Time to first token for agent turns in managed worktrees of the
Paperclip repository.

**Subsystem affected**
Execution workspace provisioning
(`server/src/services/workspace-runtime.ts`) and the repository script
`scripts/provision-worktree.sh`.

**Current behavior**
Every agent turn in a Paperclip worktree first runs `bash
./scripts/provision-worktree.sh`. In a local test, a `pnpm install
--frozen-lockfile` with nothing to change took 6.1 s and 6.3 s, and a
full install from a warm store took 9.0 s. The script's fingerprint skip
does not prevent this, because the script also requires every
base-checkout `node_modules` path (up to depth 4) to exist in the
worktree. On a developer machine that list had 1,735 entries, including
other worktrees under `.claude/worktrees/` and `.worktrees/` and pnpm's
internal `node_modules/.pnpm/*/node_modules`. A managed worktree never
contains the other worktrees' paths, so the script always decides to
reinstall.

**Proposed behavior**
The Paperclip repository does not ship a file at
`scripts/provision-worktree.sh`. The setup script becomes
`scripts/setup-paperclip-worktree.sh`. People can run it by hand or set
it as the project `provisionCommand` to opt in. The harness default
stays for other repositories.

**Reason and benefit**
Agents install dependencies when they need them (`node
scripts/agent-preflight.mjs`, as `AGENTS.md` says). Many turns never run
code. Instance setup stays available on demand:
`scripts/provision-worktree-runtime.sh` runs the setup script when a dev
server starts in a worktree that has no config. Database seeding was
already deferred.

**Breaking changes**
A project that explicitly sets `provisionCommand` to `bash
./scripts/provision-worktree.sh` for this repository will fail with
"file not found". Change it to `bash
./scripts/setup-paperclip-worktree.sh` or clear it.

## What Changed

- Rename `scripts/provision-worktree.sh` to
`scripts/setup-paperclip-worktree.sh`. The content is unchanged.
- `scripts/provision-worktree-runtime.sh` calls the new name when the
worktree has no config.
- Tests read the real script from its new path
(`provision-worktree-self-heal`, `workspace-runtime`,
`worktree-seed-server-spawn`). Server fixtures still copy it into their
test repositories as `scripts/provision-worktree.sh`, so the harness
default stays covered.
- `doc/DEVELOPING.md` says that the Paperclip repository deliberately
has no `provision-worktree.sh`, and tells people how to opt in.

## Verification

- `node --test scripts/__tests__/provision-worktree-self-heal.test.mjs`:
33 passed, 1 skipped (the skip already existed).
- `cd server && npx vitest run src/__tests__/workspace-runtime.test.ts
src/__tests__/heartbeat-workspace-source-trust.test.ts
src/__tests__/execution-workspace-reopen.test.ts`: 242 passed on the
rebased head.
- `cd server && npx vitest run
src/__tests__/worktree-seed-server-spawn.test.ts`: 2 passed.
- Manual: in a Paperclip worktree, start an agent turn. The run log does
not show a `workspace_provision` operation for the script.

## Risks

- Explicit `provisionCommand` settings that name the old path will fail.
See "Breaking changes".
- Open pull requests that edit `scripts/provision-worktree.sh` (for
example #16213, #11335, #11733, #19764) will conflict on the rename. Git
rename detection should carry most edits over.
- An agent that runs tests in a new worktree without first running
preflight will see a missing-module error. `AGENTS.md` already tells
agents to run `node scripts/agent-preflight.mjs`.
- The script's install check still has the bug described above. It now
runs only on demand. A separate fix can address it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits). Standard context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-09 14:03:24 +00:00
DottaandClaude Opus 5.5 2b461b2f58 fix(native): require publication for files named under a Deliverable label (#20231) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agents finish a task with a completion report, and a server
gate checks that requested files are published to the task
> - The gate finds a file request only from a creation verb, such as
"Write report.pdf"
> - Delegated briefs often list the output under a "Deliverable:" label
instead, and the gate did not know `.html` or `.css` names
> - So a worker can finish a website task with `index.html` only in its
own workspace, and the person cannot open it
> - This pull request makes a file named under a Deliverable label a
requested file output
> - The benefit is that the worker gets the existing "use
register_deliverable" feedback before the task closes

## Linked Issues or Issue Description

No public issue exists. Bug description:

**What happened**
A person asked the CEO: "Make me a simple website introducing our
business." The CEO delegated a task. Its brief said "Deliverable: - A
self-contained static website in this task's workspace: index.html plus
a stylesheet (for example styles.css)". The worker wrote `index.html`
and `styles.css` in its workspace. It did not call
`register_deliverable`. It cited `index.html` and `styles.css` as
completion evidence, and the gate accepted the report. The task was
marked done with no attachment. The CEO then told the person that the
files were "in the task workspace".

**Expected behavior**
A brief that names a file as its deliverable is a file request. The
completion gate must reject a workspace-only file and tell the worker to
publish it with `register_deliverable`.

**Steps to reproduce**
1. Call `explicitlyRequestsFileOutput` with the brief above.
2. See that it returns `false`, so `validateNativeDeliverableEvidence`
accepts the bare `index.html` evidence ref.

## What Changed

- `explicitlyRequestsFileOutput` now also returns true when a
`Deliverable:` / `Deliverables` label line (plain, bold or heading), or
a bullet or numbered item in the list below it (blank lines between
items allowed), names a file. Relative paths such as `dist/index.html`
count.
- Under the label, web page files (`.html`, `.htm`, `.css`) count with
the existing file types.
- Items that name an input ("a summary of input.csv", "a comparison of
input.csv and baseline.csv"), describe a repository change, pull
request, commit or branch, forbid the named file (`do not create
report.pdf`), name a file to edit in place (`update styles.css`) without
asking to publish it, send the file inline (`index.html and styles.css
inline in the reply`), or forbid creating files, do not count. An inline
or no-upload instruction about another output in the same item
(`index.html and a short summary in the reply`, `index.html; do not
upload it externally`) does not cancel the named file. The
repository-change check ignores file names, so `branch-report.pdf` still
counts.
- The sentence-based detector is unchanged.
- `doc/AGENT-ARTIFACTS.md` describes the label rule.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-deliverable-feedback.test.ts
src/services/native-runtime/native-runner-file-handoff.test.ts` (197
pass), then after the review fixes
`native-deliverable-feedback.test.ts`,
`native-completion-format-feedback.test.ts` and
`chat-attachment-reuse.test.ts` (211 pass). The 13 first-round review
cases fail on the original head.
- New cases: the real delegated brief, inline labels, heading lists,
numbered lists and bold labels require publication. Unlabelled text,
inline-chat deliverables, input references, repository changes, text
after the list ends, and "Deliverable quality ..." prose do not.
- The chat eval `onboarding.first-work` exercises this path end to end
on its candidate branch.

## Risks

- Low risk. The change only adds a true result for labelled file
outputs; an inline-only or no-file label item stays false, as the
sentence detector already decided.
- A labelled brief that names a file which is not a repository change
now needs `register_deliverable` or an accessible work product. That is
the documented delivery rule.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 07:15:11 -06:00
DottaandClaude Opus 5.5 4c15eea770 fix(accounting): a run's unproven accounting never holds the company (#20254) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, each lost provider receipt leaves an accounting
debt that settles later
> - In restart-chaos G1 (seed 9802, task t02), six debts each waited the
full 10-minute closure bound, and the board's accepted Retry waited with
them
> - The last debt settled as an unpriced charge. Under the blocking
unpriced-usage policy, that charge hard-stopped the whole company, which
cancelled the Retry, and the task showed "budget"
> - Dotta's plan decision 1 says accounting is never a company-wide
gate: admission reads per-run exposure and real hard stops only
> - This pull request makes those debts settle at once when the stop is
proven, records accounting-settled charges without a company hold, and
labels the remaining accounting wait correctly
> - The benefit is that a restart no longer stalls a company's work for
10 minutes or stops it outright

## Linked Issues or Issue Description

Refs #20250 (Phase 2: a run that only inherited its predecessor's runner
identity settles at once; this PR builds on it), #19944, #19826, #19857

**What happened?**
Restart-chaos wave w-20261109T111954-s9802 (G1), task t02, master
c5da0b88ad. After a K-fault restart at 11:27, six `run_accounting_debt`
records were "unprovable". Each settled exactly 10 minutes after its run
finished: bf9ea6b4 and c2e222a2 at 11:30, 097d0036 at 11:35, f64f9f46
and aa2c9f03 at 11:36, and 0b4b404f at 11:37:23.
- Four runs were `cancelled` (issue_terminalized), with
`native.local_process_stopped` as their last process event and no
recorded pid. The receipt-closed rule accepted that stop proof only for
`failed` runs.
- 0b4b404f failed with `native_session_cleanup_quarantined` before it
started any process. It journaled no receipt, but it carried pid 11356
of the previous run's warm runner, so the never-started rule did not
apply.
- At 11:37:23, 0b4b404f settled as an unpriced unknown-billing charge
with zero tokens. With `unpricedUsagePolicy: block`, the company hit a
hard stop at 11:37:24 (24 of 3000 cents). The pause cancelled the
board's Retry (751751ea, accepted at 11:28:37), and the task showed
`userAction: "budget"`.

**Expected behavior**
Accounting debts whose provider stop is proven settle at once.
Accounting never holds or pauses the whole company. A wait caused by
unpriced usage is labeled as accounting, not budget.

**Steps to reproduce**
Restart the control plane while native runs are being cancelled, with
the company's unpriced-usage policy set to block. Before this change,
the debts wait 10 minutes and then hard-stop the company.

## What Changed

- `run-cost-accounting.ts`: a verified local process stop closes the
receipt source for `cancelled`, `timed_out` and `interrupted` runs as
well as `failed` ones. The inherited-pid run (0b4b404f) is covered by
#20250, so this PR does not change the never-started rule.
- `budget-exposure.ts`: a final charge that accounting settled without
its receipt (receipt unavailable, closure timed out, capture failed, or
lost on restore) never trips the blocking unpriced-usage policy.
Unpriced usage from a complete receipt still does. A capture failure
stays exempt even if a later report adds usage, because its receipt is
never certified complete. This removes the restart-derived special case,
because the rule now covers every accounting-settled charge.
- `task-interruption-presentation.ts`: a `budget.blocked` wait caused by
the unpriced-usage refusal is shown as `accounting.unpriced_usage`, with
accounting wording, for both a held Retry and a never-started task. Only
the fixed refusal phrase is matched; the stored message is never shown.
- `doc/SPEC-implementation.md`: the admission and accounting rules are
updated to match.

## Verification

- `closed-native-receipt-accounting.test.ts`: new tests for cancelled,
timed-out and interrupted runs with a verified stop; they were red
before the fix. When a timed-out closure settles under the blocking
policy, the board's saved Retry is promoted (due at once) and never
cancelled. That assertion was red on master, where the hard stop
cancelled it. Seven tests that asserted the old rule (an
accounting-settled charge keeps the blocking policy) now assert that the
charge is recorded and the company stays active and admits work. All 61
pass.
- `task-interruption-query.test.ts` (Postgres): a Retry held by the
unpriced refusal, and a skipped wake, are labeled
`accounting.unpriced_usage`. A budget hard stop keeps `budget.blocked`.
The test was red before the fix.
- Also passing: `run-cost-accounting`, `cost-data-quality`,
`idle-accounting-work`, `native-usage-delivery.integration`,
`task-interruption-presentation`, and the accounting and budget tests in
`heartbeat-process-recovery`. Server typecheck has no errors in the
changed files. The only local errors are in
`native-session-executor.ts`, from a stale local runner build.
- Not run: the G1 chaos rerun. Phase 3 runs it after merge.

## Risks

- Behavior change: a charge that accounting settled without a receipt no
longer hard-stops a company under the blocking policy. That includes a
remote run whose closure evidence never arrived, which may still have
spent. The charge stays in Costs with provenance and an activity entry,
and the spend still counts toward the budget limit. This follows plan
decision 1.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 07:02:47 -06:00
DottaandClaude Opus 5.5 8525bb534c fix(chat): post a turn's answer when a card it did not create is pending (#20220) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People talk to agents in Agent Chat. The agent's final message
becomes the chat reply.
> - A chat thread can hold a pending card that the board addressed to
another person, for example a human-only confirmation for Riley.
> - When Alex asks the agent to act on that card, the agent correctly
gets `403 interaction_human_only` and writes a final answer that
explains the denial.
> - The shared answered rule (`conversationTurnAnswered()`, from #20179
and #20204) counted any pending human-only card in the thread as the
turn's own response. So the answer was suppressed, and the chat stayed
`active` with no reply.
> - This pull request limits that rule to cards that the turn's own run
created. Other cards still hold the task status.
> - The benefit is that the person always gets the turn's answer, unless
the turn itself asked a question with a card. Live completion and
restart recovery stay consistent.

## Linked Issues or Issue Description

Refs #20179, #20204 (the shared answered rule this PR corrects).

**What happened**

In an Agent Chat, the board created a pending `request_confirmation`
card with `resolverPolicy: human_only`, addressed to another user
(Riley). The card has no source run. Alex wrote "I approve Riley's
request." The agent tried to accept the card and got `403
interaction_human_only`. It then wrote a correct final answer: the card
needs Riley. The run succeeded with a `yielded` result and a
`response_wake` continuation. Run completion recorded
`presentationDecision: none`. No comment was created. The conversation
stayed `active`, so Alex never got a reply.

Cause: `conversationTurnAnswered()` reused the status-gate query
`pendingNativeInteraction()`. In conversation mode, that query excludes
another run's card only when it is an ordinary question or an `anyone`
confirmation. A human-only card addressed to someone else is kept. The
rule then treated that card as the card that "presents the turn",
although this run did not create it.

**Expected behavior**

A successful chat turn posts its final answer exactly once, and the
conversation moves to `waiting`. Only a pending question or confirmation
card that the turn's own run created presents the turn in place of its
prose. A card from an earlier turn, or one the board addressed to
another person, can hold the task status, but it does not suppress the
answer.

**Steps to reproduce**

1. Open an Agent Chat as Alex.
2. As the board, create a pending `request_confirmation` card in the
chat with `resolverPolicy: human_only` and `addresseeUserId` set to
another person.
3. Send "I approve Riley's request."
4. Let the agent try to accept the card, get `403
interaction_human_only`, and end its turn with a final answer.
5. See that no reply appears, `presentationDecision.commentAction` is
`none`, and `conversationState` stays `active`.

## What Changed

- `server/src/services/native-runtime/native-governance.ts`:
`conversationTurnAnswered()` now checks only pending cards whose
`sourceRunId` is the turn's run. `pendingNativeInteraction()` gets an
optional `createdByRun` filter for this. The status gate
(`pendingNativeGovernance()`) does not change, so another person's card
still holds the task.
- Both callers get the fix through the one shared rule
`conversationTurnPresentsAnswer()`: live run completion
(`run-completion.ts`) and background presentation recovery
(`recoverCommittedNativeChatResponse`).
- `server/src/services/heartbeat/run-completion.test.ts`: new live cases
"someone else's confirmation card" (board-created, human-only, addressed
to another user, no source run) and "earlier turn's question card"
(created by an earlier run of the same agent). Each case now also checks
that the conversation moves to `waiting`.
-
`server/src/services/native-runtime/agent-chat-response-recovery.test.ts`:
new restart case for the board-created card, plus a `waiting` check
after recovery.
- `doc/execution-semantics.md`: state that only the turn's own card
presents the turn.

## Verification

- Thesis: the reply is suppressed because the answered rule counts any
pending human-only card in the thread, not only this run's card. If this
is true, a turn with only a board-created card addressed to someone else
posts no comment on master, and restricting the rule to `sourceRunId =
runId` posts exactly one comment and settles the conversation to
`waiting`, with the own-card cases unchanged.
- Failing first: without the fix, the two new live cases failed
(`expected [] to deeply equal [answer]`). The new recovery case also
failed (no comment).
- After the fix:
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts -t "posts a waiting
conversation turn"`: 9 passed (includes the two own-card cases, which
still post nothing).
- `cd server && npx vitest run
src/services/native-runtime/agent-chat-response-recovery.test.ts`: 62
passed.
- Both suites use a real embedded PostgreSQL database.
- Turn start: no change. The rule runs only after the provider turn
ends.

## Risks

- Low. Only the presentation rule changes. The status gate, the
conversation settle rule (it already required `sourceRunId = run.id`)
and the external-chat path do not change.
- Behavior shift: a turn now posts its answer while another run's or the
board's card is pending. Before, the answer was lost. A card that the
turn itself created still presents the turn, as before.
- Previous-release state: no persisted shape change. A run that the
previous release already recorded as `none` keeps that decision.
Recovery does not re-present it.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell,
file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 03:22:01 -06:00
DottaandClaude Opus 5.5 6fd46c0025 perf(budgets): judge Paperclip's share of restart recovery; report the provider's session load (#20183) 
## Thinking Path

- Speed budgets protect the time to first visible agent activity, and
restart recovery is one of the judged metrics
(`native-speed-budgets.ts`, wave grader).
- In the B1 concurrency-6 chaos wave (`w-20261109T045320-s9807`), Codex
`restart_recovery` was p50 8.6 s and p95 11.2 s, against a budget of 6.5
s and 8.0 s. At concurrency 4 (K13) it was 3.8 s and 5.0 s.
- The preparation spans on a copy of the kept database break it down:
- Paperclip's share (ready → claim → prepare → runtime stage → runner
launch) is a steady 1.2–2.0 s.
- `provider.session.load`, the Codex app-server boot plus
`thread/resume` of long saved sessions, takes 3.0–6.3 s, 65–75% of each
recovery.
- Resumes run concurrently, and the worktree lock and the startup
reconciler are not on the path.
- Judging the full latency therefore mixes our code with provider and
session-size effects. Budget the part we control, so a regression in our
code shows up regardless of session size or concurrency, and report the
rest.

## Linked Issues or Issue Description

**What happened?**
`restart_recovery` breached its budget whenever many long provider
sessions resumed at once, although Paperclip's own recovery work stayed
within about 2 s. The breach did not point at anything Paperclip could
fix, and a real regression in our code could hide inside the
provider-load variance.

**Steps to reproduce**
1. Run a local chaos wave at concurrency 6 with long-running Codex tasks
and K1/K2/K4 faults (for example B1, seed 9807).
2. Compare `restart_recovery` in `speed.json` with
`provider.session.load` in the recovered runs' run logs.

**Expected behavior**
The budget judges Paperclip's share of a recovery. The provider's
session load and the full latency are reported but not judged.

## What Changed

- `server/src/services/native-runtime/native-speed-budgets.ts`:
- New budgeted metric `restart_recovery_control_plane`: from `ready` to
the start of `provider.session.load`, or to the next provider event when
nothing reloads (for example a reattached runner).
- Its budget is 2.5 s / 4.0 s for claude, codex and opencode. Measured:
1.2–2.0 s per run at concurrency 6.
  - The full `restart_recovery` stays in the table for reporting.
  - Breach payloads carry `restartTotalMs` and `providerLoadMs`.
- `server/src/services/native-runtime/native-run-trace.ts`:
- The per-run check records when `provider.session.load` starts, keeps
the recovery measurement, and judges it when the run finishes.
- It judges after the deferred diagnostics drain, so a provider span
committed after the provider event still counts.
  - It emits `restart_recovery_control_plane` breaches only.
- `tests/restart-chaos/ttft-baseline-stats.ts`: each recovery sample
gains `controlPlaneMs` and `providerLoadMs`. The provider-load start is
the span event's write time minus its duration.
- `tests/restart-chaos/wave-speed.ts`:
- Three restart rows per adapter: `restart_recovery` (report-only),
`restart_recovery_control_plane` (budgeted) and `restart_provider_load`
(report-only).
  - All three have outage time removed.
- `tests/restart-chaos/ttft-baseline.ts`: the recovery table adds a
"Paperclip share" column.
- Docs: `doc/run-log-events.md` (metric and payload) and the
speed-budget table in
`doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`.

## Verification

- `server`: `npx vitest run
src/services/native-runtime/native-speed-budgets.test.ts`, 18 passed.
Two new tests:
  - a recovery dominated by provider load does not breach;
- a slow Paperclip share breaches with `restartTotalMs` and
`providerLoadMs`, even when the provider span is committed after the
provider event.

The existing recovery tests moved to the new metric. All six were red
before the trace change.
- `tests/restart-chaos`: `npx vitest run --config
tests/restart-chaos/vitest.config.ts wave-speed.test.ts
ttft-baseline-stats.test.ts`, 29 passed. New tests cover the sample
split and the grader rows (a judged share, a report-only provider load,
zero breaches).
- Server typecheck: no errors in the touched files.

## Risks

- **Lower sensitivity to provider-side slowdowns.** A provider
regression in session load no longer breaches. It stays visible in the
report-only rows and in breach payloads.
- **Runs with no `provider.session.load` span** (reattach, or an adapter
that doesn't emit it) are judged on the full latency against the
control-plane budget. That's conservative.
- **Run-log consumers that filter on `metric: "restart_recovery"`** see
`restart_recovery_control_plane` from now on. The doc notes the change
date.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 01:22:28 -06:00
DottaandClaude Opus 5.5 ad2304dc88 fix(chat): recover a held conversation turn's answer after a restart (#20204) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People talk to agents in Agent Chat. The agent's final message
becomes the chat reply.
> - A chat turn can end while its task stays held: a pending approval,
an execution stage or a dependency blocker keeps the status.
> - #20179 made live run completion post that held turn's answer. It
added `conversationTurnAnswered()`.
> - Background presentation recovery still required a
`conversation_turn_finished` decision and a `done` result.
> - If the server restarts between the status commit and the live
presentation, recovery skips the held turn. The person never gets an
answer.
> - Restarts are product failures. A committed answer must not be lost.
> - This pull request makes recovery use the same answered rule as the
live path, through one shared helper.
> - The benefit is that a held turn's answer reaches the chat exactly
once, with or without a restart.

## Linked Issues or Issue Description

Refs #20179 (live fix; this PR closes the "Known gap" in its Risks
section).

**What happened**

In an Agent Chat, the agent resubmitted a hire approval and ended its
turn with a yielded result. The finalizer committed
`governed_response_waiting` and moved the task to `in_review`. The
server restarted before run completion posted the reply. Background
presentation recovery found the run as a candidate, but declined it,
because it accepted only `conversation_turn_finished` with a `done`
result. No reply was posted. The run stayed a pending presentation
candidate.

**Expected behavior**

After a restart, recovery posts the held turn's final answer exactly
once. It never posts a second copy when the live path already presented
the answer. When the turn's own question or confirmation card presents
it, recovery posts nothing and records that decision, as the live path
does.

**Steps to reproduce**

1. Open an Agent Chat. Link a pending hire approval to the chat task.
2. Let the agent end the turn with a yielded result and a
`response_wake` continuation. The status decision commits as
`governed_response_waiting`.
3. Stop the server after the status commit and before run completion
presents the reply.
4. Start the server. Presentation recovery runs.
5. See that the chat shows no reply.

## What Changed

- `server/src/services/native-runtime/native-governance.ts`: add
`conversationTurnPresentsAnswer()`. It is the one rule for whether a
successful conversation turn's prose is its answer:
`conversation_turn_finished`, or a committed decision with no pending
card of the turn (`conversationTurnAnswered()`).
- `server/src/services/heartbeat/run-completion.ts`: the live path calls
the shared helper. No change in behavior.
- `server/src/services/native-runtime/native-run-finalizer.ts`
(`recoverCommittedNativeChatResponse`):
- A current conversation turn no longer needs
`conversation_turn_finished`. The turn must still be the unsettled
current turn: same agent, user, session generation, `conversationState:
active`, and no other execution run. A `conversation_turn_finished` turn
still needs `in_progress`.
- A conversation turn with a `yielded` result is accepted next to
`done`.
  - The resolver gets `conversationTurnFinished` from the shared helper.
- When the turn's own card presents it, recovery records
`presentationDecision: none` (reason
`committed_conversation_interaction_presented`) and settles the
conversation turn, like the live path. The run then leaves the pending
presentation set. Before, such a run stayed a candidate.
-
`server/src/services/native-runtime/native-response-presentation-query.ts`:
the candidate prefilter matches the same turns: any committed reason for
an active conversation turn, and `yielded` conversation results.
-
`server/src/services/native-runtime/agent-chat-response-recovery.test.ts`:
restart regressions with the real finalizer and a real database.
- `doc/execution-semantics.md`: state that recovery applies the same
rule.

## Verification

- Failing first. With only the new tests applied, 5 of 7 new cases
failed: pending approval, revision-requested approval, execution stage
(all yielded), pending approval with `done`, and the own-card case. The
run was a candidate, but recovery posted nothing and did not settle it.
- After the fix: `cd server && npx vitest run
src/services/native-runtime/agent-chat-response-recovery.test.ts
src/services/heartbeat/run-completion.test.ts
src/__tests__/idle-native-finalization.test.ts
src/services/presentation-ownership-read.test.ts
src/services/chat-monitor-continuation.test.ts
src/services/native-runtime/native-preserved-monitor-wait.test.ts
src/services/native-runtime/native-response-publication-retry.test.ts
src/services/heartbeat/run-lifecycle.test.ts
src/__tests__/scoped-deadline-work.test.ts
src/services/native-runtime/paperclip-control-plane-port.test.ts
src/__tests__/heartbeat-run-summary.test.ts
src/__tests__/agent-conversations.test.ts`: 12 files, 627 tests passed.
- The new cases check:
- Held turns: the run is a candidate, the check posts the exact final
answer once, the presentation decision is recorded, the run leaves the
candidate set, and a second recovery returns false with one comment.
  - Live already presented: no candidate, no second comment.
- Own question card: no comment, `presentationDecision: none`, the card
stays pending, the conversation settles to `waiting`, and the run leaves
the candidate set.
- Previous-release live decision `none` (`yielded_control_plane_wait`):
kept, not re-presented.
- `tsc --noEmit -p server`: no errors in changed files. Four unrelated
errors remain in this local worktree (`esbuild` not linked in
`native-harness-prestage.test.ts`, and a linked-package type mismatch in
`run-dispatch/adapters/postgres.ts`).
- Turn start: no change. The helper runs only in run completion (after
the provider turn ends) and in background presentation recovery. Nothing
new runs before `turn.start`, and no files move. The live path keeps the
same queries: it short-circuits on `conversation_turn_finished`, and
otherwise runs the same card read as before.

## Risks

- Low to medium. Recovery can now post the answer of a conversation turn
whose task is held. This matches the live path since #20179.
- A card-presented turn found by recovery is now settled with
`presentationDecision: none`. Before, it stayed a pending candidate and
kept the presentation worker (and idle-sleep checks) busy.
- A `conversation_turn_finished` turn keeps the `in_progress` check.
Held turns accept any status their gate committed. The unsettled-turn
checks (active conversation state, session generation, execution run)
still block stale or superseded turns.
- Previous-release state: no persisted shape changes. A run committed by
the previous release with no saved presentation decision (the restart
window) is recovered by the new code. The test fixture uses the
unchanged finalizer, so it saves exactly the previous release's shape. A
run whose live path already recorded a decision keeps it. A held turn
that the previous release recorded as `yielded_control_plane_wait` is
not re-presented, so no late duplicate or stale reply appears.
- Not changed: generic stranded-work recovery in `recovery/service.ts`
still skips only `conversation_turn_finished` runs during the
publication window.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell,
file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 01:08:10 -06:00
Nicky LeachandClaude Opus 5.5 990de1e0bf perf(environments): release a stopped plugin task once (#20203) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugin environment drivers with `supportsTasks` run agents as
provider tasks, and warm runs keep a parked task for the next run
> - A follow-up run can find that its parked task already stopped. Core
then retires that task before it acquires a new one
> - Core sent `stop`, `status`, and `complete` in the settle step. Then
the plugin release hook sent `stop`, `status`, and `complete` again for
the same task. Every call was on the critical path of the new run
> - This pull request removes the repeated calls and keeps every call
that does work no other call does
> - The benefit is a faster start for a run whose parked task already
stopped

## Linked Issues or Issue Description

No issue exists. This follows the enhancement template.

**What existing behavior does this improve?**

The release of a plugin task lease that has already stopped. This occurs
when a follow-up run finds that its parked warm task stopped.

**Subsystem affected**

Server: plugin environment drivers and the plugin Runner task lifecycle.

**Current behavior**

Core sends `stop`, `status`, and `complete` in the settle step. Then the
plugin release hook sends `stop`, `status`, and `complete` again for the
same task. Core also sends `stop` after resume already saw that the task
stopped. On a hosted test deployment, 8 of 40 follow-up runs took this
path. From the resume to the new acquire, these runs spent P50 1014 ms
and P90 1080 ms. These 8 runs are the full overall P90 (1275 ms).

**Proposed behavior**

Core does not call the release hook when settle confirmed the stop and
completed the same task. After a resume with no lease, Core sends
`status` first, and sends `stop` only when `status` does not confirm the
stop.

**Reason and benefit**

The repeated calls do no new work. Removing them saves about 355 ms,
plus the `stop` share of about 187 ms, for each such run. These numbers
are modeled from trace timings.

**Breaking changes**

None for the plugin API. A task driver must release task-scoped
resources in its `status` or `complete` handling. The doc now says this.

**Additional context**

The settle `status` stays. Providers release task storage there, and it
records the stop that gates the credential writeback and `complete`.

## What Changed

- `settleDispatchedEnvironmentTask` now returns `"completed"` when it
confirmed the stop and completed the task in this call, `"stopped"` when
the lease was already settled, and `false` otherwise.
- The plugin driver release does not call `environmentReleaseLease` when
settle returned `"completed"`. In all other cases it calls the hook as
before.
- When `environmentResumeLease` returns no provider lease, the release
sends `status` first. It sends `stop` (and a second `status`) only when
that `status` does not confirm `executionStopped: true`. Startup and
startup-cleanup leases keep the old `stop` first order.
- The cleanup sweep (`retryPendingSandboxTeardown`) and
`destroyRunLease` do not change. A settle failure still leaves the lease
in `pending_cleanup`, and the sweep runs `stop`, `status`, `complete`,
and the release hook.
- `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` tells drivers about the new
order and that task-scoped resources must be released in `status` or
`complete`.

### Dispatch count for a stopped parked task (resume to new acquire)

| | Calls |
| --- | --- |
| Before | resume, stop, status, (credentials), complete, release hook
(stop, status, complete), acquire: 9 provider calls (10 with Codex
writeback) |
| After | resume, status, (credentials), complete, acquire: 4 provider
calls (5 with Codex writeback) |

The modeled saving is about 540 ms for each such run: about 355 ms for
the repeated release, and about 187 ms for the settle `stop` and
`status`. This pull request keeps the settle `status`, so the real
saving is about 355 ms plus the `stop` share of the 187 ms. These
numbers are modeled from trace timings. They are not measured on this
branch.

### What covers each removed call

- Release hook `stop`, `status`, `complete`: settle already sent `stop`
and `status` and checked `executionStopped: true` for the same task id
(`lease.providerLeaseId`), then sent `complete`. The skip uses settle's
success as its condition, not the code path.
- Settle `stop` after a resume with no lease: the next `status` must
report `executionStopped: true`. If it does not, Core sends `stop` and
`status` as before.

### What each kept call protects

- Settle `status`: providers release task storage and cancel
preparations here. It also writes `runnerTaskExecutionStopped`, which
the credential writeback and the `complete` bookkeeping need.
- Credential writeback: Codex can rotate its token. The writeback must
land before the new task installs credentials. It still runs inside
`complete`, before the new acquire.
- Settle `complete`: it retires every run lease that owned the task and
clears the dispatch marker.
- Release hook when settle did not complete a dispatched task in this
call: the provider may still hold the lease.

## Verification

- `pnpm --filter @paperclipai/shared --filter @paperclipai/plugin-sdk
build`
- `pnpm --filter @paperclipai/server typecheck`
- `npx vitest run src/__tests__/plugin-warm-tasks.integration.test.ts
src/__tests__/plugin-task-credential-writeback.test.ts
src/__tests__/environment-task-runtime.test.ts`
- Related suites: `environment-runtime`,
`plugin-runner-task.integration`, `task-creation-recovery.integration`,
`heartbeat-plugin-environment`, `native-startup-cleanup-recovery`,
`plugin-environment-driver-ready-recovery`,
`environment-task-authority.integration`, `native-workspace-warm-reuse`,
`remote-workspaces.integration`, `legacy-workspace-restore-recovery`,
`plugin-task-source-recovery.integration`,
`managed-plugin-environments`.

Locally, all of these pass except `heartbeat-plugin-environment`
"inherits the instance default environment across companies…". That test
also fails locally with the `origin/master` service files, so this
change does not cause it.

The existing `plugin-runner-task.integration` same-run replacement test
now expects no release hook after a completed settle.

New tests:

- The stopped parked path sends exactly resume, `status`, `complete`,
acquire. The `status` goes to the old task.
- A resume with no lease for a task that still runs sends `status`,
`stop`, `status`, `complete`.
- A failure at `status`, `complete`, or `stop` leaves the lease in
`pending_cleanup`, and the sweep then sends `stop`, `status`,
`complete`, and the release hook.
- With an observed stop, the Codex writeback runs before `complete`. A
writeback failure leaves the lease for cleanup, and the retry writes
back and completes.
- A startup-cleanup lease still sends `stop` first.

## Risks

- A third-party task driver that releases resources only in
`onEnvironmentReleaseLease` would not get that call after a completed
settle. The doc now states the requirement. The cleanup sweep still
calls the hook after any failure.
- A driver that returns no lease from resume while the task still runs
costs one extra `status`. The behavior stays correct.
- No migration. No API change for plugins.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-08 23:00:49 -08:00
Devin FoleyandPaperclip 9b71d03215 fix: recover interrupted native preparations and unblock upgrades (#20177) 
## Thinking Path

> - Paperclip manages AI-agent companies and their work.
> - A native run prepares its workspace before a runner PID or
checkpoint exists.
> - A server crash can interrupt that stage and leave its controller on
another host.
> - Recovery needs a durable owner proof, a delivered recovery wake, and
the saved execution contract.
> - The old paths missed controller lease registration and boot proof,
skipped checkpoint-free preparations, and counted retained work as live
upgrade blockers.
> - This PR connects preparation to the existing controller lease and
recovery paths, and separates retained work from the upgrade census.
> - The result is bounded recovery without a second executor or loss of
saved work.

## Linked Issues or Issue Description

**What happened?**

A native preparation interrupted by a server crash could remain running
indefinitely with no runner PID or checkpoint. Native controller stamps
did not wake the shared instance lease. That lease was disabled on
volume-backed Railway services, and native takeover omitted the boot
binding needed for its cross-host proof. Scoped recovery selected
checkpoint-backed runs but skipped these preparations. The upgrade probe
then counted the retained row as current execution and could block every
later upgrade.

**Expected behavior**

Track the preparation's controller before a runner starts. Recover it
through the normal classifier after proving its owner stopped or fencing
that boot. Keep live or unprovable owners held. Deliver recovery wakes
to preparations without checkpoints. Preserve unresolved historical work
and report it separately from current execution during upgrade
observations.

**Steps to reproduce**

1. Save an observed native preparation with no runner process, result,
provider-start evidence or checkpoint.
2. Replace its controller with a process in another PID space. Let its
owner lease expire and send a scoped recovery wake.
3. Before this fix, recovery skips the checkpoint-free preparation or
cannot prove its foreign controller stopped. Upgrade readiness continues
to count that old row as a live blocker.

Related work: #18037, #19651, #19868, and #20168. The foreign-PID guard
from #19868 remains. The queued first-turn input fix from #20168 is
included through the current base.

## What Changed

- Register the existing instance lease when native execution and restart
claims are written. Keep passive identity reads free of ownership
effects. Keep that lease enabled on volume-backed Railway services
unless explicitly disabled.
- Preserve the existing PID/start-time proof within the same readable
PID space. Bind foreign or unknown-space takeover to its saved boot and
last write. A live lease or grace period keeps the run held with a
scheduled check. Repeated evidence waits preserve their timestamp.
- Include explicitly fenced, checkpoint-free preparations in the
existing finite recovery candidate inventory. Normal launch, checkpoint,
provider evidence and admission checks still decide whether work may
resume.
- Expose the existing board stop-confirmation operation for historical
foreign-host owners. Bind confirmation to the exact run, boot, PID,
process start and host. Reject changed ownership.
- Report narrowly defined old, unowned preparations as
`retainedBlockedRuns` outside the live upgrade census. Local executions,
pending admission, drain state, truncation and positive checkpoint proof
retain their barriers.
- Update the execution contract and add PostgreSQL, real-route, race and
preservation regressions.

## Verification

- Genuine red/green regressions cover the old upgrade census, missing
native lease registration, the volume-backed default, foreign-host
confirmation, and scoped recovery skipping checkpoint-free preparation.
Test bodies ran against real PostgreSQL before their corresponding
production changes.
- Combined verification before the final idle correction (head
`055be6c84d9`): **459 passed, 22 skipped, 11 files passed**. Covers
controller lease/identity, native restart classification and
integration, host confirmation, heartbeat recovery, instance route
authorization, upgrade readiness, and queued first-turn input. The
focused command used `pnpm exec vitest run` with those 11 suites and
`--maxWorkers=1 --no-file-parallelism`. The 22 skipped cases require the
Rust debug fake-Codex fixture, which was not built locally; the
real-process CI coverage must pass.
- `pnpm --filter @paperclipai/server typecheck`: passed on head
`055be6c84d9b058064daad0af71bb5d893ab9529`.
- Actual route proof uses the real heartbeat service and PostgreSQL.
Unauthorized access returns 403. The retained run and owner records
remain unchanged. Recovery integration admits one bootstrap claim across
two replacement controllers, keeps a live owner held, and preserves
local execution.
- `pnpm verify`: exit 2, requests broad CI because the execution
contract changed. Full local suite not run. Current-head CI must supply
broad coverage.
- Local source checkout on macOS ARM64, Node v25.6.1, pnpm 9.15.4. Tests
use isolated PostgreSQL. No live provider restart is claimed by those
fixtures.
- CI found an idle-query regression in the first version of the lease
wiring. Two focused PostgreSQL regressions reproduced the misplaced
identity-read notification and the missing claim notification. The fix
moves registration to actual ownership writes. `env -u npm_execpath pnpm
exec vitest run server/src/__tests__/maintenance-server-routes.test.ts
--maxWorkers=1 --no-file-parallelism`: **17 passed**, including 90 real
seconds and two qualified hourly windows with no extra idle queries. An
initial local setup attempt inherited npm metadata and did not run
tests; clearing that metadata selected the correct pnpm helper. Three
focused controller/native suites also passed **168 tests, 22
fixture-gated skips**. Server typecheck passed after the correction on
head `4a527326dcd5b3a9e286f0e4c3e761d3056af4d9`.
- The older CI server shard also exposed a same-space graceful-restart
regression. The new boot fallback had overridden local process-exit
proof. Seven suspension cases and a focused unit regression reproduced
it; the fix limits boot fallback to foreign or unknown PID spaces. Fresh
verification on `acf1065d98150210c1e227c7f3b05a1eb489730b`: **251
passed, 22 fixture-gated skips, 4 files passed**, including all **40**
graceful-restart suspension tests. Server typecheck passed. The local
regression failed before this correction, and the same-space and
foreign-space paths pass together after it.
- Current-head CI completed successfully on
`acf1065d98150210c1e227c7f3b05a1eb489730b` (run 42568109784). Greptile
reviewed that exact head at 5/5 with no unresolved review threads.
- Secret and private-data scans passed. No workflow, lockfile or
migration changes.

## Risks

- The instance lease now applies to volume-backed Railway services. Its
existing 30-second renewal, quiescence, grace and self-fence behavior
applies while work is held. Idle servers retain the existing no-query
behavior. An explicit `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=off` remains
available.
- A controller lease proves only the server owner's state. It does not
prove an agent process stopped or authorize replay, resource release or
volume removal. The existing classifier and provider/runner checks
remain responsible for those decisions.
- Previous-release records keep their shape and saved state. A missing
boot lease stays unprovable and requires the scoped stop-confirmation
path. A changed confirmation target is rejected.
- The upgrade observation does not recover a held task by itself. Source
tests do not qualify a hosted replacement. Deployment and the affected
task's useful result must be verified separately.
- This repairs recovery after a crash; it does not claim to identify or
fix every source of heap exhaustion.

## Model Used

OpenAI `gpt-6.1-sol`, xhigh reasoning effort, through Codex in T3 Code,
with repository, shell, and GitHub tools. Context window size is not
exposed by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-08 21:53:13 -08:00
DottaandClaude Opus 5.5 95bf90bd2c fix(chat): post a conversation turn's answer while an approval holds the task (#20179) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People talk to agents in chat. The agent's final message becomes the
chat reply.
> - A chat turn can end while the task stays held. For example, the
agent resubmits a hire approval and the approval is pending.
> - The finalizer keeps the task in review for the approval
(`governed_response_waiting`). That decision is not
`conversation_turn_finished`.
> - Run completion then treats the yielded turn as a control-plane wait.
It records `presentationDecision: none` and posts no comment.
> - The person never sees the agent's answer. The chat shows nothing
after the request.
> - This pull request counts a committed conversation turn as answered
unless its own pending question or confirmation card presents it.
> - The benefit is that the person always gets the reply, whatever state
the task keeps.

## Linked Issues or Issue Description

Refs #19971 (post the full final reply). That fix covers the reply text.
This fix covers replies that were never posted.

**What happened**

In a chat, the person asked the CEO agent to change a proposed hire to a
cheaper model and send it back for approval. The agent resubmitted the
approval correctly. It called `paperclip_finish`, and Paperclip accepted
the report with "task is still waiting for approval". The run succeeded
and the task moved to `in_review`. The run event
`run.presentation.resolved` recorded `commentAction: none` (reason
`yielded_control_plane_wait` on master). The agent's final message never
reached the chat.

**Expected behavior**

A chat turn's accepted final answer is always posted to the
conversation. An approval, an execution stage or a dependency blocker
can hold the task. It must not hide the reply. Only a question or
confirmation card that the turn itself left pending presents the turn
instead of its prose.

**Steps to reproduce**

1. Open an Agent Chat with an agent. Link a hire approval in
`revision_requested` to the chat task.
2. Ask the agent to change the hire and resubmit it.
3. The agent resubmits the approval and ends the turn with a yielded
result and a response wake.
4. See that the chat shows no reply. The run is `succeeded` and its
presentation decision is `none`.

## What Changed

- `server/src/services/native-runtime/native-governance.ts`: move the
pending-card query of `pendingNativeGovernance()` into
`pendingNativeInteraction()`, with no change in behavior. Add
`conversationTurnAnswered()`. It returns false only when a pending
thread interaction (the same predicate as status arbitration) gates the
turn. It checks cards on their own, so a pending execution stage does
not hide a card.
- `server/src/services/heartbeat/run-completion.ts`: a successful
conversation run whose status decision committed now passes
`conversationTurnFinished: true` to the presentation resolver, unless
`conversationTurnAnswered()` is false. The existing
`conversation_turn_finished` path is unchanged.
- `server/src/services/heartbeat/run-completion.test.ts`: add a
regression table with a real database. A pending approval, a
revision-requested approval, a pending execution stage and a dependency
blocker now post the answer. A finished turn still posts. The turn's own
question card still owns the response, also when an execution stage is
pending.
- `doc/execution-semantics.md`: state the rule.

Other finish outcomes I checked:

- `done` or `needs_review` with a gate (`governed_gate_pending`): the
result is not yielded, so the reply was already posted.
- `blocked` (`paperclip_block`, waiting on a person or another task):
not yielded, so the reply was already posted.
- Yielded with a dependency blocker
(`durable_dependency_blocker_bound`), or with
`record_finalization_error` (`prior_status_preserved_no_live_path`,
`monitor_wait_authority_lost`): the reply was also dropped. This fix
covers them, because the rule does not depend on the reason code.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts`: 118 passed. Before the
fix, 4 new cases failed (both approval states, execution stage,
dependency blocker). The finished-turn and question-card cases passed
both before and after.
- `cd server && npx vitest run
src/__tests__/heartbeat-run-summary.test.ts
src/services/heartbeat/run-comment-selection.test.ts
src/__tests__/slack-conversation-lifecycle.test.ts
src/services/native-runtime/agent-chat-response-recovery.test.ts
src/__tests__/chat-run-milestone-governed-work.test.ts
src/__tests__/agent-conversations.test.ts`: all passed.
- `tsc --noEmit -p server`: no errors in the changed files. Unrelated
errors in this worktree come from an unbuilt local Runner package.
- An end-to-end chat eval run on the case branch is in progress. I will
add its result here.

## Risks

- Low risk. The change only affects successful Agent Chat runs with a
committed status decision. Ordinary tasks and external chat (Slack,
GitHub) keep their own paths. External chat returns before this check.
- A turn that yields while an approval is pending now posts its prose.
The approval stays pending and the task status does not change.
- One extra small read (the task row and pending gates) runs after a
conversation turn that is not `conversation_turn_finished`. It runs
after the provider turn ends, so it does not add time before the first
token.
- Previous-release state: no persisted shape changes. The fix reads
`finalizationPhase` and `finalizationReasonCode`, which the previous
release already writes. A run in flight at the deploy completes on the
new code and posts its reply. A run that completed before the deploy
keeps its recorded decision; nothing re-presents it.
- Known gap, unchanged: background presentation recovery for
conversations still requires `conversation_turn_finished` with a `done`
result. If the server restarts between the status commit and the live
presentation, a held yielded turn is not re-posted. That is a separate
follow-up.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell,
file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-08 23:17:36 -06:00
DottaandClaude Opus 5.5 5a51a7685d fix(budgets): raise_budget_and_resume continues the work its budget stopped, once (#20144) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A budget hard stop cancels the scope's running work; the board
resolves it with Raise budget and resume
> - In the restart-chaos H1 scenario, the cancelled tasks did not
continue after resume and each needed a manual Retry
> - Resume is the board's explicit decision to continue, so the work it
stopped should continue once, as if the board clicked Retry
> - This pull request files that Retry automatically, keyed by the
cancelled run, once the run's stop has settled
> - The benefit is that a budget raise restores the work without manual
Retries, without repeating completed work, and without charging the
run's attempt budget

## Linked Issues or Issue Description

Refs #20143 (bug B; this PR builds on it), #20113 (bug A), #20101
(superseded: split into #20143 and this PR)

**What happened?**
Restart-chaos waves w-20261108T195944-s42 and w-20261108T210118-s9802
(H1). After `raise_budget_and_resume`, CHA-3's budget-cancelled run
(enforcement stop) and CHA-1's run (agent lifecycle pause from the
budget hold) stayed cancelled until someone clicked Retry.

**Expected behavior**
Resuming continues each task its budget stopped, exactly once, with no
attempt charged, an activity entry, and no repeat of work that already
continued. A run a person stopped stays stopped.

**Steps to reproduce**
Set a budget below spend while a task runs; let the stop cancel the run;
resolve with Raise budget and resume. Before this change, no new run is
queued.

## What Changed

- New table `budget_resume_claims` (migration 0451). One row per
cancelled run, outside the run's result JSON, so a Stop that settles
during the resume cannot erase it.
- `markBudgetCancelledWorkForResume` (called by `resolveIncident` before
the incident closes): claims every task run its budget stopped since the
incident opened. That covers runs budget enforcement fenced, including
stops still in flight, and runs the agent lifecycle pause stopped for
this budget (`pauseStop`, carried from the lifecycle operation's
`pauseCause` so a resume before the Stop keeps the link). Runs a person
stopped are excluded. Two indexed passes led by `(company_id, status)`.
- `deliverBudgetResumeContinuations`: once a claimed run's stop settles,
files the resolving board member's Retry (`requestTaskRetry`,
idempotency key `budget-resume:<run>`, with the member's session source
and the chat Retry callback). The Retry path checks, under the task
lock, that the run is still the task's latest and the task is open, so
continued or completed work is never repeated. Claim settlement and its
activity entry commit together; retryable refusals and errors retry.
- Delivery runs in the always-on budget-enforcement worker after each
pass's Stop delivery. Its schedule includes deliverable claims and the
earliest stale-claim expiry. The execution-continuation worker is a
backstop.
- `agent.paused` / `agent.resumed` logging is in #20113.
`budget.cancelled_work_resume_requested` (incident) and
`budget.cancelled_work_continued` / `_not_continued` (task) are recorded
here.

## Verification

- New `budget-resume-continuation.test.ts` (Postgres, real
`resolveIncident` → claims → `requestTaskRetry` → real `resumeTaskRetry`
into a queued run):
- each task continues once, including a blocked task; repeat delivery or
resolve saves nothing new;
- a stop still in flight at resume, whose terminal write replaces the
result JSON, is still continued;
  - a 30-task scope is fully continued;
- a lifecycle-pause stop caused by the budget is continued, and one
caused by a manual pause is not;
- continued work, done tasks, a run that finished before its stop, and
person-stopped runs (including ones enforcement also fenced) are not
continued.
- `lifecycle.test.ts`: the pause records its cause on the operation and
on stopped runs, including when the budget is resumed before the pause's
Stop. `budget-enforcement-worker.test.ts`: delivery after Stop delivery,
and a wake at claim expiry.
- 287 tests across the affected suites passed. Server typecheck
(`typecheck-layers`, low-memory): no errors.
- Not run: the H1 chaos rerun (Phase 3 runs it after merge).

## Risks

- Resume now continues budget-stopped tasks automatically. A board that
wants a task stopped can Stop it again; person-stopped runs are never
continued.
- A new table and one index (migration 0451).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-08 22:56:28 -06:00