Commit Graph
43 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 566a97b882 fix(native): an unretirable predecessor sends the next attempt to a fresh sandbox (#20863)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a reused sandbox keep the previous run's warm Runner,
and a new run must retire that owner before it starts
> - #20844 retires the owner when Paperclip can prove it stopped. Some
owners can never be proven stopped: the controller journal drain
outlives the close grace, the Runner never confirms suspension, or a
parked question leaves a tool input unanswered
> - Then the new run failed before any provider work, and the task
stayed blocked with no automatic continuation
> - The general answer is to leave the old Runner on its sandbox and
start the next attempt on a fresh sandbox restored from the saved
workspace
> - This pull request adds that fallback for every path that retires a
predecessor, and settles a parked question's tool input once its Runner
is gone
> - The benefit is that an unprovable predecessor no longer fails a task
that did no work

## Linked Issues or Issue Description

**What happened?**
An automatic wake started a new run 30 to 60 seconds after a long
successful run on the same task, on the same reused sandbox. The new run
had to retire the previous run's warm session first. The retirement
failed and the run failed before any provider work, with one of:
- `Controller journal drain timed out; preserve the session before
replacing its writer.` (during instruction preparation, before runtime
selection, or in the executor);
- `provider_transport_failed: runner did not durably suspend before
checkpoint` (during instruction preparation);
- `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=journal_tool_input_unanswered` (the previous run had
parked on a question).
The tasks rested `blocked` with a reconciliation hold or with nothing
scheduled. One bounded retry reused the same sandbox and ended
`execution_finalization_deadline_exceeded`, then its recovery refused
with `allocation_ownership_ambiguous`.

**Expected behavior**
A run that did no provider work does not fail the task because its
predecessor cannot be retired. The next attempt drops the reused sandbox
and starts on a fresh one restored from the saved workspace. Unsaved
work is never dropped: when the workspace or the session backup is not
durably saved, the sandbox is kept and today's behavior applies.

**Steps to reproduce**
1. Complete a warm native run on a Daytona sandbox.
2. Wake the task again so that the next run must retire the warm owner
(configuration change, new session, changed agent files).
3. Make the owner's close fail (journal drain timeout, unconfirmed
suspension).
4. The new run fails before provider work and the task is not continued.

Refs #20844, #18373.

## What Changed

- `native-runtime/predecessor-retirement.ts`: a typed pre-provider error
`NativePredecessorRetirementUnprovenError`
(`native_predecessor_retirement_unproven`) and the `replaceBeforeReuse`
lease marker.
- Native executor: every path that retires a predecessor for a new run
(instruction-copy claim, instruction-directory reservation, session
reset, configuration change) throws the typed error when the close
fails. The owner stays registered for its own sandbox, so no run on that
sandbox launches over it. A run on a different sandbox forgets that
owner instead of replaying its cached rejected close. The executor does
not settle this error as a provider failure.
- A prior-owner fence that refuses a terminal owner on a resumed sandbox
(`prior_owner_terminal_state_indeterminate`) also throws the typed
error.
- Retirement of a finished remote owner accepts an unanswered tool input
when the owner's committed result is a governed-wait park
(`response_wake` on a task interaction) and its Runner's sandbox lease
ended elsewhere.
- Heartbeat (`heartbeat/predecessor-retirement-fallback.ts`): on the
typed error with no execution-start evidence and no result, mark this
run's reusable lease on that sandbox `replaceBeforeReuse` and log
`environment.lease_replacement_requested`. After the run ends, `failRun`
hands the prepared workspace on only through the shared unchanged-files
proof (`proveRemoteWorkspaceUnchanged`; a provisioning command or any
runner command refuses it). With the proof it records
`predecessorRetirementFallback`, closes the run's coordinator without a
result and re-queues the run in the uncharged `interrupted_preparation`
lane (native and legacy runtime modes); a run that answered a task
interaction keeps its own wake in that uncharged lane, so the retry
still carries the answer. Without the proof the workspace stays dirty,
heartbeat runs today's failed-workspace preservation and no retry is
scheduled.
- Environment runtime: a marked lease is replaced instead of resumed,
under the same rule as a failed resume
(`reusableSandboxReplacementIsSafe`: run ended, clean workspace with a
committed save, verified harness backup). The old lease goes to
`pending_cleanup` for the sweep. Otherwise the sandbox is resumed as
before.
- Chaos: session fault SR1U, the SR1 variant where the retained owner
cannot be retired. The server gets a test-only fault
(`PAPERCLIP_TEST_WARM_RETIREMENT_FAULT=unconfirmed`, armed only when
`NODE_ENV=test`). The oracle allows only the fallback failure and
requires the succeeding follow-up on a different sandbox.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md`, chaos README.

## Verification

- RED on master before this change (with #20844), then GREEN:
- `native-session-executor.test.ts -t "predecessor cannot be retired"`:
base fails with `Error: Controller journal drain timed out…` instead of
the typed error.
- `native-session-executor.test.ts -t "unretired predecessor"`
(instruction preparation): same base failure.
- `native-recovered-owner-followup.test.ts -t "unacknowledged"`: base
fails with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=journal_tool_input_unanswered`.
- New contract tests: `heartbeat-native-predispatch-retry.test.ts`
(claimed native run: lease marked, activity logged, uncharged
`interrupted_preparation` retry, no recovery action, task not blocked,
retry succeeds) and
`environment-runtime-reusable-resume-replacement.test.ts` (a marked
lease is replaced without a resume; with unsaved work or no verified
backup it resumes as before),
`remote-workspace-unchanged-release.test.ts` (an untouched workspace is
handed on; a setup-written one stays dirty) and `retries.test.ts` (an
answered interaction re-queued in the uncharged lane stays a
resolved-interaction continuation with no failure retry spent).
- Existing suites pass: the native executor, recovered-owner follow-up,
instruction working copy, run completion, run failure cause and resume
replacement test files. `pnpm exec vitest run --config
tests/restart-chaos/vitest.config.ts question-handoff scheduler config`
passes. Server and chaos harness `tsc --noEmit` pass.
- Not run: a Daytona SR1U wave. It needs a runner image that matches
this checkout.

## Risks

- A sandbox marked for replacement is resumed when it cannot be replaced
safely. That run can meet the same unretired owner again, fail again in
the bounded lane, and then rest with Retry, as today.
- The replacement session restores the latest verified harness backup.
When the old Runner's final checkpoint was never saved, a same-session
continuation resumes from that backup.
- An unanswered tool input is accepted only for a committed
governed-wait park whose Runner's sandbox lease ended on another
sandbox.
- Previous-release story: the marker and the run record are additive
JSON. An older server ignores the marker and resumes the sandbox as
before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 12:10:24 -06:00
DottaandClaude Opus 5.5 4a61a2501e fix(storage): never promise a storage retry no storage wait owns (#20676)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A run that hits a file-storage outage gets a bounded storage wait: a
retry that starts again after storage is back
> - The run error says "Paperclip will retry automatically when storage
is back"
> - On staging, a server restart reattached a remote native runner while
its turn was still running. A storage outage then failed the
reattachment
> - Restart recovery kept the workspace under a no-replay hold, so the
run took the code `native_continuation_requires_reconciliation`. Only a
`storage_unavailable` run gets a storage wait, so no retry was
scheduled. The run still showed the storage retry promise, and the task
stayed blocked for about 15 hours
> - This pull request makes the storage text follow the run's final code
and the real storage wait outcome, and reads old rows truthfully
> - The benefit is that every storage message is true: it promises a
retry only when one is scheduled, and it offers a working Retry when
nothing retries the run

## Linked Issues or Issue Description

**What happened?**
A native run on a remote sandbox was reattached after a server restart.
Its provider turn was still in flight. File storage failed during the
reattachment. Restart recovery settled the run as
`native_continuation_requires_reconciliation`, kept its files, and did
not replay the unfinished turn (this is the existing policy for a turn
with an unknown outcome). The run error said "Paperclip will retry
automatically when storage is back". No storage wait was scheduled, so
nothing retried. The task stayed blocked until an operator clicked
Retry.

**Root cause**
`server/src/services/heartbeat/run-completion.ts` chose the message and
the error code separately. The message used
`isStorageUnavailableError(err)`. The code used a chain where
`nativeTerminalFailureCode` wins over `storage_unavailable`. The storage
wait is scheduled only when the code is `storage_unavailable` (and no
native terminal failure is recorded). So the message promised a wait
that the code made impossible. The same false promise stayed when the
storage wait was refused (`not_scheduled`) or its write failed.

**Expected behavior**
A storage failure promises an automatic retry only when the storage wait
is scheduled. Otherwise it offers Retry. Rows saved by earlier releases
read the same way after deploy, with no migration.

**Steps to reproduce**
Fail a native run with a storage-unavailable error after its native
coordinator is settled as `terminal_failure` /
`native_continuation_requires_reconciliation`. See the new regression in
`run-completion.test.ts`.

## What Changed

- `storage-unavailable.ts`: add the held messages (outage and rejected
credentials). They say Paperclip is not retrying automatically and offer
Retry or a message. Add `storageUnavailableHeldMessage()` and
`presentStorageFailureMessage()`.
- `run-completion.ts` (`failRun`): compute the error code first. Use the
storage retry promise only when the code is `storage_unavailable`. Under
any other code (for example a native recovery hold), use the held
message. No storage wait is created for such a run, because it could
only be refused.
- `run-completion.ts` (`failRun` and `failRunSetup`): replace
`markStorageWaitExhausted` with `settleStorageWaitMessage`. It keeps the
promise only when a retry is scheduled. It writes the exhausted text
when the budget is spent, and the held text when the wait was refused,
its write failed, or a case handoff took over.
- `run-completion.ts`: when the promise is withdrawn, the failure's
`error` event (written before the retry decision) gets the same final
text, so the transcript does not keep the promise after reload.
- `run-completion.ts` (`failRunSetup`): a lost controller lease keeps
its own code and text, not the storage promise.
- `run-lifecycle.ts`: `decorateHeartbeatRunRuntimeStatus` presents a
saved storage retry promise under a code other than
`storage_unavailable` with the held text. This covers run detail and
issue run presence. `listRuns` in `run-state.ts` applies the same
projection, so the company run list and the Inbox failure read agree.
Rows from earlier releases read truthfully with no migration.
- `doc/run-log-events.md`: document which runs get a storage wait and
what each message means.

Previous-release story (upgrade-safe run state): this change adds no
persisted shape. Rows saved by the previous release carry forward. A row
with the storage promise under another code reads with the held text. A
`storage_unavailable` row keeps its text. The native hold policy is
unchanged: an unfinished turn with an unknown outcome is not replayed
automatically, and the existing Retry continues it.

## Verification

- Red before the fix (assertion failures, not setup errors): the new
`run-completion.test.ts` cases received "Paperclip will retry
automatically when storage is back" under
`native_continuation_requires_reconciliation`, and the
`run-lifecycle.test.ts` cases read the saved promise unchanged.
- Green after the fix:
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts
src/services/heartbeat/run-lifecycle.test.ts -t storage` (13 passed)
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts
src/services/heartbeat/run-lifecycle.test.ts
src/storage/storage-unavailable.test.ts` (213 passed)
  - `cd server && npx tsc --noEmit -p tsconfig.json` (clean)
- Existing storage cases still pass: a scheduled storage wait keeps its
promise and suppresses rapid recovery; an exhausted wait writes the
exhausted text.
- `pnpm verify` exited 2 (a doc is a global CI input), so required CI
supplies the broad coverage.
- The full `run-completion.test.ts` file passed every test locally. Its
`afterAll` hook timed out under machine load (load average about 17),
also with the new tests skipped.

## Risks

- Low risk. The change affects only the text of failed runs and one read
projection. It does not change retry scheduling, recovery ownership, or
replay policy.
- The read projection matches this repository's own two promise
constants under a code other than `storage_unavailable`. A future change
to those constants must keep the projection in sync.
- Error events already written by earlier releases keep their original
text in the run log. Live events already published before a withdrawn
promise are not resent.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, browser reads for GET-only diagnostics).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 19:22:26 -06:00
DottaandClaude 7368a0b15c fix(recovery): re-queue native runs that fail on a transient database fault before provider work (#20561)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native (Paperclip Runner) runs prepare a task, pass a final dispatch
gate, and only then start a provider.
> - A native run can fail before any provider work starts. Examples are
a Postgres serialization or deadlock error (`40001`, `40P01`) at budget
admission, or a transient database error during setup.
> - Today such a run ends `adapter_failed` or `setup_failed`. `failRun`
writes the "no provider work" record only for legacy runs, and the
bounded retry paths do not retry these codes for native runs. The
release then opens the native hold
(`native_continuation_requires_reconciliation`), and a person must click
Retry.
> - The goal is that 0% of tasks need a Retry click after a restart or a
storage/DB fault. Duplicate provider work is worse than a Retry click,
so the fix must only act when the server can prove no provider work
started.
> - This pull request records that proof for native runs and sends a
transient database failure to the existing uncharged, bounded
`interrupted_preparation` retry lane.
> - The benefit is that these runs re-queue by themselves, and runs with
any sign of provider work keep today's hold.

## Linked Issues or Issue Description

**What happened?**

In restart-chaos waves with database pressure, a native run failed at
budget admission with exhausted serialization retries (`40001`/`40P01`).
The run event history had no `native.process_start_requested` event.
About 15 s later the server recorded "Automatic recovery stopped" with
cause `native_continuation_requires_reconciliation`. The task stayed
blocked until a person clicked Retry.

**Expected behavior**

A native run that failed before any provider work, on a fault that
clears by itself, re-queues automatically. The attempt is bounded and
does not spend the failure budget. A persistent failure still ends
failed, with a working Retry.

**Steps to reproduce**

1. Start a native (Paperclip Runner) task.
2. Make budget admission or a setup database write fail with SQLSTATE
`40001` or `40P01` until its bounded retries are exhausted.
3. See the run end `adapter_failed`/`setup_failed`, the task go to
`blocked`, and a `native_continuation_requires_reconciliation` recovery
action that waits for Retry.

**Paperclip version or commit**

`master` at 902f376796.

## What Changed

- `run-failure-cause.ts`: new durable record
`resultJson.nativePreDispatchFailure = { version: 1,
providerWorkStarted: false }`, and
`isNativePreDispatchTransientFailure`. The predicate needs a native
failed run, a generic code (`adapter_failed`/`setup_failed`), the
record, and a transient database `failureCause` (`40001`, `40P01`,
`55P03`, `57P01`-`57P03`, `53300`, `08xxx`, postgres.js connection
codes).
- `run-completion.ts`: `failRun` and `failRunSetup` write the record for
a native run only when all of these are true:
- the caller reports `nativeDispatchMayHaveStarted: false` (an omitted
value means "may have started");
- the run was not stopped during the failure, and has no terminal native
coordinator failure;
- the run's `native_run_finalizations` coordinator, if any, is still the
unclaimed row that runtime selection creates (`observed`, attempt 0, no
lease owner, no controller, no result). Any executor claim, in any
generation, disqualifies;
- the run has no recorded process (`processPid`, `processGroupId`,
`processStartedAt`), no `sessionIdAfter`, and no `native_run_results`
row. Process fields copied from a warm predecessor (marked by
`runnerProfileJson.inheritedProcessIdentityRunId`) are not this run's
own process and do not disqualify; the run's own start events are still
checked;
- the run's whole event history has no execution-start evidence. This
uses the canonical `hasNativeExecutionStartEvidence` helper from restart
recovery (`native.process_start_requested`,
`native.process_identity_recorded`, `adapter.invoke`,
`tool.execution.started`, or any runner event with a source event ID).
The helper is unchanged; the earlier inline copy matched it exactly and
was removed.
- `run-completion.ts`: a recorded transient failure closes the unclaimed
coordinator as `terminal_failure` (`native_pre_dispatch_failure`),
because `hasPendingRunCleanup` otherwise holds the successor on it. It
then calls the existing `scheduleBoundedRetryForRun` with `retryReason:
"interrupted_preparation"`. An answered-interaction continuation keeps
its own retry lane. The storage-outage lane is unchanged.
- `heartbeat.ts`: passes `nativeDispatchMayHaveStarted`. It is true once
native dispatch started, once the dispatch gate entered the executor
(also when the gate then fails), while native ownership is held, or for
a restart-recovery or native-lease resume.
- No change to the native hold in
`modules/wake-queue/adapters/postgres.ts`. The release already returns
early when a queued/scheduled successor exists, so a scheduled retry
never opens the hold. See Risks for why the hold stays for the exhausted
and no-successor cases.

## Verification

- `heartbeat-native-predispatch-retry.test.ts` (assembled heartbeat,
mock native backend): budget admission fails once with `40001`. The run
records the proof, one uncharged `interrupted_preparation` successor is
scheduled, no recovery action opens, and the promoted retry succeeds
once. A failure after the dispatch gate entered the executor gets no
record and no such successor.
- `run-completion.test.ts` (embedded Postgres):
- native pre-dispatch `40P01` in `failRun` writes the record and
schedules `interrupted_preparation`; the release suppresses immediate
recovery;
  - native `setup_failed` on `40001` does the same;
- a run with `native.process_start_requested` in its history gets no
record and no retry;
- `nativeDispatchMayHaveStarted: true` and an omitted value (older
caller) get no record and no retry;
- a claimed native coordinator gets no record and no retry; an unclaimed
one is closed and the run re-queues;
- a run with a recorded process or provider session gets no record and
no retry;
  - a non-transient cause gets the record but no retry.
- `heartbeat-retry-scheduling.test.ts`: the real scheduler chains two
`interrupted_preparation` retries for a recorded native failure. The
first is due at once. `failureRetries` stays 0. A spent lane returns
`retry_exhausted`, queues nothing, and the run stays `failed`.
- `wake-queue/adapters/postgres.test.ts`: with a scheduled successor,
the release opens no recovery action and the task stays `in_progress`.
Without one, today's hold opens (`blocked`,
`native_continuation_requires_reconciliation`, Retry available).
- `run-failure-cause.test.ts`: the predicate is false for a row without
the record (older server), for `providerWorkStarted: true`, for a
non-transient cause, a non-generic code, a legacy run, and a cancelled
run.
- Command: `cd server && npx vitest run
src/__tests__/heartbeat-native-predispatch-retry.test.ts
src/services/heartbeat/run-failure-cause.test.ts
src/services/heartbeat/run-completion.test.ts
src/modules/wake-queue/adapters/postgres.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts` (all new cases pass).
`tsc --noEmit` for `server` passes. `pnpm verify` selected 2451 files
because `heartbeat.ts` is a hub; I stopped it and rely on CI for the
broad suite.
- Live path unproven: I did not run a restart-chaos wave for this
change.

## Risks

- A wrong "no provider work" record could cause duplicate work. The
record needs several independent facts (no gate entry, no resume claim,
no claimed coordinator, no process, session or result, no
execution-start evidence in the whole history). Any read error keeps the
hold.
- Rows saved by an older server have no record. They keep today's hold
and manual Retry. Nothing is migrated. A run resumed after a deploy is a
restart-recovery or lease resume, so it gets no record.
- The normal path gains no I/O. The evidence reads run only on the
failure path, and only when dispatch was never entered.
- The native hold is not skipped when no successor exists. Skipping it
would send the run to the generic immediate-recovery tail, whose new run
does not carry this lane's counter. That could loop without a bound. So
a spent lane, and a crash between the failure write and the retry
scheduling, still end in today's hold with a working Retry.
- The lane is bounded by `INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`
(12) with backoff after the first attempt, and it never revives a chain
whose failure budget is already spent.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code
with tool use (file edits, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #20484, #20385.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 11:24:37 -06:00
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00
DottaandClaude Opus 5.5 0b20ea6fe9 fix(accounting): keep budget admission and usage handoff alive under run-row updates (#20484)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Before a run starts provider work, Paperclip reserves its budget and
binds a usage recorder. Both write the run's `heartbeat_runs` row.
> - Run-log appends, output progress and lifecycle events update that
same row while the run prepares and streams.
> - #20385 fixed this race for the standalone usage-receipt write only.
Budget admission and the recorder handoff still take a repeatable-read
snapshot before they lock the run row.
> - Each concurrent row update makes that snapshot stale. Four retries
run out with SQLSTATE `40001`, and the run ends `adapter_failed` with
raw SQL as its error.
> - This pull request applies the #20385 pattern (READ COMMITTED under
the existing locks) to those writers and to the final dispatch gate. It
also gives exhausted retries a bounded cause.
> - The benefit is that busy runs start and account normally, and a
remaining failure names its cause.

## Linked Issues or Issue Description

Refs #20385. Builds on #20483 (bounded failure cause and
`withoutSqlStatementText`).

**What happened?**

The restart-chaos L1 workload with `--db-pressure 4` (harness writers
that update each running run row, one row per statement, like a fast
stream) failed on master. In one wave, 0 of 4 tasks passed. Every first
run failed in `reserveRunBudget` before spawn. In an earlier wave, a run
failed in `createRunUsageRecorder`. The stack in both cases is `could
not serialize access due to concurrent update` (`40001`) on the run-row
`FOR UPDATE`. The board `runError` showed the full `select ... from
heartbeat_runs ... for update` statement. The server log line `heartbeat
execution failed` dropped the database cause.

Realism: the harness writers are synthetic, but the product has the same
writers on the same row. `appendHeartbeatRunEvent` increments
`next_event_seq`, output progress updates `last_output_*`, and lifecycle
events land while a run prepares. On a hosted database the window
between snapshot and row lock is wider than locally, so a busy run
reaches the same race without the harness.

**Expected behavior**

Budget admission and the recorder handoff commit while other writers
update the run row. Budget invariants stay intact: one reservation per
run, a reached hard stop still refuses work, and committed holds count
against later admissions. Usage receipts are journaled once. If retries
still run out, the run shows a bounded cause without SQL, and the server
log records the SQLSTATE.

**Steps to reproduce**

Run `server/src/services/run-accounting-contention.test.ts` against the
parent commit. The deterministic cases commit a real
`appendHeartbeatRunEvent` after the writer's transaction checks the
company and before it locks the run. Six of the eight cases fail with
`40001`.

## What Changed

- `reserveRunBudget` runs at READ COMMITTED. All mutable reads follow
the shared admission fences, the agent `FOR SHARE` lock and the run-row
`FOR UPDATE`. Exposure stays one SQL statement (hold-to-ledger atomic).
The `budget_reservations_run_idx` unique index and the run lock still
prevent a second hold.
- The final dispatch gate's accounting phase
(`dispatchResolvedInteractionIfCurrent` through `withIssueThenRunLocks`
in `run-dispatch/adapters/postgres.ts`) runs at READ COMMITTED. Its
ownership phase already read the same staleness facts at READ COMMITTED
under the same issue and run locks. Retry promotion and stale-run
cancellation keep their repeatable-read snapshot. The first L1 proof
wave found this writer exhausting `40001` once the other three were
fixed.
- `createRunUsageRecorder` runs its receipt-source handoff and its
capture-failure fence at READ COMMITTED. Both lock the run row first,
the same as `persistUsageReceipt` after #20385.
- `withOperationAccountingTransaction` raises
`AccountingContentionError` when retries run out. It has a bounded
message, `code` keeps the SQLSTATE (`40001`/`40P01`), `cause` keeps the
database error, and a warning logs both.
- `failRun` shows that bounded message. The code stays `adapter_failed`,
so chat and recovery retry paths still apply. #20483's `failureCause`
records `40001`. All other SQL text handling comes from #20483
(`withoutSqlStatementText`), unchanged here.
- `doc/DATABASE.md` documents the isolation choice and the failure
contract.

### Inventory of REPEATABLE READ / SERIALIZABLE transactions that write
run-scoped rows

| Writer | Status | Notes |
| --- | --- | --- |
| `persistUsageReceipt` (`usage-receipts.ts`) | Fixed in #20385 | READ
COMMITTED under the run lock. |
| `reserveRunBudget` (`budget-reservations.ts`) | Fixed here | Seen in
waves (`budget-reservations.ts:90`). |
| `createRunUsageRecorder` handoff (`usage-receipts.ts`) | Fixed here |
Seen in waves (`usage-receipts.ts:572`). |
| `createRunUsageRecorder` capture-failure fence (`persistFailure`) |
Fixed here | Same shape: run lock first, runs mid-stream. |
| Native finalizer `claimCoordinator` (`native-run-finalizer.ts`) | Safe
| A default READ COMMITTED transaction with `FOR UPDATE` on the
coordinator row. It cannot raise `40001`. The wave errors there were
`native_finalization_missing`, a symptom of the failed reservation (no
coordinator existed yet). |
| Final dispatch gate, accounting phase
(`dispatchResolvedInteractionIfCurrent`) | Fixed here | Seen in proof
wave `w-20261110T075857-s20386` (runs `140ddb10`, `c028c6c4`,
`fea0ee5a`). Its ownership phase already used READ COMMITTED under the
same locks with the same staleness reads. |
| Retry promotion and stale-run cancellation (`promoteOrCancelDueRetry`,
`cancelStaleQueuedRun`, same lock helper) | Follow-up | Kept at
repeatable read: `loadGateFacts` combines wakes, comments, dependencies,
tree holds and repair state in one decision. They act on queued or
scheduled runs, not streaming ones. Not seen failing in waves. If
retries run out, they now fail with the bounded cause. |
| Run settlement `accountRunCost` (`run-cost-accounting.ts`) | Follow-up
| Seen with `40001` in a wave log. Leaves the run pending;
`reconcileRunCosts` retries it durably. Multi-aggregate (coordinator,
receipts, images, closure). |
| Tool cost events (`paperclip-runner-tool-authority.ts`, `costs.ts`) |
Safe | Insert `cost_events` only; the run row is referenced, not locked
or updated. |
| Budget enforcement and evaluation (`budgets.ts`), billing
reconciliation, accounting-integrity repair | Safe | Lock policy,
invoice or company rows, not a streaming run row. Run reads are plain
selects. |
| Browser use, decisions, prompt-injection, fast responses | Safe |
Write their own invocation/reservation rows, not `heartbeat_runs`. |
| Read-only RR snapshots (`withAccountingReadSnapshot`, idle-sleep, tool
gateway, customer success, `routes/decisions.ts`) | Safe | Read only, or
no run-scoped writes. |

## Verification

- Red on the parent commit, green on this branch:
`server/src/services/run-accounting-contention.test.ts` (real embedded
PostgreSQL).
- `reserveRunBudget > reserves once when run events commit between its
company check and its run lock` (red: `40001`)
- `reserveRunBudget > reserves every run while back-to-back writers
update the run row` (red: `40001`)
- `reserveRunBudget > counts every committed hold, so successive
admissions never exceed the budget` (red)
- `reserveRunBudget > still refuses a run once the hard stop is reached`
(red: `40001` instead of `409`)
- `createRunUsageRecorder > binds its receipt source when run events
commit between its company check and its run lock` (red: `40001`; green
asserts the spooled receipt is journaled exactly once)
- `createRunUsageRecorder > binds while back-to-back writers update the
run row` (red: `40001`)
- `reserveRunBudget > never creates a second reservation for one run
when two dispatches race` (invariant; passes before and after)
- `final dispatch gate > admits and hands off a reserved run when run
events commit between its company check and its run lock` (red:
`AccountingContentionError` after 4 × `40001`)
- `createRunUsageRecorder > persists its capture-failure fence once when
run events commit between its company check and its run lock`
- `final dispatch gate > admits while back-to-back writers update the
run row` (red: `40001`)
- `exhausted retries > fail with a bounded, SQL-free cause that keeps
the SQLSTATE` (new contract)
- `run-completion.test.ts > saves a bounded cause and keeps the
retryable generic code when accounting retries are exhausted` (saved
code, message and `failureCause`, and log fields, through `failRun`)
- Related suites pass: `usage-receipts-read`,
`admission-independent-of-cleanup`, `budget-live-attempts`,
`idle-accounting-work`, `budget-enforcement-scoping`,
`closed-native-receipt-accounting`, `budget-evaluation-work`,
`run-dispatch/adapters/postgres`, `heartbeat/run-completion` (all 126
run-completion cases pass; its `afterAll` embedded-PostgreSQL teardown
exceeded 30 s on this loaded host, the known local flake).
- `pnpm --filter @paperclipai/server typecheck` passes.
- Latency of `reserveRunBudget` on the normal path (no contention,
embedded PostgreSQL, 350 measured reservations per run, 3 runs each):
before p50 14.7 / 15.0 / 18.6 ms, p95 25.6 / 31.4 / 39.4 ms; after p50
14.2 / 14.0 / 16.6 ms, p95 29.7 / 24.0 / 29.3 ms. No added latency: the
statements are the same, only the isolation level changes. Under
contention it removes the retry backoff.
- Live restart-chaos L1 waves (`--seed 20386 --db-pressure 4`, codex and
claude, K1):
- Master before this change: `w-20261110T050818-s20386`, 0/4. Every
first run failed in `reserveRunBudget` with `40001`.
- Head `90530789b0` (first three writers fixed):
`w-20261110T075857-s20386`, 2/4. Both codex tasks passed. Both claude
failures (t02, t03) trace to the dispatch gate, now fixed. Those runs
already showed the bounded cause, with no SQL on the board. The failure
was logged with `sqlState` 40001. No `reserveRunBudget` or recorder
`40001` remained.
- Head `d5ed21926f` (dispatch gate fixed): `w-20261110T090531-s20386`.
All 4 tasks reached done with no retry and R1–R6 passed. There were no
failed runs and zero `40001` in the server log, under 310k pressure
updates. Each task still failed the T-oracle `company_spend_settled`
with `pending_runs=1`. That pending run is t02's continuation run
`601ae80a`, cancelled before provider work as `issue_terminal_status`,
whose accounting debt was not yet settled when graded about 15 s later.
It is not a contention failure. Caveat: I edited the worktree during
this wave (`worktree_changed_mid_wave`), so a clean wave on the final
head follows.
- Final head `9f1c480717`, clean worktree: `w-20261110T094417-s20386`.
All 4 tasks reached done with no retry, and R1–R6 passed for each. There
were no failed runs and zero serialization failures in the server log,
under about 523k pressure updates. Each task still fails only the
T-oracle `company_spend_settled` (`pending_runs=1`). It is the same
cancelled pre-provider continuation run (`issue_terminal_status`), whose
accounting debt was still unsettled at grading. That is a separate
settlement-timing follow-up, not database contention. Master was 0/4 on
the same seed.

## Risks

- READ COMMITTED gives each statement its own snapshot. Admission reads
policy, scope and exposure in separate statements. The shared
company/project fences and the agent share lock exclude concurrent pause
and policy edits, and exposure stays one statement, so each read is at
least as fresh as the old snapshot.
- Concurrent admissions of different runs stay best-effort, as
`doc/SPEC-implementation.md` already states ("concurrent starts can
exceed the same observed balance"). This change does not widen that
window: each statement snapshot is newer than the old transaction
snapshot. It does not add a company mutex.
- A run that fails after exhausted retries now records
`accounting_contention` instead of `adapter_failed`. Retry rules that
key on `adapter_failed` spawn-like messages did not match this failure
before, so no automatic retry changes.
- No schema, migration or persisted-format change.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with shell, file-edit and code-execution tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 03:47:48 -06:00
DottaandClaude a14ae9b5c9 fix(workspaces): name a failed workspace setup command on the task page (#20478)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before an agent starts, Paperclip prepares the task workspace and
runs the project's configured setup commands (for example the workspace
provision command)
> - When that command fails, the run fails as `setup_failed`, and the
task stops with a Retry
> - The task page shows only "Failed" and a generic recovery notice. The
real error (the command, its exit status and output) is visible only on
the run page
> - The user cannot see from the task which setting is broken or what to
change
> - This pull request records a bounded, named cause for a failed setup
command and shows it on the task page
> - The benefit is that the task names the broken setting and what to
change, and classification, retry and recovery stay the same

## Linked Issues or Issue Description

**What happened?**
The restart-chaos config fault wave CF1 sets a project
`workspaceStrategy.provisionCommand` to `bash
./scripts/provision-worktree.sh`, a script that does not exist. The run
fails as `setup_failed` with `Execution workspace provision command
"bash ./scripts/provision-worktree.sh" failed: bash:
./scripts/provision-worktree.sh: No such file or directory`. The task
page shows "Failed" and "Paperclip automatically retried continuation …
still has no live execution path". The grader fails the task as
`CF1/provision_command:message_only_in_run_detail`.

**Expected behavior**
The task page names the failed setup step, the configured command, its
exit status and a short tail of its output, and says what to change.
Retry behavior stays the same.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs
tests/restart-chaos/config-wave.ts --faults CF1 --adapters codex --tasks
2 --concurrency 2 --restarts K1,K2 --seed 204` on `master`.

**Paperclip version or commit**
`master` at 5ab98b0 (wave `w-20261110T043929-s204-cf`, tasks t01 and
t03).

## What Changed

- `WorkspaceCommandExitError` now carries the setup step
(`workspace_provision`, `workspace_seed`,
`workspace_runtime_provision`), the configured command and the command
output.
- New `server/src/services/workspace-setup-command-failure.ts`: it
builds a bounded, redacted `setupCommandFailure` record (step, command
up to 200 characters, exit code, last 300 characters of output). It also
recognizes the older message format, so runs recorded before this change
still get a name.
- `run-completion.ts` adds `setupCommandFailure` to the setup failure
`resultJson`. The error code (`setup_failed`) and all retry and recovery
fields stay the same.
- `TaskInterruptionPresentation.setupFailure` (shared type) is set for
`setup_failed` runs. It goes through the same per-run secret redaction
and user-name redaction as `workspaceFailure`.
- `ExecutionBlockerNotice` shows "Couldn’t set up the task workspace.
<command> failed with exit code N: <output>. Fix the command in the
project’s workspace settings, then retry." A scheduled transient retry
keeps its countdown and only adds the name.
- The restart-chaos config wave reads `interruption.setupFailure` as a
task-level text for its actionable-message check.

## Verification

- Red before, green after (the tests fail with the source changes
reverted):
- `server/src/__tests__/workspace-runtime.test.ts` — "names a failed
project provision command with its exit status and output (CF1)" runs a
real git worktree with a missing provision script (exit 127).
- `server/src/services/task-interruption-presentation.test.ts` — "setup
command failure presentation (CF1)": names the command, keeps
`state`/`canRetry`/`userAction` identical, recognizes the older message,
keeps a scheduled retry, bounds output and strips credentials from the
command.
- `ui/src/components/ExecutionBlockerNotice.test.tsx` and
`tests/restart-chaos/config-faults.test.ts` cover the notice text and
the harness check.
- `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter
@paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui
typecheck` pass.
- Live proof: config wave `w-20261110T083227-s204-cf` (CF1, codex, 2
tasks, K1 and K2, seed 204) on commit ae9a4b7: **2/2 pass**. Both tasks
show `The project's workspace provision command "bash
./scripts/provision-worktree.sh" failed with exit code 127: bash:
./scripts/provision-worktree.sh: No such file or directory` on the task,
rest again after the restart, and finish after the repair and Retry. On
`master` the same wave failed both tasks as `message_only_in_run_detail`
(`w-20261110T043929-s204-cf`).

## Risks

- Low risk. The change adds a presentation field. It does not change the
error code, retry, recovery or blocking behavior.
- The command and output are user-configured text. They are bounded,
passed through `redactSensitiveText`, the run's secret registry and
user-name redaction. If redaction fails, the field is omitted, like
`workspaceFailure`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 02:35:18 -06:00
DottaandClaude 1e13c58a4e fix(heartbeat): name the cause of generic adapter failures and keep SQL text off the board (#20483)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Heartbeat run settlement records why a run failed, and the board
shows that record.
> - Runs that fail without a named code settle as `adapter_failed` (or
`setup_failed`) with no cause.
> - A user, an operator, and the restart chaos harness cannot tell what
went wrong from such a run.
> - Some run errors also show raw SQL statement text to users.
> - This pull request records a bounded `failureCause` code for generic
failures, shows it on the board, and keeps SQL text out of run errors.
> - The benefit is that a generic failure now names its cause, for
example "adapter failed (40001)", without leaking messages, SQL, or
secrets.

## Linked Issues or Issue Description

Refs #20385 (the nested-cause extraction this change reuses). Related:
#20421 (restart chaos universal checks U1-U4).

**What happened?**

The restart chaos audit of 79 saved waves found 30 failed runs with
`errorCode: "adapter_failed"` and a null or generic error. Every U3
("diagnosable") warning was `adapter_failed`. Nobody could tell what
failed. Some board run errors also showed raw SQL statement text from
wrapped database errors (Drizzle `Failed query: ... params: ...`).

**Expected behavior**

A run that fails with a generic code names a short, safe cause. The
board shows the cause next to the stop reason. Run error text never
shows SQL statements or their parameters.

**Steps to reproduce**

Make an adapter throw a wrapped database error (an `Error` whose nested
`cause` has `code: "40001"` and whose message has a Drizzle `Failed
query:` statement). Before this change, the run settles as
`adapter_failed` with no cause, and `heartbeat_runs.error` contains the
SQL statement and parameters. The new tests in
`server/src/services/heartbeat/run-completion.test.ts` reproduce this.

## What Changed

- Runner: move the #20385 event-commit cause extraction into
`safeErrorCauseCode`
(`packages/paperclip-runner/src/control-plane/safe-error-cause.ts`) and
export it. The Runner's own diagnostics use the same function, unchanged
in behavior.
- Shared: add the `failureCause` contract
(`packages/shared/src/run-failure-cause.ts`): one `[A-Za-z0-9_.:-]`
token of at most 80 characters, `normalizeRunFailureCause`,
`readRunFailureCause`, and `withoutSqlStatementText`.
- Server: add `server/src/services/heartbeat/run-failure-cause.ts`.
Settlement records `resultJson.failureCause` only when the code is
generic (`adapter_failed`, `setup_failed`). `errorCode` stays the same.
- Thrown errors (`failRun`, `failRunSetup`): `provider_http_<status>`,
else `provider_<category>`, else the first safe code in the error and up
to three nested causes (`40001`, `ECONNRESET`), else the error class
name, else `unknown_error`.
- Adapter results without a code (`completeRun`): the error family,
`signal_<name>`, `process_cancellation_failed`, `exit_code_<n>`,
`adapter_reported_error`, or `unknown_error`.
- The failure log lines (`heartbeat execution failed`, `heartbeat
execution setup failed`, and a new `heartbeat run failed` warning for
generic adapter results) include `failureCause` and `errorCode`.
- Server: replace SQL statement text in run error text with "Database
query failed (<cause>)" at all three settlement paths, including the
cancelled branch. The execution blocker's `runError`, the attention
inbox (failed-run excerpt and metadata), and continuation summaries
apply the same replacement on read, so older rows also stop showing SQL
there.
- Server: Retry admission and failed-preparation replay accept only a
closed set of `resultJson` keys. `failureCause` is added to each of
those lists (`task-retry.ts`, `chat-channels/failed-run-retry.ts`,
`cancelled-native-startup.ts`, `fresh-workspace-continuation.ts`). It is
a diagnostic code with no execution or replay authority, so a
failed-preparation run stays retryable.
- Server: `runsForIssue` returns a validated top-level `failureCause`.
It is null for older rows, non-string JSON values, and malformed values.
- UI: `RunForIssue.failureCause`; `IssueRunLedger` shows "adapter failed
(40001)". Older runs keep "adapter failed". The Storybook run-ledger
fixture has a failed run with a cause.
- Restart chaos harness: run samples keep `failureCause`. U3 treats a
generic code with a specific `failureCause` as diagnosable. A bare
`Error`, `unknown_error`, or the no-code fallback
`adapter_reported_error` still counts as vague. U3 stays a warning.
- Docs: `doc/run-log-events.md` and the harness README describe the
field and the SQL replacement.

## Verification

- Red before the fix: the four new `run-completion.test.ts` cases failed
against `origin/master`'s `run-completion.ts` (no `failureCause`, SQL
text in `error`).
- Green:
- `server/src/services/heartbeat/run-completion.test.ts`: all tests
passed (real embedded PostgreSQL). New cases cover a nested `40001`
three levels deep for execution and setup failures, SQL and a secret
parameter absent from `error` and `resultJson`, the bound (an
81-character code and a SQL "code" fall back to the class name),
`exit_code_137` for a code-less adapter result, a cancelled result with
SQL text, no cause for a named code, and the log line. On this loaded
host the suite's `afterAll` database teardown hit its 30 s hook limit
after every test passed (the known local teardown limit, also seen in
#20385). CI runs the suite.
- `server/src/services/heartbeat/run-failure-cause.test.ts` (4),
`server/src/__tests__/activity-service.test.ts` (recorded, legacy,
malformed, overlong, boolean, and number cause projections),
`server/src/__tests__/issue-continuation-summary.test.ts` (legacy SQL
row).
- `packages/shared/src/run-failure-cause.test.ts` (6),
`ui/src/components/IssueRunLedger.test.tsx` (21), restart chaos
`universal.test.ts` and `observer.test.ts`.
-
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`:
207 passed (the stall-diagnostic cases still read `40001` and
`ECONNRESET`).
- Typecheck: shared, runner, server (all layers), UI, restart chaos
harness. `pnpm check:token-gates`: all gates clean.
- `pnpm verify` returned exit 2 (global doc input changed); required CI
supplies broad coverage.
- CI on `6295f8d` found that the new key made failed-preparation runs
ineligible for Retry (`chat-channels.integration.test.ts` saved
board-message Retry got 409;
`workspace-recovery-preparation-dispatch.test.ts` dispatched 0 Retries).
Both reproduced locally and pass after the allowlist fix, with the other
459 Retry cases in `chat-channels.integration.test.ts` and all 25 in
`workspace-recovery-preparation-dispatch.test.ts`.
- No chaos waves were run. Storybook was not opened in a browser; the
fixture typechecks and the label is covered by the UI test.

## Risks

- Low risk. `errorCode` values do not change, so retry, recovery, and
comparison logic is unaffected.
- No schema change and no migration. `failureCause` is a new optional
`resultJson` key. Rows saved by the previous release have no key and
render as before; a new row read by an old server is ignored.
- The SQL replacement only acts on a Drizzle `Failed query:` marker, or
on `$n` placeholder text with a statement shape. Ordinary error text
does not change. When it acts, it also drops the text after the
statement.
- Code that checks `resultJson` against a closed key list must allow
`failureCause`. I found and updated the four lists on the Retry path; a
list I missed would refuse a Retry for a run with a generic failure.
- This is the run log / instance database path. It adds no Telemetry
event.
- The server failure log line still logs the raw error message
(unchanged behavior); only the board-facing text is changed.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file editing) and code execution for
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 02:14:53 -06:00
DottaandClaude Opus 5.5 693de27c90 fix(storage): wait out expired or rotated S3 credentials and re-resolve the key (#20456)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In cloud mode, run logs, provider traces, attachments and remote
workspace snapshots go to S3
> - The server uses the default AWS credential chain, and real keys
expire or rotate while runs are in flight
> - #20299 made runs wait out a storage outage, but a 403
`ExpiredToken`, `InvalidAccessKeyId` or `SignatureDoesNotMatch` was a
permanent `adapter_failed`
> - The SDK also caches a key until its stated expiration (forever for a
key with none), so a rotated key was never picked up
> - This pull request treats credential rejections as a storage wait and
re-resolves the key from the credential chain after a rejection
> - The benefit is that runs survive key expiry and rotation without
failed attempts or a manual Retry

## Linked Issues or Issue Description

Refs #20299 (storage outage wait). Harness that found this: #20455.

**What existing behavior does this improve?**
Runs wait out S3 outages (5xx, timeouts, resets) but fail at once when
the store rejects the server's credentials.

**Subsystem affected**
server storage (S3 provider) and heartbeat run completion.

**Current behavior**
Restart-chaos wave w-20261110T041556-s501 (`--storage s3
--storage-faults ST5`): one 41 s window of 403 `ExpiredToken` failed 4
of 6 tasks. Each run ended `adapter_failed` with "The provided token has
expired.", and the automatic retries failed within one second, so the
tasks needed a board Retry (R3 `retry_did_not_complete`).

**Proposed behavior**
A credential rejection is a transient infrastructure condition.
Critical-path storage calls wait it out like an outage, and a run that
stops on it gets the `storage_unavailable` wait retry. After each
rejection the S3 client re-resolves its key with `forceRefresh`, so the
next attempt signs with a rotated key. The run error names the
credentials. If they never come back, the exhausted-wait message tells
the operator to check the storage service and its credentials.

**Reason and benefit**
Expiring or rotated keys are normal in cloud deployments. They must not
fail work.

## What Changed

- `storage-unavailable.ts`: `ExpiredToken`, `ExpiredTokenException`,
`TokenRefreshRequired`, `InvalidToken`, `InvalidAccessKeyId`,
`InvalidClientTokenId` and `SignatureDoesNotMatch` answered by the store
with 400/403 count as storage unavailable. `AccessDenied` and a
credential chain that cannot load any key (`CredentialsProviderError`)
stay permanent. New credential-specific run message; the exhausted
message also names credentials.
- `s3-credential-refresh.ts` (new): S3 client middleware that calls the
client's credential provider with `forceRefresh` after a rejection, once
per credential generation (requests that signed with an already-replaced
key do not refresh again), then rethrows so the existing storage wait
owns the retry.
- `s3-provider.ts`: installs the middleware on both S3 clients.
- `run-completion.ts`: uses the credential-aware message.

## Verification

- `cd server && npx vitest run
src/__tests__/storage-s3-credential-rotation.test.ts src/storage/
src/__tests__/storage-outage-wait.test.ts
src/__tests__/storage-s3-transient-retry.test.ts
src/__tests__/storage-s3-failure-diagnostics.test.ts
src/services/heartbeat/run-completion.test.ts` (186 pass).
- The new `storage-s3-credential-rotation.test.ts` fails 5 of 6 cases on
master (red) and passes with this change (green). It covers a real key
rotation through the default credential chain, 403 windows for each
code, the message, and `AccessDenied` staying permanent.
- `cd server && npx tsc --noEmit` passes.
- `s3-credential-refresh.test.ts`: two parallel requests signed with the
old key, rejected 50 ms and 400 ms apart, cause one forced lookup (3
before the generation check, 2 after).
- Restart-chaos rerun on a build with this fix and #20455:
w-20261110T050044-s503 (`--env local --workloads A1,C1 --faults K1,K2
--tasks 8 --storage s3 --storage-faults ST5,ST5R --storage-fault-seconds
30-90`) passed 8/8 tasks (R1–R6 and T) with a 39 s `ExpiredToken` window
and an 85 s real key rotation (76 credential 403s). Before the fix,
w-20261110T041556-s501 failed 4/6 tasks.

## Risks

- Scope: calls with a Readable body (streamed `putObject`, buffered
multipart) cannot be resent, so they still fail on the first rejection,
as they do for outages. The middleware has already re-resolved the key,
so the caller's next attempt (the run-level wait retry for run work)
uses the new key.

- A key that the store rejects for good (for example a wrong secret) now
waits up to the storage wait budget (5 min in the call, then up to 10
run retries) before the board shows the exhausted message, instead of
failing at once.
- `forceRefresh` re-runs the credential chain after a rejection. Calls
are coalesced, so a burst of rejections causes one refresh.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 00:33:39 -06:00
DottaandClaude Opus 5.5 4a2eb920cf fix(storage): wait out a store outage on a run's critical path instead of failing it (#20299)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A cloud instance keeps run logs and evidence in object storage, so a
run touches the store as soon as it starts
> - In storage chaos wave `w-20261109T145628-s9902` (ST1: 75–176 s
windows where every S3 call returns 503), only 1 of 8 tasks finished. A
run starting or resuming during an outage failed within a second as
`adapter_failed`, generic recovery burned its attempts seconds apart,
the task was escalated to the board, and the user saw the raw S3 error
> - This pull request treats a transient storage error as an
infrastructure wait: the run is retried on a minute-scale schedule,
rapid recovery is suppressed, the board shows it as working, and the run
error says storage was unavailable
> - The benefit is that a store outage of several minutes delays work
instead of failing and stranding it

## Linked Issues or Issue Description

Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (master
`610b54f551`, `--storage s3`, ST1).

**Where the run fails.** At run start or resume, `run-output` `begin`
opens the run log: `getRunLogStore().begin` → object run log store →
object byte journal `claim` → S3 conditional PUT. The journal retries
only 3 times within about 300 ms, so a 503 escapes and fails the run as
`adapter_failed`. The async stack ends in the SDK middleware with no app
frame. `log_store` stays null on the failed run, which pins the claim.

**What followed.**
- `failRun` released the issue into generic immediate recovery. Recovery
retried 1–2 s apart inside the outage (t01: 14:58:33, :35, :59,
14:59:01), then stopped and opened a board-owned
`stranded_assigned_issue` action. These are the open recovery actions on
CHA-1, CHA-4, CHA-7 and CHA-8; a later Retry hit the outage again and
escalated the same way.
- The run error was the raw S3 text ("Injected storage outage").

**Slow store (ST2, `w-20261109T151946-s9903`).** With about 9.7 s of
added latency per call, the same claim hit the journal's 15 s
per-operation deadline (`AbortSignal.timeout(15_000)`). The run failed
after about 36 s with `AbortError: Request aborted` and burned 3 runs
the same way; 4 of 8 tasks needed a Retry.

t02/t03 (Retry parked with `process_identity_missing`) are a separate
native-recovery class and are handled in a follow-up.


**Daytona storage chaos ST1 (`w-20261109T161202-s9912`, 1/8):** with S3
returning 503 for 72 to 154 s at a time:
- Workspace restore failed at once (`remote_workspace_restore_failed:
step=snapshot_download; storage=provider_error`), three times in 40 s,
then `setup_failed`. It left an active recovery action behind.
- Runs reattached after a restart died within 2 s on the run-log claim,
then became reconciliation-required.
- Setup failed on other storage reads.
- 6 of 7 failures needed a UI Retry.

So the rule here is one class rule: a transient store error on a run's
critical path waits, it never fails the run.

## What Changed

- New `server/src/storage/storage-unavailable.ts`:
- `isStorageUnavailableError` reads the recorded storage failure
diagnostics first, then the SDK error shape. Unavailable means
429/500/502/503/504, the S3 throttling and timeout names, network codes
(`ECONNRESET`, `ECONNREFUSED`, `ETIMEDOUT`, `EPIPE`, `EAI_AGAIN`,
`ENOTFOUND`), or an `AbortError` whose cause is a `TimeoutError` (a
per-operation deadline). 403, 404 and 412 are not outages, and neither
is an abort caused by the run's own Stop.
- It defines the `storage_unavailable` code, the
`storage_unavailable_wait` retry reason, the retry policy (60 s × 10
attempts, about 10 minutes) and the user-facing message.
- `server/src/services/heartbeat/run-completion.ts`:
- `failRun` classifies such a failure as `storage_unavailable` with the
plain message.
- Unless native recovery owns the run, it schedules the storage wait
with `scheduleBoundedRetryForRun`, skips the interaction-continuation
retry, and releases the issue with `suppressImmediateRecovery`.
  - `failRunSetup` uses the same policy for this code.
- `server/src/services/heartbeat/retries.ts`: `storage_unavailable`
joins the transient infrastructure codes. It gets its own retry budget
and policy, like `runner_revision_mismatch`.
- `server/src/services/task-interruption-presentation.ts`:
`storage_unavailable_wait` is a resource wait, so the board shows the
task as working with no Retry while it waits.

- **One shared outage wait for the run's critical path**
(`server/src/storage/storage-unavailable.ts`):
- `retryThroughStorageOutage` retries a transient store failure with
backoff (2 s doubling to 30 s) until that storage operation's outage has
lasted 5 minutes, then gives the error back. The outermost wait of an
operation records its first failure, and nested and parallel waits
inside it share that start; no other request's success extends it.
Shared probes and shared reads run under their own scope, so one run's
Stop never cancels work another run waits on. A stop signal ends the
wait, and non-outage errors are never retried.
- Inside `waitOutStorageOutages`, every replayable S3 request waits
instead of failing: get, head, delete, buffered and file puts, and
multipart create, file parts and complete. A consumed upload stream is
never resent, and requests outside the scope keep failing fast.
- The scope covers run execution (`heartbeat.ts` `executeRun`: setup,
workspace restore, run-log claim and the run itself), plus
`loadWorkspaceSnapshotArchive` and `uploadWorkspaceSnapshot`, which
copy-back and recovery workers use too.
- The run-log claim retries as a whole, because the journal bounds each
operation with a 15 s deadline.
- Past the budget, the run takes the bounded `storage_unavailable` wait
below.
- The classifier follows the error's `cause` chain, so a restore failure
wrapping an outage counts. `failRunSetup` checks storage before restore
failures, so that case takes the storage wait instead of two quick
retries and an escalation (the R6 leak).
- `object-byte-journal.ts`: when its deadline expires, the journal
rethrows the recorded storage error instead of a bare `TimeoutError`.

## Verification

Red/green (each new test fails against master and passes with this PR):
- `run-completion.test.ts` "waits a minute and suppresses rapid recovery
when a run fails because storage is unavailable (storage chaos ST1)". An
S3-shaped 503 records `storage_unavailable` with the plain message (no
raw S3 text), schedules `STORAGE_UNAVAILABLE_RETRY`, releases with
`suppressImmediateRecovery: true`, and skips the continuation retry.
"keeps an ordinary adapter failure on the normal recovery path" guards
the default.
- `task-interruption-query.test.ts` "does not present an overdue
storage_unavailable_wait wait as a failed attempt": working, `canRetry:
false`.
- `storage-unavailable.test.ts`: which errors count as an outage and
which do not, including recorded diagnostics, a deadline-aborted call
(ST2), and a Stop-aborted call that must not count.

Static checks:
- `tsc --noEmit` passes for `server`.

Live verification: Phase 3 will rerun ST1 (seed 9902) with the MinIO
fault proxy after merge.

- `server/src/__tests__/storage-outage-wait.test.ts` runs against a
local S3 that answers 503 for 3 s:
- Inside the scope, a snapshot `getObject` and a run-log conditional PUT
succeed once the store is back.
  - Outside it, a request fails fast.
  - A stop ends the wait.
- Unit cases cover the budget, non-outage errors, and a
`RemoteWorkspaceRestoreError` wrapping an outage.
- `run-completion.test.ts`: a setup failure from a restore that wrapped
an outage takes the storage wait (red before).
- `object-byte-journal.test.ts`: a real `claim` whose head PUT hits its
deadline is classified as a storage outage (red before).

## Risks

- A storage outage now holds a task for up to about 10 minutes of
automatic waits instead of escalating within seconds. A longer outage
still ends in the existing board recovery.
- A run whose failure is misclassified as storage-unavailable waits
instead of using fast recovery. The classifier accepts only server-side
or network failures from the storage SDK, never 4xx answers.
- No schema change.

- A run can now wait until a storage outage has lasted 5 minutes before
it fails over to the bounded wait. It shows as working throughout, and
Stop ends the wait.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 18:15:48 -06:00
Devin FoleyandClaude Opus 5.5 90ec0b9673 fix(sentry): report a reviewed chat setup wait only after its retry decision (#20205)
A reviewed chat attestation that exhausts its short lock or delivery wait fails setup with reviewed_chat_execution_binding_not_ready, and the bounded retry then owns recovery. The failed-status write sent the Sentry report before the retry was scheduled, so a wait that the retry recovered still created an unexpected-failure event.

The setup path now marks that write as pending and reports once after the retry decision. A scheduled retry keeps the failure local; an exhausted, unscheduled or failed retry decision still reports. Other setup failures keep their report on the failed-status write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 12:32:15 -08:00
DottaandClaude Opus 5.5 ad2304dc88 fix(chat): recover a held conversation turn's answer after a restart (#20204)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People talk to agents in Agent Chat. The agent's final message
becomes the chat reply.
> - A chat turn can end while its task stays held: a pending approval,
an execution stage or a dependency blocker keeps the status.
> - #20179 made live run completion post that held turn's answer. It
added `conversationTurnAnswered()`.
> - Background presentation recovery still required a
`conversation_turn_finished` decision and a `done` result.
> - If the server restarts between the status commit and the live
presentation, recovery skips the held turn. The person never gets an
answer.
> - Restarts are product failures. A committed answer must not be lost.
> - This pull request makes recovery use the same answered rule as the
live path, through one shared helper.
> - The benefit is that a held turn's answer reaches the chat exactly
once, with or without a restart.

## Linked Issues or Issue Description

Refs #20179 (live fix; this PR closes the "Known gap" in its Risks
section).

**What happened**

In an Agent Chat, the agent resubmitted a hire approval and ended its
turn with a yielded result. The finalizer committed
`governed_response_waiting` and moved the task to `in_review`. The
server restarted before run completion posted the reply. Background
presentation recovery found the run as a candidate, but declined it,
because it accepted only `conversation_turn_finished` with a `done`
result. No reply was posted. The run stayed a pending presentation
candidate.

**Expected behavior**

After a restart, recovery posts the held turn's final answer exactly
once. It never posts a second copy when the live path already presented
the answer. When the turn's own question or confirmation card presents
it, recovery posts nothing and records that decision, as the live path
does.

**Steps to reproduce**

1. Open an Agent Chat. Link a pending hire approval to the chat task.
2. Let the agent end the turn with a yielded result and a
`response_wake` continuation. The status decision commits as
`governed_response_waiting`.
3. Stop the server after the status commit and before run completion
presents the reply.
4. Start the server. Presentation recovery runs.
5. See that the chat shows no reply.

## What Changed

- `server/src/services/native-runtime/native-governance.ts`: add
`conversationTurnPresentsAnswer()`. It is the one rule for whether a
successful conversation turn's prose is its answer:
`conversation_turn_finished`, or a committed decision with no pending
card of the turn (`conversationTurnAnswered()`).
- `server/src/services/heartbeat/run-completion.ts`: the live path calls
the shared helper. No change in behavior.
- `server/src/services/native-runtime/native-run-finalizer.ts`
(`recoverCommittedNativeChatResponse`):
- A current conversation turn no longer needs
`conversation_turn_finished`. The turn must still be the unsettled
current turn: same agent, user, session generation, `conversationState:
active`, and no other execution run. A `conversation_turn_finished` turn
still needs `in_progress`.
- A conversation turn with a `yielded` result is accepted next to
`done`.
  - The resolver gets `conversationTurnFinished` from the shared helper.
- When the turn's own card presents it, recovery records
`presentationDecision: none` (reason
`committed_conversation_interaction_presented`) and settles the
conversation turn, like the live path. The run then leaves the pending
presentation set. Before, such a run stayed a candidate.
-
`server/src/services/native-runtime/native-response-presentation-query.ts`:
the candidate prefilter matches the same turns: any committed reason for
an active conversation turn, and `yielded` conversation results.
-
`server/src/services/native-runtime/agent-chat-response-recovery.test.ts`:
restart regressions with the real finalizer and a real database.
- `doc/execution-semantics.md`: state that recovery applies the same
rule.

## Verification

- Failing first. With only the new tests applied, 5 of 7 new cases
failed: pending approval, revision-requested approval, execution stage
(all yielded), pending approval with `done`, and the own-card case. The
run was a candidate, but recovery posted nothing and did not settle it.
- After the fix: `cd server && npx vitest run
src/services/native-runtime/agent-chat-response-recovery.test.ts
src/services/heartbeat/run-completion.test.ts
src/__tests__/idle-native-finalization.test.ts
src/services/presentation-ownership-read.test.ts
src/services/chat-monitor-continuation.test.ts
src/services/native-runtime/native-preserved-monitor-wait.test.ts
src/services/native-runtime/native-response-publication-retry.test.ts
src/services/heartbeat/run-lifecycle.test.ts
src/__tests__/scoped-deadline-work.test.ts
src/services/native-runtime/paperclip-control-plane-port.test.ts
src/__tests__/heartbeat-run-summary.test.ts
src/__tests__/agent-conversations.test.ts`: 12 files, 627 tests passed.
- The new cases check:
- Held turns: the run is a candidate, the check posts the exact final
answer once, the presentation decision is recorded, the run leaves the
candidate set, and a second recovery returns false with one comment.
  - Live already presented: no candidate, no second comment.
- Own question card: no comment, `presentationDecision: none`, the card
stays pending, the conversation settles to `waiting`, and the run leaves
the candidate set.
- Previous-release live decision `none` (`yielded_control_plane_wait`):
kept, not re-presented.
- `tsc --noEmit -p server`: no errors in changed files. Four unrelated
errors remain in this local worktree (`esbuild` not linked in
`native-harness-prestage.test.ts`, and a linked-package type mismatch in
`run-dispatch/adapters/postgres.ts`).
- Turn start: no change. The helper runs only in run completion (after
the provider turn ends) and in background presentation recovery. Nothing
new runs before `turn.start`, and no files move. The live path keeps the
same queries: it short-circuits on `conversation_turn_finished`, and
otherwise runs the same card read as before.

## Risks

- Low to medium. Recovery can now post the answer of a conversation turn
whose task is held. This matches the live path since #20179.
- A card-presented turn found by recovery is now settled with
`presentationDecision: none`. Before, it stayed a pending candidate and
kept the presentation worker (and idle-sleep checks) busy.
- A `conversation_turn_finished` turn keeps the `in_progress` check.
Held turns accept any status their gate committed. The unsettled-turn
checks (active conversation state, session generation, execution run)
still block stale or superseded turns.
- Previous-release state: no persisted shape changes. A run committed by
the previous release with no saved presentation decision (the restart
window) is recovered by the new code. The test fixture uses the
unchanged finalizer, so it saves exactly the previous release's shape. A
run whose live path already recorded a decision keeps it. A held turn
that the previous release recorded as `yielded_control_plane_wait` is
not re-presented, so no late duplicate or stale reply appears.
- Not changed: generic stranded-work recovery in `recovery/service.ts`
still skips only `conversation_turn_finished` runs during the
publication window.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell,
file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 01:08:10 -06:00
DottaandClaude Opus 5.5 95bf90bd2c fix(chat): post a conversation turn's answer while an approval holds the task (#20179)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People talk to agents in chat. The agent's final message becomes the
chat reply.
> - A chat turn can end while the task stays held. For example, the
agent resubmits a hire approval and the approval is pending.
> - The finalizer keeps the task in review for the approval
(`governed_response_waiting`). That decision is not
`conversation_turn_finished`.
> - Run completion then treats the yielded turn as a control-plane wait.
It records `presentationDecision: none` and posts no comment.
> - The person never sees the agent's answer. The chat shows nothing
after the request.
> - This pull request counts a committed conversation turn as answered
unless its own pending question or confirmation card presents it.
> - The benefit is that the person always gets the reply, whatever state
the task keeps.

## Linked Issues or Issue Description

Refs #19971 (post the full final reply). That fix covers the reply text.
This fix covers replies that were never posted.

**What happened**

In a chat, the person asked the CEO agent to change a proposed hire to a
cheaper model and send it back for approval. The agent resubmitted the
approval correctly. It called `paperclip_finish`, and Paperclip accepted
the report with "task is still waiting for approval". The run succeeded
and the task moved to `in_review`. The run event
`run.presentation.resolved` recorded `commentAction: none` (reason
`yielded_control_plane_wait` on master). The agent's final message never
reached the chat.

**Expected behavior**

A chat turn's accepted final answer is always posted to the
conversation. An approval, an execution stage or a dependency blocker
can hold the task. It must not hide the reply. Only a question or
confirmation card that the turn itself left pending presents the turn
instead of its prose.

**Steps to reproduce**

1. Open an Agent Chat with an agent. Link a hire approval in
`revision_requested` to the chat task.
2. Ask the agent to change the hire and resubmit it.
3. The agent resubmits the approval and ends the turn with a yielded
result and a response wake.
4. See that the chat shows no reply. The run is `succeeded` and its
presentation decision is `none`.

## What Changed

- `server/src/services/native-runtime/native-governance.ts`: move the
pending-card query of `pendingNativeGovernance()` into
`pendingNativeInteraction()`, with no change in behavior. Add
`conversationTurnAnswered()`. It returns false only when a pending
thread interaction (the same predicate as status arbitration) gates the
turn. It checks cards on their own, so a pending execution stage does
not hide a card.
- `server/src/services/heartbeat/run-completion.ts`: a successful
conversation run whose status decision committed now passes
`conversationTurnFinished: true` to the presentation resolver, unless
`conversationTurnAnswered()` is false. The existing
`conversation_turn_finished` path is unchanged.
- `server/src/services/heartbeat/run-completion.test.ts`: add a
regression table with a real database. A pending approval, a
revision-requested approval, a pending execution stage and a dependency
blocker now post the answer. A finished turn still posts. The turn's own
question card still owns the response, also when an execution stage is
pending.
- `doc/execution-semantics.md`: state the rule.

Other finish outcomes I checked:

- `done` or `needs_review` with a gate (`governed_gate_pending`): the
result is not yielded, so the reply was already posted.
- `blocked` (`paperclip_block`, waiting on a person or another task):
not yielded, so the reply was already posted.
- Yielded with a dependency blocker
(`durable_dependency_blocker_bound`), or with
`record_finalization_error` (`prior_status_preserved_no_live_path`,
`monitor_wait_authority_lost`): the reply was also dropped. This fix
covers them, because the rule does not depend on the reason code.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts`: 118 passed. Before the
fix, 4 new cases failed (both approval states, execution stage,
dependency blocker). The finished-turn and question-card cases passed
both before and after.
- `cd server && npx vitest run
src/__tests__/heartbeat-run-summary.test.ts
src/services/heartbeat/run-comment-selection.test.ts
src/__tests__/slack-conversation-lifecycle.test.ts
src/services/native-runtime/agent-chat-response-recovery.test.ts
src/__tests__/chat-run-milestone-governed-work.test.ts
src/__tests__/agent-conversations.test.ts`: all passed.
- `tsc --noEmit -p server`: no errors in the changed files. Unrelated
errors in this worktree come from an unbuilt local Runner package.
- An end-to-end chat eval run on the case branch is in progress. I will
add its result here.

## Risks

- Low risk. The change only affects successful Agent Chat runs with a
committed status decision. Ordinary tasks and external chat (Slack,
GitHub) keep their own paths. External chat returns before this check.
- A turn that yields while an approval is pending now posts its prose.
The approval stays pending and the task status does not change.
- One extra small read (the task row and pending gates) runs after a
conversation turn that is not `conversation_turn_finished`. It runs
after the provider turn ends, so it does not add time before the first
token.
- Previous-release state: no persisted shape changes. The fix reads
`finalizationPhase` and `finalizationReasonCode`, which the previous
release already writes. A run in flight at the deploy completes on the
new code and posts its reply. A run that completed before the deploy
keeps its recorded decision; nothing re-presents it.
- Known gap, unchanged: background presentation recovery for
conversations still requires `conversation_turn_finished` with a `done`
result. If the server restarts between the status commit and the live
presentation, a held yielded turn is not re-posted. That is a separate
follow-up.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell,
file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 23:17:36 -06:00
Nicky Leach 45c2bd94c6 fix(native): name the interrupted attempt when an exact resume is refused (#19709) 2026-11-07 06:21:46 -08:00
Devin FoleyandPaperclip bf12b2d8ad fix: retry reviewed chat setup after temporary contention (#19787)
Retry reviewed chat preparation with the original accepted input.

Bound retries for authorization proof waits that expire before provider
work. Preserve the original answer receipt and queue later wakes
separately. Keep plan confirmation semantics by assigning the interaction
marker only to question answer delivery and legacy question payloads.

Verification: 433 tests across four backend suites, all 17 project-chat
browser tests, and all server source/test typecheck layers passed. The
plan approval regression was reproduced on the previous head before the
fix. All applicable CI passed and exact-head Greptile Apex scored 5/5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-07 02:14:42 -08:00
DottaandClaude Opus 5.5 c97c0dd248 fix(workspaces): give run preparation its own source lock scope (#19734)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A run prepares its task workspace under per-source advisory locks
(`workspace-source-lock.ts`). The locks live in a transaction on a
dedicated connection. An `AsyncLocalStorage` scope lets nested calls
reuse that transaction.
> - Timers and detached promises keep the async context that created
them. A native recovery timer or dispatch can start a run from inside
another operation's source scope.
> - When that run prepares its workspace, it sees the other scope as
"held". If that scope already committed, its dedicated connection is
closed. The lock query then fails with `Failed query: select
pg_try_advisory_xact_lock(...)` and cause `CONNECTION_CLOSED`. If that
scope is still open, the run's lock is tied to the other transaction and
is released early.
> - In a restart chaos wave, a reattached follow-up run failed setup
this way 50 ms after "Recovering the existing native runner process".
The setup-failure log did not show the driver cause.
> - This pull request makes only a live scope count as held, and makes
run workspace preparation take its locks in an independent scope.
> - The benefit is that a recovered or dispatched run gets its own
source lock, and a lost-connection failure is diagnosable from the log.

## Linked Issues or Issue Description

**What happened?**
After a control-plane restart, a native run was reattached
(`reattach_remote_runner`). Its workspace preparation failed at once
with `Failed query: select
pg_try_advisory_xact_lock(hashtextextended($1, 0)) as acquired params:
isolated_repository_source:<company>:<issue>`. The run ended
`failed`/`setup_failed`. The stack showed the "already held scope"
branch of `withWorkspaceSourceLocks`, not a new acquisition. The same
failure occurred earlier in the same wave on a different issue without a
restart.

**Steps to reproduce**
1. Call `withIsolatedRepositorySourceLock` for issue A. In its action,
start a detached promise that later calls
`withIsolatedRepositorySourceLock` for issue B.
2. Let the outer action return so its transaction commits.
3. Let the detached call run. It fails with `Failed query ...
pg_try_advisory_xact_lock`, cause `CONNECTION_CLOSED`.

**Expected behavior**
Work that starts after a source scope ends, or independent work such as
a run's preparation, acquires its own locks on its own connection.

**Environment**
Restart chaos campaign on Daytona (OpenCode, workload C1, fault K1,
comment trigger), embedded Postgres. Reproduced locally on embedded
Postgres by the new tests.

Related: #14910 (deferred comment wake reopens a closed issue). This PR
does not change wake promotion.

## What Changed

- `server/src/services/workspace-source-lock.ts`: a scope has an
`active` flag. The flag is cleared when its transaction ends.
`withWorkspaceSourceLocks`, `retainManagedProjectSource`,
`retainCurrentWorkspaceSource`, and `assertCurrentWorkspaceSourceHeld`
use only a live scope.
- `server/src/services/workspace-source-lock.ts`: new
`withIndependentWorkspaceSourceLocks`. It acquires locks in a new scope
and ignores an inherited one.
- `server/src/services/heartbeat/workspace-preparation.ts`: run
workspace preparation uses `withIndependentWorkspaceSourceLocks`.
- `server/src/services/heartbeat/run-completion.ts`: the "heartbeat
execution setup failed" log records a bounded driver `causeCode` (for
example `CONNECTION_CLOSED` or a SQLSTATE). It does not log the cause
text.
- Tests: two embedded-Postgres tests in
`isolated-remote-workspace-source-lock.test.ts`, one unit test in
`run-completion.test.ts`.

## Verification

- `cd server && npx vitest run
src/services/isolated-remote-workspace-source-lock.test.ts` — 4 passed.
- Red (old scope rule): "gives work detached from a finished scope its
own source lock" failed with `Failed query: select
pg_try_advisory_xact_lock(...)`, cause `CONNECTION_CLOSED`. This is the
production error.
- Red: "keeps an independent preparation's lock after the scope that
started it commits" failed because the lock was released with the other
transaction.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts` — 95 passed.
- `cd server && npx vitest run
src/services/remote-workspace-online-source.test.ts
src/services/heartbeat/workspace-preparation.test.ts
src/services/isolated-repository-rehome.test.ts
src/services/isolated-task-directory-retention.test.ts
src/services/isolated-task-directory.test.ts
src/__tests__/execution-workspace-reopen.test.ts` — all passed.
- `cd server && npx tsc --noEmit` — clean.
- No live Daytona replay was run.

## Risks

- Low. A nested call inside a live scope behaves as before.
- A call that inherited an ended scope now opens its own dedicated
connection and lock transaction. Before, it failed.
- Run preparation never joins another operation's lock scope. No caller
relies on that today. If one did, it would now take a second,
independent lock and could see `workspace_source_busy` instead of
silently sharing.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, embedded Postgres tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 23:08:57 -06:00
DottaandPaperclip b94963f91e fix: retry temporary repository catalog failures before dispatch (#19690)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - A task must check repository access before it starts agent work.
> - Repository discovery can fail while the user's existing grant
remains valid.
> - Discovery previously replaced a known temporary provider refusal
with an empty catalog.
> - The task then reported lost access and did not schedule recovery.
> - This change keeps temporary discovery failures distinct and uses the
existing bounded retry owner.
> - Each retry must pass a fresh access check before it can use the
saved repository.

## Linked Issues or Issue Description

**What happened?**

A task with a saved catalog repository stops with an access denial when
GitHub returns an exhausted-quota response or HTTP 503. The task does
not start provider work, but the discovery layer discards the temporary
failure and returns an empty catalog.

**Expected behavior**

Known temporary failures before provider dispatch use the existing
bounded infrastructure retry. The successor checks the same responsible
user's current grants and reads the saved files only after fresh
discovery succeeds. A completed denial remains terminal.

**Steps to reproduce**

1. Bind a task to an authorized catalog repository with saved local
changes.
2. Make its authorized GitHub discovery return HTTP 403 with zero
remaining quota, or HTTP 503.
3. Let discovery recover before the existing retry deadline.
4. Observe one successful successor using the original repository bytes.

**Paperclip version or commit**

Reproduced on master at `b8d94b3ad1`. A search found no existing PR for
this temporary catalog failure.

## What Changed

- Keep a safe typed failure for known HTTP availability and quota
responses. Honor the later provider deadline and bound it to one day.
- Propagate that failure only when fresh discovery cannot find a
required repository. Preserve successful partial catalogs and the
existing connection setup error contract.
- Classify the failure only before provider dispatch. Reuse existing
bounded retry, interaction ownership, and bootstrap settlement rules.
- Keep current grant checks, saved repository selection, normal Stop,
and provider-started failure policy.
- Keep provider deadlines when accepted interactions use their own retry
owner. Recognize secondary HTTP 403 rate limits with valid retry advice
while keeping ordinary permission denials terminal. This matches [GitHub
rate-limit
guidance](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api).
- Keep direct repository request responses outside the automatic
bootstrap retry contract. Test successful partial discovery and
missing-required-repository recovery with two independent eligible
connections.
- Document task repository discovery recovery.

## Verification

- Behavioral RED: real heartbeat and PostgreSQL fixtures reached actual
HTTP 403 and503 discovery responses, started no adapter, and found zero
successors instead of one. Both cases failed before the production
change.
- Behavioral GREEN: both cases waited for the real retry deadline, used
the normal queue worker, ran the controlled adapter once after fresh
discovery, and read the original saved bytes. Two cases passed in 74.73
seconds. No external model or provider request was made.
- Deadline RED/GREEN: a combined Retry-After and quota-reset response
initially selected the earlier deadline. The regression now selects the
later deadline.
- Initial repair at `d8d4714a18` (same production source as
`018e8c52d345d79972d7907468080dc9908c299f`): seven affected files
passed, 69 cases in 185.16 seconds. That test-only cleanup then passed
all five real-database acceptance cases in 134.76 seconds.
- Commands: `pnpm exec vitest run` with
`heartbeat-repository-catalog-recovery.test.ts`,
`github-catalog-timing.test.ts`, `github-grant-metadata.test.ts`,
`project-repositories-persistence.test.ts`,
`task-workspace-repositories-persistence.test.ts`,
`github-repository-catalog-failure.test.ts`, and
`services/heartbeat/retries.test.ts` from the server source tree. The
acceptance file also ran fresh at `018e8c52d3`. A fresh five-case run
after the P1 repairs passed at `f490ca6b78` in 122.06 seconds. The final
eight-case file passed fresh at
`e5b2efab11919467850f55fd6345317132b9b672` in 124.62 seconds, using real
owned embedded PostgreSQL. It covers automatic recovery with saved
bytes, genuine denial, provider-started classification, normal Stop, the
direct HTTP route, and both partial-catalog outcomes.
- Review repairs at `f490ca6b787cf77df96d3d6edfb2ab9115eb19a6`: both
reported P1s reproduced RED. The real database interaction scheduler
chose a retry after 24 seconds before the provider deadline of one hour.
A secondary HTTP 403 with valid Retry-After lost its retry
classification. The narrow fixes then passed 44 cases across scheduler,
header, and metadata-contract files in 37.66 seconds.
- A real direct request through the task workspace router and normal
error middleware reproduced a 500 response instead of the existing 403.
Required-ID retry mode now applies only to admission and its rechecks.
The route and both two-connection partial-catalog cases then passed in
29.09 seconds. Both partial cases use actual owned grants and credential
bindings.
- All seven server type layers passed again at final head
`e5b2efab11919467850f55fd6345317132b9b672`. The affected test type layer
passed again after the test-only cleanup. Environment: macOS arm64, Node
26.4.0, pinned pnpm 9.15.4, Vitest 5.0.3, owned disposable PostgreSQL 18
on loopback. The current checkout's shared packages, plugin SDK, and
Runner were built first.
- One local acceptance run under concurrent owned test/typecheck load
hit the unchanged 15-second Stop-case timeout. Its four other cases
passed. The unchanged complete file then passed serially. The timeout
receipt remains recorded; no assertion or timeout was weakened.
- A provider-started failure retains the existing generic adapter retry
policy. It does not receive the catalog no-provider certificate or a
provider retry deadline. Fixture cleanup cancels its scheduled retry
through the normal owner.
- Final head `e5b2efab11919467850f55fd6345317132b9b672` has 59
successful checks, 3 allowed skips, no pending or failed check, and
Greptile 5/5 with no open findings. [Completed CI
workflow](https://github.com/paperclipai/paperclip/actions/runs/42322202977).
- Additional local diagnostic for the late recovery-owner advisory: on
unchanged `e5b2efab`, the dedicated scheduler returned `retry_exhausted`
with attempt 3/max 2. The normal stranded sweeper then escalated once,
created no extra run, and started no adapter, even after catalog access
recovered. One case passed in 17.32 seconds. The existing
failed-predecessor path delegates to that bounded scheduler and returns;
it does not enter a new generic retry budget. No production registry
change was made. This diagnostic was not a CI case. Its local harness is
preserved separately at `74e3854eea3f7c4121fdcd9c8a8ad522cd8c7477` on
`codex/catalog-exhaustion-acceptance`; its production source is
identical to `e5b2efab`.
- Module boundary and whitespace checks passed.

## Risks

- A temporary failure from another eligible connection can delay a truly
missing repository for the existing limit of two attempts. It cannot
grant access.
- Unknown network errors and genuine access denials retain their
existing behavior. This change handles known HTTP failures only.
- Historical provider quota causality is unproven. This repair has a
separate local producer regression. It does not rewrite past failures or
queued user intents.
- Acceptance uses real local database and heartbeat owners with
controlled HTTP and adapter fixtures. It does not claim a live
native-provider recovery.

## Model Used

OpenAI GPT-6.1 Codex (`gpt-6.1-sol`), with high reasoning, repository
tools, code execution, and controlled local PostgreSQL tests. The host
did not expose a context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 19:29:40 -06:00
DottaandPaperclip 0819f4ea78 fix(native): verify controller ownership before retrying saved turns (#19638)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native task turns can keep a provider conversation from an earlier
run.
> - A copied conversation checkpoint names the new run before its
controller has attached.
> - A failed attachment can make recovery treat that copied identity as
current execution authority.
> - The current run needs its own durable controller checkpoint, or a
valid staged attachment, before exact resume.
> - This pull request checks that ownership and uses the normal Retry
path to continue a saved user turn after verified cleanup.
> - The task can continue once with its saved work and history.

## Linked Issues or Issue Description

Refs #19399 and #18860.

**What happened?**

A new user comment resumed a completed native conversation. The retained
runner process was gone before the new controller attached. The new run
still had a copied provider session ID. Recovery selected exact resume
for that new run and then failed its unchanged owner and lease checks.
Ordinary first-attempt failure settlement also omitted the saved-user
Retry handoff that restart recovery already uses.

**Expected behavior**

Exact resume requires current controller evidence. A saved user turn can
request one normal Retry after this infrastructure failure. The normal
path must verify provider stop, saved files, current user authority and
all execution gates before it starts a fresh session.

**Steps to reproduce**

1. Finish a native warm run and retain its provider checkpoint.
2. Admit a user comment as a new run and copy the provider checkpoint
through the real resume producer.
3. Stop the retained runner before attachment. The real remote process
check reports that the process is not running.
4. Observe that the copied provider ID selects exact current-run resume.
Observe that ordinary failure settlement saves no Retry request.

## What Changed

- Check a bounded encrypted controller header for fenced remote Runner
recovery. Keep local Runner recovery on its existing disk-backed path.
Match the run, conversation and runner identity. Preserve the existing
staged-attachment path.
- Share the existing saved-user Retry admission between restart recovery
and ordinary terminal native settlement. Keep its exact infrastructure
errors, user wake, comment, provider-turn and retry-chain checks.
- Add real PostgreSQL and stopped-process regressions. Verify malformed
and foreign checkpoint holds, current-owner resume, user authority and
Stop. Verify saved-file recovery creates one fresh-session successor.
- Document the ownership and Retry contract.

## Verification

- RED: the real checkpoint rebind producer and stopped-process check
selected exact current-run resume without a current controller
checkpoint.
- RED: ordinary settlement created zero Retry receipts for the real
native failure code; expected one.
- Review RED: a real current local controller file existed with no
database checkpoint rows. The initial gate returned ambiguous state. The
fix takes a required trusted execution-target kind and applies the
database requirement only to the remote persistence path.
- Initial local revision: `3662055880ccf8e09c3c50a340897e16c78627e7`,
macOS ARM64, Node 26.4.0, pnpm 9.15.4, embedded PostgreSQL.
- Initial server Vitest: 42 selected cases passed across controller
recovery integration, remote restart adoption, ordinary run completion
and retained workspace recovery. The cases cover the stopped-process
producer, current/foreign/malformed ownership, automatic user
continuation, Stop, unsettled turns, system wakes and a manual Retry on
first use.
- `pnpm --filter @paperclipai/server typecheck`: passed with this
worktree's Runner, shared and SDK builds and all seven server layers.
- `node scripts/check-module-boundaries.mjs`, whitespace checks and the
added-code-comment audit: passed.
- Review correction: `8c795e2e0e44b5209da3e228593c5cd35e948cae`. The
real local/remote controller case passed against a fresh PostgreSQL
cluster. It verifies local disk recovery without database rows, remote
current-owner resume, and remote missing/foreign/malformed holds. Fresh
`node scripts/typecheck-layers.mjs` passed all seven server layers. The
existing offline chat reader now requests metadata only and retains its
full independent native-state safety check.
- A broader local recheck hit PostgreSQL setup/cleanup hook timeouts.
These are retained as failed setup attempts; the fresh-cluster
regression above passed. Required CI supplies broad final-head coverage.
- Settlement-order correction:
`dafb721c3bbb7ae86fbc3c9c6c953ace29ca088b`. The real PostgreSQL
regression first proved both failures: Stop after an already-failed row
created one Retry instead of zero; a DB-native promotion with an old
legacy caller snapshot created zero instead of one. All six settlement
scenarios now pass. The hook requires the local failed outcome and the
canonical settled native row. Fresh seven-layer server typecheck passed
again.
- Required shared fixture base integrated at
`d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. The six runner-skills
fixture cases and all seven server typecheck layers passed. Required CI
passed for this head:
[workflow](https://github.com/paperclipai/paperclip/actions/runs/42309419592),
including all server layers, tests, build, Runner verification and E2E.
The final snapshot has 57 successful checks, three intended skips and
one neutral result. Current-head Greptile is 5/5; both concrete review
threads are resolved. The independent re-review ended without a final
response; its prior Stop finding has a real RED/GREEN repair.

The focused command was run from `server/`:

```sh
../node_modules/.bin/vitest run --config vitest.config.ts src/services/heartbeat/run-completion.test.ts src/services/native-runtime/native-controller-recovery.integration.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/workspace-recovery.test.ts -t 'replays saved user intent|does not assign current-run resume authority|remote controller restart adoption|continues a user turn once|does not continue automatically'
```

## Risks

- A terminal failure of a fenced Runner adds one exact-run header read.
The read selects one encrypted header and limits inspection to 1 MiB.
Invalid or missing ownership evidence stays held.
- Saved user input can request one automatic Retry. It uses normal
preservation, current authorization, budget, approval, Stop and cleanup
gates. Unsettled provider turns and system wakes do not start an
automatic successor.
- This PR changes no provider process, credential, model or lease
authority checks.

## Model Used

OpenAI GPT-6 family through Codex, with reasoning, tool use and code
execution. The model is inherited; no override was used. The exact
serving model ID and context-window size are not exposed by this
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 17:31:23 -06:00
DottaandClaude Opus 5.5 a4bb059f59 fix(heartbeat): retry transient sandbox provider lease refusals and explain the block (#19412)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task runs in an execution environment. A managed sandbox
environment gets a lease from a sandbox provider (for example Daytona)
before the agent starts.
> - When the provider answered sandbox create with a temporary error
(HTTP 502), the run failed as the generic `setup_failed`.
> - `setup_failed` is the fail-closed code for unknown setup errors. It
gets no automatic retry, so the task went to `blocked` at once. The task
showed only "Couldn't finish this attempt".
> - A user had to find and press Retry. The Retry then worked, which
shows that the failure was temporary.
> - This pull request classifies temporary provider refusals during
lease acquisition from structured fields. It retries them within the
existing bounded bootstrap budget, and it explains the block when the
retries are used up.
> - The benefit is that a short provider outage no longer stops a task.
When the outage continues, the task shows the cause and a Retry that
works.

## Linked Issues or Issue Description

**What happened?**
A task was assigned to an agent that uses a managed Daytona sandbox
environment. Sandbox creation returned `HTTP 502`, and the cleanup of
the provisional sandbox could not be confirmed. The run failed with
`setup_failed` and the error `Failed to acquire lease for environment
"Paperclip Sandbox" (sandbox): Daytona sandbox creation failed (HTTP
502); cleanup could not be confirmed for paperclip-create-…`. Three
seconds later the task was `blocked`. No automatic retry was scheduled.
The task showed no specific reason. A manual Retry started a new run
that completed normally.

**Expected behavior**
A temporary provider error (HTTP 5xx, 429, timeout) before the agent
starts must retry automatically with bounded backoff. The task must show
why it waits. If the retries are used up, the task must show the cause,
a next action, and a Retry that works on the first click.

**Steps to reproduce**
1. Configure an agent with a sandbox environment.
2. Make the provider return HTTP 502 on sandbox create.
3. Assign a task to the agent.
4. See that the run fails with `setup_failed` and the task goes to
`blocked` with no retry.

**Paperclip version or commit**
`master` at `612da1e8e0`.

## What Changed

- Plugin SDK: an `environmentAcquireLease` error now carries a closed
diagnostic envelope (HTTP status or a closed errno code) across the
worker RPC. It takes the status of the create call only, not of the
compensating cleanup lookup. The existing creation-cleanup ownership
envelope is unchanged. New module: `environment-acquire-error.ts`. The
resume diagnostic helpers are shared.
- Server: new `EnvironmentProviderUnavailableError` and code
`environment_provider_unavailable`. The environment run orchestrator
sets it as the cause of `lease_acquire_failed` only when structured
evidence shows a transient status (408, 425, 429, 500, 502, 503, 504, or
a closed connection errno). It never parses message text. Saved-sandbox
resume and stopped-worker refusals keep their own codes.
- Setup failure (`failRunSetup`): a fresh, pre-dispatch run records
`environment_provider_unavailable` with a `providerFailure` (phase
`startup`, retryability `transient`). The run uses the bounded bootstrap
retry (30 s, then 60 s). An accepted interaction continuation keeps its
own continuation retry, as for `plugin_worker_unavailable`.
- Recovery: the new code is registered with the other bootstrap codes.
It is not sent to the generic continuation retry. It does not create an
execution-reconciliation hold, because no provider work started. On
exhaustion the stranded notice has the title "Sandbox provider was
temporarily unavailable" and a specific next action.
- UI: `ProviderFailureNotice` labels the `sandbox` provider as "Sandbox
provider". The task notice shows the provider failure, "Paperclip will
retry automatically" while a retry is scheduled, and Retry after
exhaustion.
- Docs: `doc/execution-semantics.md` describes the policy and the
sandbox ownership rule.

## Verification

Red before the fix, green after the fix:

- `server/src/__tests__/heartbeat-process-recovery.test.ts` › "transient
sandbox provider refusal during lease acquisition":
- "retries automatically and runs exactly one successful successor
without user action": the real heartbeat. The lease runtime returns a
Daytona 502 once, then delegates. Before the fix: one `failed` run and
no successor. After the fix: one `scheduled_retry` successor with no
hold and a `retry_scheduled` presentation. After promotion, exactly one
successor succeeds.
- "explains the block with a working Retry once automatic retries are
used up": three refusals. The task is `blocked` with a "Sandbox provider
was temporarily unavailable" notice. The presentation is `{ state:
"failed", canRetry: true, providerFailure: { httpStatus: 502 } }`, and
no execution-reconciliation hold exists. One `requestTaskRetry` is
accepted, and its run succeeds.
- `server/src/services/heartbeat/run-completion.test.ts` › "routes a …
sandbox provider lease refusal through bounded bootstrap recovery": the
real orchestrator wrapping. Only the fresh, structured 502 is retried. A
message-only 502, a 400, a run after provider dispatch, and a retained
native run stay `setup_failed`.
- `packages/plugins/sdk/tests/environment-acquire-error.test.ts`: the
real worker RPC. The test checks the create status (not the 404 cleanup
lookup), the errno, no provider text, the unchanged cleanup envelope,
and that forged envelopes are rejected.

Commands:

```sh
pnpm --filter @paperclipai/plugin-sdk exec vitest run tests/environment-acquire-error.test.ts tests/environment-creation-cleanup.test.ts tests/environment-resume-error.test.ts
cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts src/services/recovery/ src/services/heartbeat/run-completion.test.ts src/services/heartbeat/retries.test.ts src/services/legacy-execution-recovery.test.ts src/services/task-interruption-presentation.test.ts src/services/task-retry.test.ts src/__tests__/environment-runtime.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts
cd packages/plugins/sandbox-providers/daytona && npx vitest run
pnpm --filter @paperclipai/plugin-sdk typecheck && pnpm --filter @paperclipai/server typecheck
```

I did not run a live Daytona 502 on staging. Re-check on staging after
deploy (see Risks).

## Risks

- Sandbox leaks or double creation: each retry is a new run with a new
durable creation record and a new provider name
(`paperclip-create-<operationId>`). The failed attempt stays owned by
its own record, and the existing creation reconciler settles it. A retry
never reuses or abandons it. While a provisional sandbox waits for
reconciliation, up to two more sandboxes can be created for the same
task. This is the same as a manual Retry today.
- Plugins built with an older SDK do not send the envelope. Their
refusals stay `setup_failed`, which is the current behavior.
- An in-process error with `status`/`statusCode` 500–504 or a connection
errno during lease acquisition is now retried twice. The budget is
bounded and shared with the other bootstrap codes.
- An RPC timeout on the host side is not classified by this change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code as an agent with tool use (shell, file edits, test runs).
Extended reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 00:08:44 -06:00
DottaandClaude Opus 5.5 6ca7b151f7 feat: compact an agent's context from the task Properties panel (#19378)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task's Properties panel shows the assignee agent's context window
(paperclipai/paperclip#19273)
> - The board can see that the context is nearly full, but it cannot act
on it
> - A full context makes the agent compact at an unplanned time, often
in the middle of work
> - This pull request adds "Compact now" to the Context hover card
> - The benefit is that the board can compact the agent's context
between runs, at a time it chooses

## Linked Issues or Issue Description

Refs paperclipai/paperclip#19273

**Problem**
The Context row shows how full the agent's context window is. The board
cannot ask the agent to compact it.

**Proposed solution**
A "Compact now" button in the Context hover card. It runs the harness's
own compaction: Codex `thread/compact/start`, and Claude's `/compact`
command through ACP.

**Alternatives considered**
A separate PRP `session.compact` command. Both harnesses run compaction
as a turn, so a compaction turn keeps the existing durable turn binding,
interruption, and recovery. A separate command needs a second turn
lifecycle.

## What Changed

- UI: the Context hover card shows Compact now with three states:
available, Compacting…, and unavailable with a reason (agent mid-run,
harness unsupported, task closed). The Storybook story shows each state.
- Server: `POST /api/issues/:id/context/compact` (board only,
`decideIssueAccess` `issue:mutate`, activity log
`issue.context_compaction_requested`). It queues a short run marked by
`contextSnapshot.contextCompactionRequestId`. `GET
/issues/:id/context-usage` returns the `compaction` state.
- Server: a compaction run does no task work. The native status arbiter
preserves the task status. The lifecycle skips handoff, liveness
continuation, review-path disposition and the issue comment, as it does
for goal-control runs.
- Runner TS: `startTurn({ compact: true })`. The Codex harness session
sends `thread/compact/start`. The runnerd transport maps it to PRP
`turn.start` with `compact: true` (Codex) or `text: "/compact"`
(Claude/ACPX). The native runtime option `sessionCompaction` requires a
resumed session and settles with a synthesized yielded result.
- Runner Rust: `turn.start` accepts `compact: true` for Codex.
`CodexProvider::start_compaction_turn` sends `thread/compact/start` and
binds the turn from the `turn/started` that follows the empty response.
- Fix (from #19273): Codex emitted `context.compacted` on both the start
and the end of one compaction item. It now emits it once, and the count
is distinct by `compactionId`.
- Fix (from #19273): the conversation event view dropped
`context.compacted`, so a finished run never showed the divider. The
view now keeps compactions, plus context readings for runs that
compacted. A finished run's compactions show as thread dividers.
- Fix: the Codex harness stamps before/after sizes on the compaction
event. Codex reports the compacted size before the compaction item ends,
so the old neighbour-based guess showed the wrong size.

## Verification

- `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, `node
scripts/storybook-catalog.mjs`: pass.
- New tests: `server/src/__tests__/context-compaction.test.ts`,
`status-arbiter.test.ts`, `issue-context-usage.test.ts` (duplicate
compaction counts once), `ui/.../context-usage-row.test.tsx`,
`native-session-runtime.test.ts` (context compaction), Codex driver and
runnerd transport tests, Rust
`codex_compaction_turn_binds_the_turn_reported_after_its_empty_response`.
Rust `codex_provider` suite: 115 passed.
- `pnpm test:run` locally: one failure in
`chat-channels.integration.test.ts`. It reads
`.claude/skills/slack/SKILL.md` from the main checkout, which is not
present in this worktree. It is not related to this change.
- Live, local worktree instance, Codex agent on the Paperclip runner:
clicked Compact now. The run succeeded. Context went from 35k to 9.4k
tokens, and later 9.4k to 9.2k. The task stayed In Review. The thread
shows "Context compacted · 9.4k → 9.2k tokens" and the meter dropped.
- Live Claude proof: pending. The local Claude agent needs an Anthropic
API key. claude-agent-acp handles `/compact` as an echo-less turn, and
the runner already maps its "Compacting completed." notice to
`context.compacted`.

## Risks

- A compaction run resumes the agent's session. If the session cannot
resume, the run fails with `native_context_compaction_unavailable` and
the task is unchanged.
- Compact now is not available while the agent runs on the task, because
compaction interrupts the active turn.
- The conversation view now returns context readings for runs that
compacted. Runs that never compacted return the same events as before.
- An old runner binary ignores `compact` and rejects the `turn.start`
because it has no text. It does not start an ordinary turn.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:07:12 -06:00
Nicky LeachandClaude Opus 5.5 e6c74982fe fix(workspaces): report why a workspace provider operation failed (#19362)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A project can store its task workspaces with a workspace storage
provider plugin. Core calls the plugin's `workspaceOperation` method to
prepare, inspect, finalize and remove a workspace.
> - `callWorkspaceProvider` catches every rejection of that call and
throws the same conflict: "Workspace storage operation is incomplete;
retry the same workspace".
> - A provider error, an RPC timeout, a crashed worker and a worker that
never got the call all look the same in the run, the log and the API.
The message tells the user to retry, also when the failure is a fixed
configuration problem that a retry cannot fix.
> - In one diagnosis, three different configuration failures in a row
showed this same message. Finding each cause took trace span counts and
database reads.
> - This pull request lets a provider send a reason from a closed set,
and makes Core tell transport failures apart from provider failures.
> - The benefit is that a failed run says what failed and whether a
retry can help. Provider text and values still never cross into Core.

## Linked Issues or Issue Description

**What happened?**
A workspace provider's `workspaceOperation` failed. The run failed with
"Workspace storage operation is incomplete; retry the same workspace",
`errorCode: setup_failed`, and no reason in the run record or the
`heartbeat execution setup failed` log. Retrying did not help, because
the cause was a provider configuration problem.

**Expected behavior**
Core tells these cases apart: the worker did not receive the call, the
call timed out, the worker stopped during the call, or the provider
reported a failure. For a provider failure, Core shows a bounded reason.
Core tells the user to retry only when a retry can help.

**Steps to reproduce**
1. Install a workspace storage provider plugin whose
`onWorkspaceOperation` throws during `prepare` (for example, because of
missing provider settings).
2. Run an agent task in a project that uses that provider.
3. See the failed run: the error is the generic retry message, and the
run record and logs do not show the cause.

**Paperclip version / commit**
`master` at 5ac05871b5.

**Deployment mode**
Any deployment with a workspace storage provider plugin.

## What Changed

- **SDK reason contract.** New
`packages/plugins/sdk/src/workspace-operation-error.ts`:
`WORKSPACE_OPERATION_FAILURE_REASONS` (11 reasons),
`PluginWorkspaceOperationError`, `workspaceOperationErrorData` and
`readWorkspaceOperationFailureReason`. It uses the same envelope design
as `environment-task-error.ts`. The worker host attaches the envelope
only to `workspaceOperation` errors. It copies only the reason, never
the message.
- **Duck-typed reason.** The worker host also forwards an own
`workspaceFailureReason` property from any error, when the value is in
the closed set. A provider that builds against an older SDK can set the
reason before it can import the class. An inherited or unknown value is
ignored. The value can only select one of Core's fixed messages.
- **Core classification.** New
`server/src/services/workspace-provider-failure.ts`.
`callWorkspaceProvider` maps each rejection to a
`WorkspaceProviderFailureError` (409) with `kind`, `reason` and
`retryable`:
- `worker_unavailable`: no request was sent. The
`PluginWorkerUnavailableError` stays as the cause, so run setup retries
it as `plugin_worker_unavailable` (the existing bounded bootstrap
retry).
- `timeout` and `worker_exited`: the outcome is unknown, so the message
still says to retry the same workspace.
  - `transport`: any other rejection. The message is the old message.
- `provider` with a reason: a fixed message per reason. Only transient
reasons say to retry.
- `provider` without a known reason: "Workspace storage provider
reported a failure; retry the same workspace".
- **Run settlement.** `failRunSetup` records
`resultJson.workspaceProviderFailure` (`kind`, `reason`, `retryable`).
The `heartbeat execution setup failed` log carries the same fields and
the run's `errorCode`. The run keeps `errorCode: setup_failed` (except
for the worker-not-running case above), so the existing setup-failure
retry, recovery and chat paths do not change.
- **Span allowlist.** The host accepts a `plugin.workspace.failure` span
and a `plugin.failure_reason` attribute. The value must be a reason from
the closed set or `unknown`.
- **Docs.** `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` describes the
failure kinds and the reasons. `doc/observability.md` lists the new span
and attribute.

## Verification

- `cd packages/plugins/sdk && npx vitest run`: 21 files, 261 tests pass.
The new `tests/workspace-operation-error.test.ts` sends errors through
the real worker RPC host.
- `cd server && npx vitest run
src/__tests__/workspace-provider-failure.test.ts
src/__tests__/plugin-host-services-span.test.ts
src/services/heartbeat/run-completion.test.ts`: 3 files, 118 tests pass.
These tests cover:
- every transport kind, and that a worker refusal is still found as the
cause;
  - a real worker RPC rejection with a reason, and without a reason;
  - a fixed message and retry decision for every reason;
  - span clamping of `plugin.failure_reason`;
- run setup for a provider reason (no retry, reason in `resultJson` and
in the log) and for a stopped worker (bounded retry).
- `cd server && npx vitest run` for the 4 plugin workspace integration
suites (`plugin-workspaces`, `plugin-fresh-workspaces`,
`project-workspace-provider`, `workspace-delivery-cleanup`): 91 tests
pass. Their lost-reply fixtures still get the old "incomplete" message.
- `npx tsc --noEmit` in `packages/plugins/sdk`: no errors. `npx tsc
--noEmit -p server/tsconfig.json` shows no errors in the changed files.
The remaining errors come from workspace packages that I did not build
locally.
- Not run locally: the full `pnpm -r typecheck`, `pnpm test:run` and
`pnpm build`. CI runs them.

## Risks

- **Behavior change for a stopped worker.** A `workspaceOperation` call
that the worker did not receive now gets `errorCode:
plugin_worker_unavailable` and the bounded bootstrap retry, as other
plugin calls already do. Before, it was a `setup_failed` run with no
retry. No request reached the provider, so the retry is safe.
- **Message changes.** Provider-reported failures and timeouts get new
fixed messages. Code that matches the exact old message only sees it for
`transport` and `remote_incomplete` failures. I did not find a match on
this message outside tests.
- **No new error codes.** Provider failures keep `setup_failed`. Retry,
recovery and chat-retry routing does not change.
- **Duck typing.** Any error with an own `workspaceFailureReason` from
the closed set is forwarded. A provider can only select one of Core's
fixed messages. It cannot send text or values.
- **Observability.** The new span attribute is a closed set and stays
inside the host allowlist. No Telemetry change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, test runs). Extended thinking was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related:
[paperclipai/paperclip#19326](https://github.com/paperclipai/paperclip/pull/19326)
added the same envelope design for Runner revision mismatches. I found
no duplicate PR for this change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:28:49 -08:00
Nicky LeachandClaude Opus 5.5 a6ab6a301c fix(server): retry a run when its remote Runner is out of date (#19326)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can run on a remote Runner that an environment plugin
provides
> - A release can update the app before the remote Runner, so for a
short time the app sees an older Runner
> - In that window the plugin can refuse the task, but Core cannot tell
this refusal from other errors
> - Today the run fails as `setup_failed` with no retry, and the task
goes to board recovery. Before #19246, it was also reported as a project
authentication conflict
> - Agent verification also counts the refusal as a failed harness test,
and it parks the agent after three tries
> - This pull request gives the plugin a stable error code for a Runner
mismatch, and Core waits and retries when it gets that code
> - The benefit is that a release no longer blocks tasks or parks
agents, and a real authentication conflict keeps its own clear notice

## Linked Issues or Issue Description

Refs #19246 (it stopped reporting every precheck error as a project
authentication conflict)
Refs #19249 (the same bounded bootstrap retry for a stopped plugin
worker)
Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope
is unchanged)
Refs #19172 (related open PR: it reports an undescribable plugin Runner
as a retryable check in the same precheck)
Refs #18667 (a real `ai_connection_incompatible` conflict, which now
gets its own notice)

**What happened?**
A run started while its remote Runner was older than the app. The
environment plugin refused the task because the installed Runner did not
match the revision it expects. Core recorded a generic setup failure,
scheduled no retry, and blocked the task. Before #19246, Core reported
this as a project authentication conflict. An agent in `verifying`
counted the same refusal as a failed harness test. After three tries it
stopped verification until a person chose Retry.

**Expected behavior**
Core treats a Runner mismatch as temporary. A run retries without
blocking its task. A `verifying` agent checks again until the Runner
matches. Only a real `ai_connection_incompatible` result blocks the
task, with a notice that names the conflict.

**Steps to reproduce**
1. Use an agent on a plugin environment whose installed remote Runner
revision differs from the revision the plugin expects.
2. Wake the agent on a task, or put a new agent through verification.
3. The run fails with `setup_failed` and the task is blocked.
Verification parks after three tries.

**Paperclip version or commit**
master at 377e98f1a2.

**Deployment mode**
Authenticated deployment with a plugin environment that provides a
remote Runner.

## What Changed

- Plugin SDK: add `PluginEnvironmentTaskError` and
`ENVIRONMENT_TASK_ERROR_CODES` with one code,
`runner_revision_mismatch`. The worker RPC host sends it as structured
`environmentTask` error data with the schema
`paperclip/environment-task-error-code/v1`
(`environmentTaskErrorCodeData`, read with
`readEnvironmentTaskErrorCode`). This uses the same envelope method as
the other environment RPC errors. Core reads only the code, never the
message. The source refusal diagnostic from #19302 keeps its own schema
and API unchanged, and it takes precedence on a source `submit`.
- `executeEnvironmentTask` still replaces each worker error with a
generic error. After the #19302 diagnostic check, it now keeps only that
closed code.
- Run setup: a Runner mismatch before provider dispatch gets the error
code `runner_revision_mismatch`. Setup schedules a bounded retry with
its own reason, `runner_revision_wait`, and budget (10 attempts, 60 s
apart). The reason is in the same resource-wait lane as workspace and AI
connection waits, so a wait does not spend the failure retry budget. The
guards are the same as for `plugin_worker_unavailable`. Recovery and
legacy reconciliation treat the code like the other bootstrap-only
codes. `scheduleBoundedRetryForRun` uses the Runner wait reason and
budget for a mismatch run when a caller gives no reason, and the
recovery sweeper reports exhaustion from that budget
(`failedRunRetryBudgetSpent`). An answered interaction that hits a
mismatch also waits for the Runner, and its successor keeps the resolved
interaction continuation identity.
- Move the managed project auth classification from `heartbeat.ts` to
`classifyManagedProjectAuthFailure` in `run-preparation.ts`. Only a
found `ai_connection_incompatible` conflict becomes a configuration
failure. All other errors keep their own code.
- Recovery notices: `ai_connection_incompatible` now has its own notice
instead of the "secret/env bindings" text. An exhausted Runner wait gets
the notice "Remote Runner is out of date".
- Agent verification: the Runner probe and the managed project auth
precheck report a mismatch as a retryable check.
`assertHarnessTestPassed` maps it to the new lifecycle code
`runner_revision_mismatch`. That code never counts toward the
three-failure park. The agent checks again every 60 s and becomes ready
when the Runner matches.
- Document the error code in `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`.

## Verification

- `cd packages/plugins/sdk && npx vitest run
tests/environment-task-error-code.test.ts
tests/environment-task-error.test.ts tests/worker-rpc-host.test.ts`: the
code crosses the real `environmentTask` RPC. A plain error with the same
message gets no code. Other methods get no envelope. The #19302
diagnostic tests still pass.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts`: a fresh or wrapped
mismatch gets `runner_revision_mismatch` and the runner retry budget,
with no configuration failure. A started, untrusted (message only) or
native run keeps `setup_failed` with no retry. A real
`ai_connection_incompatible` stays `configuration_incomplete` with no
retry. Other errors, such as an unreachable execution host, are not
reported as an authentication conflict.
- `npx vitest run src/services/execution-recovery-attempt.test.ts`:
Runner waits do not change `failureRetries`, and earlier failures do not
shorten the wait.
- `npx vitest run src/services/heartbeat/retries.test.ts
src/modules/run-dispatch` (embedded Postgres): a recovery caller without
options gets the Runner wait budget and its exhaustion. An answered
interaction waits for the Runner, keeps its continuation identity and
does not spend failure retries.
- `npx vitest run src/__tests__/environment-task-runtime.test.ts`:
sanitization keeps only the code. No worker text or cause survives.
- `npx vitest run
src/modules/agent-lifecycle/__tests__/verification-retry.test.ts`
(embedded Postgres): a `verifying` agent keeps checking after five
mismatches. It never gets `retryRequired`. It becomes ready when the
next check passes.
- `npx vitest run
src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts
src/__tests__/agent-harness-verification-remote.test.ts
src/services/recovery/stranded-notice.test.ts
src/services/recovery/service.pause-durability.test.ts
src/services/legacy-execution-recovery.test.ts`.
- `npx tsc --noEmit` passes for `packages/plugins/sdk` and `server`.
- One test in `src/__tests__/heartbeat-process-recovery.test.ts`
("redacts opaque environment-bound credentials from Sentry diagnostics")
fails on my machine. It fails the same way on the base commit without
this change.
- I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm
build`. CI covers them.

## Risks

- Low to medium. The change adds one new setup failure code and one
lifecycle code. Other failures keep their current behavior.
- A Runner mismatch must come from a plugin that throws
`PluginEnvironmentTaskError`. Until an environment plugin does that,
behavior is the same as today.
- The run retry budget is 10 minutes. If a Runner update takes longer,
the task goes to board recovery with the Runner notice.
- A remote Runner run without a managed AI connection reaches the Runner
check only after native dispatch. That configuration is not supported,
so this change does not handle it. Such a run keeps the native session
retry (3 attempts, 30 s apart).
- #19172 edits the same precheck call in `agent-environment-test.ts`.
Whichever PR merges second needs a small rebase. Keep the mismatch check
before its generic retryable check.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (code search, shell and test runs). Reasoning
mode: standard agentic coding.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 14:34:23 -08:00
DottaandClaude Opus 5.5 cb5d7e440c fix(server): retry setup when the plugin worker was not running (#19249)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote agent runs call a sandbox provider plugin, such as Daytona,
through a plugin worker process.
> - Sometimes that worker is not running. It can be stopped, restarting
after a crash, or retired by the host.
> - Then the host refuses the call before it sends anything to the
worker.
> - Setup recorded this refusal as a generic `setup_failed`. It
scheduled no retry, so the task became stranded and needed a person.
> - This pull request types the refusal and gives it its own transient
setup code. The bounded bootstrap retry then runs.
> - The benefit is that the task continues on its own when the worker is
available again. Users do not see a blocked task.

## Linked Issues or Issue Description

Refs #19150 (removed one trigger for the worker retirement)
Refs #19245 (restarts a worker after a fail-closed retirement)

**What happened?**
A plugin worker stopped. Each run that then called `environmentExecute`
during setup failed with `Cannot call "environmentExecute" — worker for
"<plugin>" is stopped`. The run got `setup_failed`, no retry was
scheduled, and the task moved to board recovery. On one staging
instance, 15 runs failed this way in 12 minutes.

**Expected behavior**
No provider work started, so the failure is safe to retry. The run must
retry within the bootstrap budget, as it does for a reusable sandbox
resume failure or a preparation timeout.

**Steps to reproduce**
1. Stop the sandbox provider plugin worker, or let the host retire it.
2. Wake an agent whose run needs that provider during setup.
3. The run fails with `setup_failed`. No retry is scheduled.

**Paperclip version or commit**
master at 76e43852.

**Deployment mode**
Authenticated cloud deployment (staging).

## What Changed

- Add `PluginWorkerUnavailableError` with the code
`plugin_worker_unavailable`
(`server/src/services/plugin-worker-unavailable-error.ts`).
- The plugin worker manager now rejects with this error when the worker
is stopped, crashed, restarting, not running, or not registered. The
message text does not change.
- Setup failure handling maps this error to `plugin_worker_unavailable`,
but only before provider dispatch and only for a legacy run with no
native session or runner instance. These are the same guards as for the
sibling transient codes.
- `isTransientWorkspacePreparationCode` includes the new code, so setup
schedules the bounded bootstrap retry.
- Legacy execution recovery and the recovery service treat the code like
the other bootstrap-only codes. The stranded-task notice has a title and
a next action for it.

## Verification

- Red: the new `run-completion` test failed on the unchanged code. A
fresh refusal got `setup_failed` and no retry.
- Green: the same test passes. A fresh refusal gets
`plugin_worker_unavailable`, `providerWorkStarted: false` and a bounded
retry. A started, untrusted (message-only) or native run keeps
`setup_failed` and gets no retry.
- New worker manager test: a stopped handle and a missing worker both
reject with the typed error. The call is not sent, and `onWorkerSettled`
runs once.
- `pnpm --filter @paperclipai/server exec tsc --noEmit -p .` passes.
- Related suites pass: run-completion, plugin-worker-manager,
legacy-execution-recovery, heartbeat-retry-scheduling and recovery (17
files, 439 tests).

## Risks

- Low risk. The retry uses the existing bounded bootstrap budget. When
the budget is exhausted, the existing board recovery action takes over.
- The retry applies only before provider dispatch. A worker that stopped
after work started keeps the old behavior.
- A wrapper that drops the error `cause` keeps the old `setup_failed`
behavior. It does not cause a wrong retry.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 09:05:14 -06:00
DottaandClaude Opus 5.5 2ce5f03cd8 fix(workspaces): name remote restore failures and leave a board recovery action (#19207)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A remote task restores its saved workspace into a sandbox before the
agent starts.
> - When that restore fails, the run records only
`remote_workspace_operation_failed: operation=restore exit=1`.
> - The run then fails as `setup_failed`. No retry starts, and a
`blocked` task gets no recovery action.
> - An operator cannot tell which step failed, and nobody owns the next
step.
> - This pull request makes the sandbox scripts name their failed step
in a closed grammar, and it gives a restore failure a bounded retry or a
named board recovery action.
> - The benefit is a visible cause and a clear owner. Saved work stays
protected.

## Linked Issues or Issue Description

Refs #19152 and #19187. Those changes name copy-back (save) failures.
Refs #18852 and #19195. Those changes added the bounded bootstrap retry
that this change reuses.

**What happened?**

A board reply woke an agent on a `blocked` parent task. The run failed
during setup 13 seconds later. The only run event was
`remote_workspace_operation_failed: operation=restore exit=1`. The run
had no workspace operation row. The task stayed `blocked`, and its
recovery actions showed `active: null`. The same opaque family also
appeared for workspace preparation (`exit=1` only).

**Expected behavior**

The run names which restore step failed, without command output or
paths. A failure before provider work uses a bounded retry when a retry
can help. Otherwise, and when the retry budget is spent, the board gets
a named recovery action. The saved version is never changed or
overwritten.

**Steps to reproduce**

1. Save a remote workspace, then start a new run on a different
allocation.
2. Put a folder at the published workspace path in the sandbox before
restore.
3. Observe that the base code records only `operation=restore exit=1`,
then `setup_failed`, with no recovery action for a `blocked` task.

**Paperclip version or commit**

Base: `87dfba6818`.

**Deployment mode**

Legacy runtime with sandbox (Daytona) remote workspace storage.

## What Changed

- The prepare, repository, and restore sandbox scripts track their
current stage. On failure they print `remote_workspace_<op>_failed:
stage=<stage>` plus closed fields: `reason=` (the script's own
snake_case refusal), `archive=` (the archive helper's fixed reason),
`code=` (errno), `disk=full`, `git=` (a fixed label chosen from Git
output), and the existing `exit`/`signal`/`program` fields.
- The server accepts that text only when it matches a strict grammar for
the same operation. Other output is dropped, as before.
- Provider refusals during the restore upload and restore command are
recorded by closed labels, like copy-back.
- A failed saved-workspace restore raises `RemoteWorkspaceRestoreError`
with the server step (`snapshot_lookup`, `snapshot_download`,
`sandbox_restore`). It is retryable only when the owner re-reads the
same writer, epoch, and saved version with no unsaved work, and the
sandbox did not name a refusal reason.
- Setup failure handling classifies a pre-dispatch legacy restore
failure as `remote_workspace_restore_failed`. A retryable failure uses
the existing bounded bootstrap retry. A non-retryable or exhausted
failure records a board-owned recovery action with the redacted cause,
when stranded-work recovery does not scan the task status (for example
`blocked`).
- Stranded-work recovery gets a named notice for this code. Generic
continuation recovery does not grant it a second budget, and legacy
reconciliation does not invent provider actions for it.
- `doc/execution-semantics.md` documents the contract.

## Verification

- Red: the new transport tests failed on the base code with
`remote_workspace_operation_failed: operation=restore exit=1` (and the
same for prepare and repository). The new completion tests failed
because the run became `setup_failed` with no retry and no recovery
action.
- Green: `npx vitest run src/services/remote-workspace-transport.test.ts
src/services/heartbeat/run-completion.test.ts
src/services/remote-workspaces.integration.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts
src/services/legacy-execution-recovery.test.ts src/services/recovery
src/services/remote-workspace-save-concurrency.test.ts
src/services/native-runtime/native-workspace-finalizer-failure.test.ts
src/__tests__/native-workspace-finalization-work.test.ts` passes (563
tests).
- The transport tests use real sandbox commands. They prove an occupied
folder reports `stage=inspect_root reason=occupied_root` and keeps its
files. They prove a wrong manifest reports `stage=extract
archive=manifest_mismatch`. They prove a missing branch reports
`stage=checkout_branch`, and a changed origin reports
`stage=verify_origin`. Forged output with a path is dropped.
- The real-PostgreSQL cold-restore test proves an occupied root gives a
non-retryable `sandbox_restore` failure and keeps the unknown files.
- The completion tests cover retried, exhausted, refused, post-dispatch,
and `in_progress` cases.
- `tsc --noEmit` for the server reports no errors in the changed files.

## Risks

- A retry keeps the same restore path. If the cause is deterministic but
unnamed, the run uses its two retries before the board action. Named
refusals skip the retries.
- The board action is recorded only for statuses that stranded-work
recovery does not scan, and only when no other action is active. This
prevents a second recovery identity for one failure.
- Script stderr grammar is new. Old sandboxes run the script that the
server sends, so no runner update is necessary.
- No migration, deployment, or credential change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with
tool use, code execution, and browser tools for read-only staging
evidence. Context window and reasoning mode are host-managed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 02:40:01 -06:00
DottaandClaude Opus 5.5 87dfba6818 fix(native): retry an unpublished accepted task reply durably instead of dropping it (#19200)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run commits its status decision and its final reply comment
in separate steps
> - Background presentation recovery publishes an accepted reply when
the live path does not record one
> - That worker dropped the run after one refused attempt and only threw
on an error, with no record on the run
> - A task could then show done while the user never saw the answer,
until an unrelated wake reloaded the queue
> - This pull request makes the publication of an accepted ordinary task
reply durable, bounded, and recorded
> - The benefit is that a task cannot close silently without its answer

## Linked Issues or Issue Description

Refs #19130 (verify requested publication at finalization, for
documents), #19183 (deliver messages saved while a native run
finalizes), #19186 (stale owner done). This change does not duplicate
them. It covers the final reply comment.

**What happened?**
A native run accepted a result with disposition `done`. The status
decision set the task to done. The accepted final reply was not
published as a task comment. The run had no publication record (no
`run.presentation.resolved` event and no `presentationDecision`). The
reply was published by background recovery 58 minutes later, after an
unrelated wake reloaded the presentation queue.

**Expected behavior**
The accepted reply is published promptly. A refused or failed attempt is
retried with backoff and never posts twice. If publication fails
permanently, the run and the task show a named failure with an owner.

**Steps to reproduce**
1. Commit an accepted native task result with disposition `done`, with
the final reply not yet visible to presentation recovery.
2. Run the native response presentation worker check once. The attempt
is refused.
3. On `master`, the worker returns `null` and drops the run. Nothing
records the attempt. Nothing retries it until another notification for
that scope.

**Paperclip version or commit**
`master` at 3b17b5ce15.

**Deployment mode**
Cloud staging, native `paperclip_runner` agent.

Root cause notes: the staging evidence shows that the live completion
path did not reach its presentation step for the affected run (no
`lifecycle run succeeded` event). Staging server logs for that window
are not available, so the exact first refusal is not proven. The worker
defect is proven: one refusal makes `check()` return `null`, which
deletes the scheduled entry. One thrown error escapes with only
in-memory backoff and no record. The fix closes that gap for every
refusal or error cause.

## What Changed

- `native-response-presentation-work.ts`: when an accepted ordinary task
reply (`done` or `needs_review`, not an Agent Chat conversation) is
still unpublished after a due attempt, the worker records the attempt
and keeps the run scheduled. A thrown error for this class is recorded
instead of escaping. When the reply is published, an open retry receipt
is set to `resolved`.
- New `native-response-publication-receipt.ts`: stores the receipt in
`heartbeat_runs.result_json.responsePublication` (attempts, last reason,
last error, next attempt). Backoff is 30 s doubling to at most 5 min,
for 6 attempts. Each retry appends a `run.response_publication.retrying`
run event. After the last attempt it records
`native_response_publication_failed` with the run's agent as owner, a
`run.response_publication.failed` run event, and an
`issue.response_publication_failed` task activity.
- `native-response-presentation-query.ts`: the candidate query exposes
the saved retry deadline, so a restarted worker reconstructs the
schedule. It excludes a run whose publication is recorded as failed, so
retries stay bounded.
- An open retry receipt is settled to `resolved` in the same write that
records the reply's `commentId` (background recovery and the live
`recordRunPresentationDecision`). A candidate that disappears for
another reason leaves the receipt unchanged, so the bound cannot reset.
- When the reply comment already exists but its decision was not
recorded, recovery now records it as `reuse` with that `commentId`. It
does not post again and does not count a refusal.
- Under the run lock, an attempt is not counted again when another
worker already saved a future deadline. The worker also honors that
saved deadline after recovery.
- Idempotency is unchanged: the comment is written only while the
presentation decision has no `commentId`, under the existing coordinator
lock. The receipt write never overwrites a run that already has a
`commentId`.
- `doc/execution-semantics.md`: documents the contract.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-response-publication-retry.test.ts`
- Red on `master` (3 failed): after one refusal `check()` returned
`null` (dropped); one injected transaction error escaped with no record;
the refusal loop never stopped (more than 10 attempts, no failure
recorded).
- Green with this change (5 passed, including an already posted reply
and two concurrent workers): one refusal keeps the run scheduled and
recorded, a reloaded worker keeps the same deadline, the reply posts
exactly once after it becomes visible, and repeated checks do not
double-post; one injected error is recorded and retried; permanent
refusal stops after bounded attempts with the named, owned failure on
the run event and task activity.
- Related suites pass: `agent-chat-response-recovery.test.ts`,
`presentation-ownership-read.test.ts`,
`idle-native-finalization.test.ts`, `chat-monitor-continuation.test.ts`,
`native-preserved-monitor-wait.test.ts`,
`paperclip-control-plane-port.test.ts`,
`chat-channels.integration.test.ts`,
`photon/channel.integration.test.ts` (2047 of 2049 passed in one
combined run). One local failure needs a bundled Slack skill file that
is missing in this worktree. The other ("materializes direct
external-chat finals once") failed once under load and then passed three
times in isolation and in the combined set with this change.
- Not run: a live staging run. The change has not been deployed.

## Risks

- Behavior change: an ordinary task reply that recovery keeps refusing
now gets a recorded failure and a task activity after about 12 minutes.
Before, it was dropped silently. A persistent integrity decline (for
example a digest mismatch) now becomes visible this way. This is
intended.
- The status decision still commits before the reply. This PR does not
hold `done` on the comment. It makes a missing reply visible with an
owner.
- Agent Chat conversations and yielded chat responses keep their
existing behavior and their completion outbox owner.
- No schema migration. The receipt lives in the existing `result_json`
column.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 01:19:04 -06:00
DottaandClaude Opus 5.5 c656b95b6b fix(environments): retry a refused sandbox resume with the bounded bootstrap budget (#19195)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote agent keeps its sandbox after a turn, so a follow-up
turn resumes the same sandbox and its files
> - The sandbox provider auto-stops an idle sandbox (Daytona: 15 minutes
by default). The warm native idle window is also 15 minutes
> - A follow-up that reaches the sandbox during that stop gets a
provider refusal. The host keeps the sandbox ("the lease was preserved
and no replacement was created"), which is correct
> - But setup then records a generic `setup_failed`. No retry is
scheduled, no cause is recorded, and the reply that woke the agent is
effectively dropped
> - This pull request gives that refusal its own failure code and routes
it through the existing bounded bootstrap retry. The retry resumes the
same recorded sandbox
> - The benefit is that a follow-up continues a few seconds later on the
same saved sandbox, instead of stopping the task with no next action

## Linked Issues or Issue Description

Refs #18852 (the bounded bootstrap retry that this change reuses).
Related: #19152 and #19187 (copy-back causes), #19178 (Daytona
errored-sandbox recovery before stop-only cleanup), #19182 and #19180
(recovery holds). This change does not overlap with them: they cover
cleanup, copy-back, and recovery after a run. This change covers a
resume refusal before provider work starts.

**What happened?**

On a staging soak, two follow-up turns failed at setup after a
successful first turn:

- A board reply woke the agent. The run failed after 4.9 seconds with
`setup_failed`: `Reusable sandbox lease "…" could not be resumed; the
lease was preserved and no replacement was created.` The issue stayed in
review with no recovery action. The reply was not delivered.
- An earlier run on another task failed with the same message.

Both resume attempts started 15 minutes 8–25 seconds after the previous
turn released the sandbox. The first turns used `keep_running` with a
warm idle timeout of 900,000 ms. The Daytona default `autoStopInterval`
is also 15 minutes. So the resume reached the sandbox while the provider
was stopping it. The provider refused within seconds; this was not a
long resume timeout. The provider cause was not recorded, so this timing
is the strongest evidence. The exact provider state remains an
inference.

**Expected behavior**

A refused resume before provider dispatch is a temporary infrastructure
condition. The host retries it with a small, bounded budget. Each retry
resumes the same recorded sandbox. A refusal does not prove that the
sandbox is lost, so the host never destroys it or creates a replacement.
Only an explicit provider not-found or a workspace identity mismatch
with a verified backup permits replacement (unchanged). The run records
why the provider refused, without provider text.

**Steps to reproduce**

1. Run a warm remote turn that keeps its sandbox.
2. Make the provider's `environmentResumeLease` throw a non-transient
error (for example, a 400/409 while the sandbox stops).
3. Wake the agent with a reply.
4. Before this change: the run fails with `setup_failed`, no retry is
scheduled, and the error has no cause.

**Paperclip version or commit**

Seen on staging at `5b99bdae`. The touched code is unchanged on current
`master` (`f7a4c5678f`).

## What Changed

- Move `ReusableSandboxResumeError` to
`server/src/services/reusable-sandbox-resume-error.ts` (re-exported from
`environment-runtime.ts`). Give it the code
`reusable_sandbox_resume_failed`.
- Add the provider refusal to the error message by closed labels only:
`(cause=<label> status=<http> code=<errno>)`. The labels come from the
existing transport classifier, now exported as
`sandboxProviderFailureLabel`. Provider text, paths, and IDs are never
copied.
- Carry the provider status across the plugin worker RPC. A JSON-RPC
error has only a message and a numeric RPC code. The worker host now
attaches a validated `paperclip/environment-resume-error/v1` envelope
(HTTP status and an allowlisted errno only) to `environmentResumeLease`
failures. The host re-validates it and adds it to the recorded cause.
- In setup-failure settlement, find this error through the orchestrator
wrapper. Record `reusable_sandbox_resume_failed` only when provider
dispatch did not start and the run has no native session or runner
identity (the same guard as `remote_workspace_prepare_timeout`).
- Add the code to `isTransientWorkspacePreparationCode`. The existing
scheduler then gives two retries (30 s, 60 s) with its issue gate, Stop,
pause, and dependency checks.
- Exclude the code from generic continuation recovery and from legacy
reconciliation (no unknown provider actions; bootstrap evidence proves
none started), like the template.
- Add a specific stranded-recovery notice: "Saved sandbox could not be
resumed".
- Document the contract in `doc/execution-semantics.md`.

## Verification

- Red (before the fix, same tests): 6 failures. The `fresh` case
recorded `setup_failed` and did not schedule a retry. The error had no
cause. Legacy reconciliation held the run. Generic recovery granted
another budget.
- Green: `npx vitest run src/services/heartbeat/run-completion.test.ts
src/services/legacy-execution-recovery.test.ts src/services/recovery/
src/__tests__/heartbeat-retry-scheduling.test.ts
src/__tests__/environment-runtime.test.ts
src/services/remote-workspace-transport.test.ts
src/services/heartbeat/retries.test.ts` (in `server/`): 619 tests pass.
Also `src/services/recovery/stranded-notice` and
`src/services/remote-workspace*`: 411 pass.
- Safety regressions:
- `run-completion.test.ts`: only a pre-dispatch legacy failure with no
native identity gets the code and a retry. `started`, `retained`
(native), and an untyped copy of the message stay `setup_failed` with no
retry. The recorded error contains the closed cause and not the provider
path.
- `heartbeat-retry-scheduling.test.ts` (real PostgreSQL): two concurrent
schedulers across a restart create exactly one successor per attempt.
The budget ends after two attempts. Stop, pause, and a new dependency
cancel the scheduled retry. Generic stranded recovery creates no extra
run after exhaustion.
- `packages/plugins/sdk/tests/environment-resume-error.test.ts`: through
the real worker RPC, the `JsonRpcCallError` has no status, and the
envelope gives 409 and the errno. Provider text does not cross. Other
methods get no envelope. Malformed envelopes are rejected.
- `environment-runtime.test.ts`: a throwing resume still preserves the
exact lease. The runtime does not call destroy or acquire. The resume
throws a `JsonRpcCallError` whose message has no status. The thrown
message names `cause=provider_error status=409` from the envelope, and
not the provider text.
- `node scripts/typecheck-low-memory.mjs` passes. `git diff --check`
passes.
- Not verified: a live staging run at this revision. The original tasks
were not retried. This PR is not deployed.

## Risks

- Low. The change affects only runs that fail with this typed error
before provider dispatch. Each retry resumes the same lease and inherits
the source wake context, so a reply is delivered once and no second
executor starts.
- If the provider cannot start the sandbox for longer than about 90
seconds, the two retries end and the task gets the specific recovery
notice. The sandbox and its saved files are kept.
- The error message now has a parenthetical cause before the semicolon.
No code parses this message.
- An issue in review does not get the immediate stranded notice
(unchanged existing behavior). The retry, not the notice, delivers the
reply in that case.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, browser reads of staging diagnostics).
Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 00:20:21 -06:00
Devin FoleyandClaude Opus 5.5 7edbc1fc07 fix(server): keep a successor's issue lock when a finalizing native run releases its checkout (#19156)
`release_checkout` now clears `checkoutRunId` and the execution lock only when they belong to the committing run, so a successor that claimed the issue while its predecessor was finalizing keeps its claim and passes reviewed-chat attestation.

`attestReviewedExternalChatRun` reports the first failed check as a closed label. Setup failures store it in `resultJson.reviewedChatBinding.denialReason`, and Sentry run-failure diagnostics export it as `execution.reviewedChatBindingDenialReason` (no IDs or text). The error message is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 17:49:03 -08:00
DottaandPaperclip 0a1e4c3d0f fix(workspaces): retry pristine remote setup timeouts (#18852)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A remote task needs a workspace before its model can start.
> - Fresh workspace preparation can reach its transport deadline.
> - Failure handling records that timeout as a generic setup failure and
stops the task.
> - An unpublished workspace has no saved task work to repeat.
> - This change routes that narrow failure through the existing bounded
retry scheduler.
> - Saved work, retained executions, and normal admission gates keep
their existing protection.

## Linked Issues or Issue Description

Refs #18827, which preserves remote timeout diagnostics. This change
builds on that merged repair. Related #16727 concerns controller-managed
clones and optional partial clones; this change keeps remote clone
contents and deadlines unchanged.

**What happened?**

A fresh remote workspace reached its 300-second command limit before
model execution. The task then stopped with `setup_failed` and no
scheduled continuation. A separate full clone completed in 137 seconds
on a disposable Daytona sandbox. The original delay remains unexplained.
This change repairs the missing recovery path; it does not claim to
remove the underlying delay.

**Expected behavior**

A fresh, unpublished workspace preparation timeout can use the existing
two automatic retries. Retry scheduling remains durable and company
scoped. Stop, pause, dependencies, budgets, task ownership, saved work,
and retained native execution must still prevent unsafe work.

**Steps to reproduce**

1. Start preparation for a fresh remote workspace with no saved version
or published allocation.
2. Make its initial directory/clone command return a transport timeout.
3. Observe that the base code records a generic setup failure without
bounded recovery.
4. With this change, the owner validates current workspace authority and
preserves a specific bootstrap failure for the normal retry scheduler.

**Paperclip version or commit**

Base: `e0276e0166`, including #18827.

**Deployment mode**

Native Paperclip Runner with sandbox workspace storage. Verification
uses real PostgreSQL, filesystem commands, and a separate live Daytona
transport probe.

## What Changed

- Recognize only the initial preparation timeout while the same company,
writer, and epoch still own a pristine workspace.
- Keep restore, additional repository, provisioning, save, changed
ownership, dirty workspace, and retained native execution failures
outside this recovery policy.
- Persist the typed failure before using the existing retry scheduler,
admission gates, and attempt budget.
- After successful terminal lease cleanup, reclaim an unpublished
bootstrap-only reservation even when a provider returns no stop receipt.
Preserve receipt requirements for saved/published work, unknown
execution, and native identities; do not synthesize termination
evidence.
- Prevent generic recovery from granting another budget after
exhaustion. Keep a specific recovery notice.
- Add regressions for persistence, concurrent scheduling, restart,
exhaustion, Stop, pause, dependencies, fresh allocation, lost replies,
and saved output after cold restoration.
- Document the recovery contract.

## Verification

- Red: the real-PostgreSQL workspace regression first failed because the
timeout had no recovery classification. The completion regression then
failed because it became `setup_failed`. A retained-native negative
failed before the admission guard. The review follow-up reproduced a
failed successor reservation after production lease release returned
void.
- Green: all 127 transport/archive/real-PostgreSQL workspace cases pass
after the review repair. All 12 receipt/void handoff cases pass against
the final query. The 141 complete completion/retry/recovery cases and
the fresh-allocation selection case pass. Safety cases cover concurrent
scheduling across restart, the two-attempt cap, Stop/pause/dependency
gates, native identities, saved/published files, active leases, failed
cleanup, and cold-restored output.
- `node scripts/typecheck-low-memory.mjs` and `pnpm build` pass at
`ba96cff771707415e4a5e893c8b3a97499ead25b`. `git diff --check` and the
module-boundary check pass. The full local test wrapper completed all
1,835 chat cases across six partitions, then was interrupted during its
serial general-server phase after the complete current-head CI suite
passed. Its interrupted run is not a full local-suite pass; its log is
retained. The owned process tree and database children exited.
- The [complete GitHub Actions
run](https://github.com/paperclipai/paperclip/actions/runs/41718108375)
is successful at this exact head, including all server/workspace/chat
partitions, all nine serialized groups, all eight browser shards, build,
typecheck, native Runner, package, Rust, and release-policy checks. The
first Rust worker shut down during cache restore before tests; one retry
passed without source changes. Expected Storybook skips remain skips.
- Earlier failures remain accounted for: the first typecheck found two
incomplete process-result fixtures, now corrected. An intermediate
completion run exposed a test double that returned no promise and a
teardown timeout; the complete 141-case suites subsequently passed. The
first-head CI had four worker shutdowns and one worker communication
loss. A three-second monitor settlement wait failed in CI and locally,
then all 21 cases passed unchanged at base and head. An earlier CI retry
found a real-process history adoption assertion failure; that exact case
passed locally at base and head. Both server groups pass in final-head
CI. These timing causes remain unknown. [Earlier CI
evidence](https://github.com/paperclipai/paperclip/actions/runs/41713712522).
- Live supporting evidence: unchanged production preparation code
completed a full repository clone in 137 seconds using Daytona SDK
0.203.0, Node 24.22.2, Git 2.39.5, and Linux x86_64, with 1 CPU, 1 GiB
memory, and 1 GiB disk. The approved diagnostic account could not access
the original fleet snapshot, so the probe used public
`node:24-bookworm`. Both probe allocations were deleted and absence
verified. Full task acceptance at the fixed revision remains unverified:
the original fleet snapshot and an explicitly configured disposable
Codex subscription fixture were unavailable. No CLI login or hosted
subscription was copied.
- Greptile is 5/5 at the final head with no actionable findings. The
Paperclip review finding is repaired, replied to, and resolved. The
separate **Paperclip Review** check remains queued. Its task failed
workspace export after publishing the first assessment; the staging
service was then observed showing a workspace recovery page after an
update failure. Subsequent browser rechecks timed out, including a fresh
tab, so the normal recovery control could not be used. Reconcile that
existing export after service access returns, then let the saved
current-head review request run. That external review is the remaining
check; this PR is not yet fully green.
- The original hosted task was last verified blocked. This PR has not
been deployed and the task has not been retried by this work.

Reviewers can run `pnpm test:run`, `node
scripts/typecheck-low-memory.mjs`, and `pnpm build`. The
remote-workspace integration suite additionally needs an owned
disposable PostgreSQL database named `remote_workspace_*test` through
`PAPERCLIP_REMOTE_WORKSPACE_TEST_DATABASE_URL`; otherwise that suite is
skipped.

## Risks

Each retry keeps the existing preparation deadline, so persistent slow
clones can consume the bounded budget before intervention. A retry
receives fresh allocation admission; it does not replay the command
against the old allocation. The workspace owner checks persisted
authority before classifying the failure, and normal admission still
rechecks that authority. Historical generic setup failures do not gain
automatic replay. No database migration, provider-pack update, model
fallback, credential change, or deployment is included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and browser tools. The host does not expose a more specific
serving model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 15:36:07 -06:00
DottaandPaperclip e16de641ba fix(heartbeat): retain Stop and receipt debt after setup failure (#18776)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server owns execution, Stop delivery, and usage settlement.
> - A restored execution can fail during preparation.
> - A preparation failure does not prove that earlier provider work
never started.
> - Replacing its result metadata can remove an existing Stop and
falsely release its accounting hold.
> - This pull request preserves those obligations at the terminal write.
> - The benefit is honest cleanup and receipt debt without new execution
authority.

## Linked Issues or Issue Description

**What happened?**

A recovered native run can fail a configuration check before dispatch in
the new controller. The setup failure writes bootstrap metadata with
`providerWorkStarted: false`. It also replaces the previous result
metadata. This can remove a pending native Stop. Accounting then accepts
the new bootstrap flag and releases the original hold despite an
incomplete receipt.

**Expected behavior**

Keep the current run-Stop and receipt debt when setup fails after
earlier native execution. Preserve an acknowledgement that commits
during the failure write. Fresh setup that never dispatched must remain
eligible for pre-provider settlement. Keep account-identity checks
effective.

**Steps to reproduce**

1. Save admitted native execution, an active provider checkpoint, an
original held reservation, and an audited pending run-only Stop.
2. Recover the run and fail preparation with an account configuration
error before the new controller dispatches.
3. Run ordinary cost reconciliation with an incomplete receipt.
4. Observe that the base removes the Stop and releases the hold as if
the run never started.

Related: #18682 repairs exact-session Stop cleanup after restart. This
change protects setup failure before that path. #18702 handles AI
connection-request waits. #18035 proposes broader accounting policy.
#18778 separately handles retained cleanup across account changes.
Account, budget, and receipt authority are unchanged here.

## What Changed

- Mark setup-failure status writes through an internal lifecycle option.
At the atomic write, a prior coordinator attempt bound to the same run,
company and task prevents false pre-provider evidence, even with a stale
loaded profile. Lock the coordinator before the run so an in-flight
claim cannot be missed. Selected execution input alone does not imply
provider work.
- Keep genuine pre-dispatch setup eligible for settlement after fresh
native input is persisted, whether the loaded profile is current or
stale.
- Preserve only exactly bound native run-Stop fields from the current
database row during failure writes, including the intentional handoff
marker. Keep the latest acknowledgement when the update waits behind
another writer. Normal Stop delivery still advances its own metadata.
Replace unrelated stale result fields.
- Add real PostgreSQL regressions for receipt debt, original holds,
stale snapshots, late acknowledgement, fresh setup, dispatched work, and
foreign or task-scoped metadata.
- Update the execution contract.

## Verification

- On untouched master, the recovered-setup regression fails six
assertions: Stop removal, false bootstrap evidence, false accounting
settlement, lost debt, and released hold. The fresh-setup control
passes.
- A second regression fails four assertions with the first repair
present when the loaded profile lacks the current saved execution. The
atomic current-row check fixes that case.
- The focused safety selection passes. It includes real row-lock
contention. A controlled concurrent writer seeds audited acknowledgement
state while the terminal update waits. The product code retains that
state and creates no acknowledgement or provider turn. This race test
does not prove physical provider termination.
- Fresh native and legacy setup can settle without a receipt. Dispatched
native and legacy work keeps its debt. Foreign run, company, issue and
task-scope metadata stays excluded. Unrelated old result fields are
replaced.
- After the final review fixes, all three completion, Stop and
budget-cancellation suites pass: 121 tests. The exact `pnpm run
typecheck:build-gaps` entry point passes after the final source and
service-test edits. Module-boundary checks pass. An ordinary adapter
pre-provider receipt control also passes.
- Native review also found that fresh selected input could be mistaken
for prior execution. Real-admission loaded/stale cases fail against the
earlier head; a prior-attempt-only control passes. The correction uses
the bound coordinator attempt and includes a foreign-task negative.
Reservation `providerStartedAt` is stamped at reservation creation and
is not used as provider-work evidence.
- A row-lock regression also exposes a first execution claim committing
during the terminal write: the prior statement snapshot misses it and
falsely releases the hold. Locking the exact coordinator before the run
fixes that race and preserves the existing claim lock order. Concurrent
Stop acknowledgement tests remain green.
- Review found two preservation gaps. New real-database reds show
handoff classification loss and overwritten budget Stop delivery
progress. Failure-only preservation and the bounded handoff marker
address them.
- Initial helper-import and reservation-order fixture errors are
retained separately from causal failures. No assertion or deadline was
relaxed to pass the regression.
- Current-head Greptile is 5/5. The optional native review stopped
before completing and returned NEUTRAL; this is not substantive
approval. Its prior finding was reproduced and fixed. Coordinator and
independent source/evidence reviews found no additional issue; neither
is a hosted recovery test.
- These are local database tests with saved provider state. No hosted
recovery, physical provider termination, historical-state repair, or
deployment is claimed. Full repository tests and build passed in
current-head CI: 55 successful checks, three expected skips, and the
explicitly incomplete optional native review returned NEUTRAL. Required
verify and e2e aggregates passed.

## Risks

- The current database row must retain canonical Stop state while
replacing other result metadata. The preservation is limited to the same
native run, company, issue and run-only Stop scope.
- Setup-failure history checks keep uncertain work pending. They do not
create a receipt, acknowledgement, reservation or charge.
- The setup-only write option leaves ordinary adapter pre-provider
receipts unchanged. No account-identity, budget, dispatch, or provider
authorization gate is bypassed.
- Already overwritten historical state needs separate evidence and
repair. This change does not reconstruct it.
- No schema, public API, provider profile, or deployment setting changes
are required.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source
inspection, code execution, and regression testing. Context-window
capacity is not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 08:49:28 -06:00
DottaandPaperclip b22ef22d56 fix(runner): show provider failure details on tasks and runs (#18622)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner starts providers and records their run outcomes.
> - Provider failures can contain useful details below a generic
protocol error.
> - Those details were lost at the sidecar, command receipt, and run
projection boundaries.
> - A blocked task therefore showed a generic failure even when the
provider gave a clear reason.
> - This pull request carries a bounded, sanitized provider failure
through those boundaries.
> - Operators can read the actual cause on the task and run pages.
Existing recovery policy still controls retries.

## Linked Issues or Issue Description

Refs #17543 for the related closed JSON-RPC classification work. Refs
#17272 for provider recovery in the legacy OpenCode adapter. This change
adds display evidence for native Runner providers and keeps recovery
policy separate.

**What happened?**

A native Claude session returned an HTTP 429 message below an ACP
`Internal error`. The task and run showed only a generic startup
failure. Operators had to inspect the provider response to learn that
the account was rate limited.

**Expected behavior**

The task and run show the sanitized provider message, provider name,
startup or turn phase, and available HTTP status. Unknown categories
keep the message. Display evidence does not grant a retry or replace a
recovery decision.

**Steps to reproduce**

1. Start a native Claude run whose account is rate limited during model
selection, or return a structured provider failure from a Codex or
OpenCode turn.
2. Let the run fail and open its task page.
3. Before this change, the page shows a generic failure. With this
change, the page shows the provider explanation. The run page shows the
same explanation after reload.

**Paperclip version or commit**

Source checkout based on `f8a888a072`.

## What Changed

- Add one JSON Schema for `ProviderFailure` and generate matching Runner
and shared TypeScript readers. Keep the Runner package independent of
the app workspace at runtime.
- Normalize known ACPX, Codex, and OpenCode error fields. Preserve
unknown messages. Redact known credential values and credential patterns
before applying the 4,000-character limit. Do not copy response bodies,
headers, stacks, or prompts.
- Carry startup failures through the sidecar, Rust Runner, durable
command receipts, and private transport error details. Keep provider
text out of generic transport error strings and Sentry failure messages.
- Keep turn failures in authenticated run events. Recover the same
evidence from persisted events during server finalization. Store the
cause in the company-scoped run result.
- Show one shared notice on failed tasks and run details. Keep existing
retry permissions, scheduled retry state, and resolved-task behavior.
- Reapply registered run-secret and current-user redaction to task
display data. Batch secret lookups for only the visible provider causes.
- Document the contract and update the capability maps. Add regressions
for nested Claude 429 errors, OpenAI quota and rate limits, OpenCode
errors, redaction, durable receipt replay, restart finalization, and UI
rendering.

## Verification

- `pnpm -r typecheck` passed. `pnpm build` passed. All GitHub Actions
build, test, package, and validation jobs passed on `d49c79ab4a`. The
security checks passed. Greptile gave this head 5/5 with no open review
threads. Paperclip Review remains pending. Its review worker exhausted
six workspace recovery attempts with export_interrupted. A fresh review
request was accepted and is waiting for that saved workspace. The
ordinary Retry action rejected reuse of the stopped request. This is an
external review-worker blocker; no check was overridden. The duplicate
local `pnpm test:run` was stopped after the CI test shards passed.
- Focused server settlement, provider evidence, and restart tests: 93
passed. Runner admission and normalization tests: 103 passed. Focused
Runner wire and terminal tests: 29 passed. The real Rust/sidecar command
receipt replay test passed.
- The new projection test failed before implementation because the task
had no provider cause. The startup producer regression also failed
before extraction moved ahead of the generic error-code replacement.
Both pass with this change.
- Live check: a rebuilt local sidecar received an actual Claude HTTP 429
during startup. The new cause kept the full provider message. A
synthetic task in an isolated local app then reached the same provider
error through the native Runner path. The dedicated run result, task
API, task banner, and run detail all showed the same message after
reload. The later reporting fix keeps run.error generic; a regression
proves the display cause still survives restart and stays out of Sentry
capture.
- The first isolated app attempt failed earlier in
`admission_credential`. An ordinary manual Retry reached the provider
and produced the 429 above. This PR does not claim to fix that separate
admission failure. An initial parallel local database test setup also
failed; the serial restart suite passed.
- Security review found that copying display text into run.error could
forward it to Sentry. A regression failed with that behavior. The fix
keeps the cause in its display field. All 38 settlement/restart tests
and the server typecheck pass.
- All 29 protocol contract tests pass, including bounds and extra-field
rejection. An earlier full Rust core run passed. The later full run hit
one 5-second receipt-limit timing assertion; its isolated rerun passed.
The full Linux and Windows Rust CI jobs passed on this head.
- Review regressions pass for explicit Codex driver identity, native
Rust startup and resume receipt propagation, oversized OpenCode/ACPX
errors, task read-time registered-secret redaction, and the standalone
source closure. The focused Runner run passed 103 tests;
projection/restart tests passed 16.
- Read-time redaction edge cases passed: a missing decryption key omits
only the unsafe optional explanation; expanded redacted messages remain
visible within the Unicode character limit. All 14 task projection tests
and server typecheck pass.
- The package Build check found an old Pi assertion against
Error.message. The probe now checks the exact closed classification in
the authenticated receipt and requires the new provider cause. Both
polling transport cases pass. A local credential-free Pi probe stopped
at admission_installation because this checkout lacks the Pi
distribution; the installed Linux package probe passed in CI on this
head.
- UI coverage includes startup and turn text, unknown categories,
literal rendering of provider text, read-only retry behavior, and
resolved-task suppression. Token gates and map checks passed.
- The live check covers Claude startup on macOS. Codex, OpenCode,
terminal failures, and restart paths have deterministic regression
coverage. The change is not deployed.

## Risks

- Provider messages are untrusted text. The boundary selects specific
fields, redacts credential patterns and known environment secret values,
bounds the message, and renders plain text. New provider-specific secret
formats may need added redaction patterns.
- Optional fields preserve compatibility with older producers and stored
runs. Old runs without the field keep their current fallback text. No
database migration is required.
- The cause stays in company-scoped run data. This change does not send
provider text to first-party Telemetry or add it to OpenTelemetry span
attributes.
- Normalized categories and retryability are display evidence. Existing
recovery classifiers, retry limits, ownership checks, and approval gates
keep their authority.

## Model Used

- OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and browser verification. The host does not expose the exact
serving variant or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 06:13:26 -06:00
DottaandPaperclip 492d537340 fix(native): recover controllers across webserver replacement (#18343)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native execution runs in a remote Runner while the webserver owns
its controller.
> - Encrypted PRP checkpoints and session snapshots already survive a
webserver restart in Postgres.
> - Ordinary warm and per-turn admission still needs local markers and
supervisor metadata to find that state.
> - Replacing the webserver with an empty native controller directory
can therefore lose a usable retained session.
> - This change adds bounded recovery metadata to the existing snapshot
transaction and reconstructs local projections from the selected durable
source.
> - The benefit is native execution continuity across webserver
replacement, with the existing ownership and agent lifecycle policies.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Native remote controller recovery and retained-session continuation
during a webserver upgrade.

**Subsystem affected**

server/ — native runtime, checkpoint persistence, recovery scheduling
and internal upgrade reporting. Product E2E qualification also changes.

**Current behavior**

Without both the local PRP marker and supervisor file, normal
continuation can return before discovering the saved Postgres
checkpoint. The snapshot already exists in Postgres, but its local reuse
compatibility envelope cannot be reconstructed.

**Proposed behavior**

Select the existing durable source, validate its identity and ownership,
restore its PRP state and supervisor projection, then use normal
authenticated admission. Missing or inconsistent evidence holds
recovery. It does not authorize fresh provider work.

**Reason and benefit**

A replacement webserver must recover active execution and later continue
its retained session without the old server's native files. This is one
component of deployment continuity. HTTP traffic switching, workspace
storage migration and Railway rollout remain separate.

**Breaking changes**

No new public endpoint, schema migration or database pool. The private
run-profile envelope is additive. Disk-independent rollback also
requires a reader for the staged controller format and compact
checkpoint part; the first production release containing these readers
is the floor. Older file-backed sessions need an owner checkpoint or the
bounded internal importer.

**Additional context**

Related foundations: #17401, #17916, #17254, #14621, #14622. The
implementation contract and open acceptance gates are in [the dated
plan](doc/plans/2026-10-30-native-controller-checkpoints.md).

## What Changed

- Persist a versioned recovery envelope, at most 4 KiB, beside the
redacted session snapshot in its existing transaction. Keep it out of
public run projections.
- Compare frozen admission separately from prepared attachment
instructions during staged recovery, preserving exact
template/identity/authority checks.
- Bound the existing lazy checkpoint pool at three connections after
measured two-slot burst backlog; keep request/renewal capacities and SQL
operation counts unchanged. The tuning is a separate commit.
- Discover and restore the exact selected retained source without local
native markers. Preserve the 32-candidate window, progress barriers and
current authority checks.
- Fence snapshot and result writes. Acquire coordinator, issue and run
locks in order and check expiry after lock waits. Combine snapshot
ownership/read work into one statement.
- Reuse existing recovery deadlines and drain/resume scheduling.
Retained-owner waits preserve the original unstarted run and actual
lease deadline. Stop, ownership and provider-progress fences precede
deferral. Add bounded native evidence reporting, retained-source import
and retention protection.
- Admit a settled warm source with an authenticated idle goal-read
suffix while retaining active-work and mixed-authority holds.
- Permit lifecycle `turn.stop` checkpoints after accepted results under
existing ownership fences. Claim the existing successor coordinator
before retiring the old warm session, and stage shared attachments only
after verified retirement. Preserve omitted isolated-home defaults and
existing fresh skill-selection order during recovery without hiding real
configuration changes.
- Carry the exact durable six-field PRP binding during active recovery
after a healthy warm handoff. Keep the Runner mismatch guard and
preserve opaque attachment IDs.
- Preserve an already verified active or idle warm process during remote
preparation; revalidate its proof/liveness and retain later
authenticated PRP attachment. Ordinary resumed leases retain suspension
checks.
- Drain retained source events under their original authority, reconcile
immutable source accounting and block unknown source tool dispatch
before target work.
- Persist compact staged handoff intent through authenticated adoption
and recover direct/listener command-loss windows without another Runner
or duplicate attachment.
- Rebuild suspended lifecycle receipts for resumed prepared tasks, and
retry missing lifecycle reads after partial PRP materialization without
overwriting contradictory evidence.
- Add normal warm/per-turn diskless continuation coverage, real Postgres
races and SQL-cost checks. Add a reproducible paired load workload.
- Give only the explicit outage fixture a 15-minute provider idle
window, with a red/green catalog regression preserving all production
timing and lifecycle policies.
- Publish the goal-less task-session pointer in the existing
post-acceptance terminal checkpoint transaction, with current ownership,
accepted-result, reset and newer-progress fences. Keep a private bound
receipt so late completion cannot duplicate or resurrect publication.
- Use a retained warm sandbox in the dedicated continuity fixture, and
accept an exact output token followed by an explanation. Keep the strict
identity oracle.
- Extend the explicit Daytona cell to a five-minute active outage and
another replacement after completion. Require retained provider
identity, a new tool receipt and a useful follow-up result. Add an owned
loopback Postgres E2E fixture for hosts that cannot start embedded
Postgres.

## Verification

**Merge recommendation — `fc68584ce6231a02e47ecd743f55b036ccea034f`:
ready for the qualified native-controller component.** Full CI, fresh
review, local application paths, controlled performance qualification
and the strict live Linux/Daytona case pass. Independent resource
cleanup is complete. Broader rollout, browser reconnection and the
unexecuted transport/platform combinations remain integration gates.
Integrated 484 passed full local recursive typecheck/build and ordinary
active→completion→cache-loss→C component qualification. Fresh review
then found a valid attached-wake failure: preparation appends attachment
instructions before the verifier compares execution to the frozen
admission. 9f4 fixes that by separately checking the pre-preparation
admission and exact prepared execution. Both hashes are captured
privately per admission; the full template, identity, configuration and
authority checks remain. Its regression fails before the fix; 36 focused
cases, 42 staged source/target cases and server typecheck pass
afterward. The review thread is resolved. Exact-head full CI passes
([run
41168228822](https://github.com/paperclipai/paperclip/actions/runs/41168228822):
48 successful jobs, one intended skip); the complete `pnpm test:run`
partitions are forced, with the five native suites in the Runner lane.
This is not a claim that an unsharded local command passed. The fresh
exact-head Greptile check succeeds with 5/5 and all six threads
resolved.

The original two-connection database comparison closes **steady PASS and
burst FAIL**. Every relative CPU/WAL/request-p99 gate passes and all
operations/ownership remain correct, but candidate arm 3 drains in
1,015.99 ms and baseline arm 4 in 1,059.67 ms against the unchanged
1,000 ms absolute gate. Those results remain failed. A separate 2→3→4→2
diagnostic reproduces drain times 1,051.57 / 956.79 / 882.02 / 1,065.72
ms: three is the smallest tested passing cap, and the repeated two-cap
failure survives run-order/cache effects.

Separately reviewable `fc68584ce6` changes the existing lazy checkpoint
pool default to three connections. Request capacity stays ten and
liveness two; idle release, shutdown fencing, transaction counts and
pool count are unchanged. Thirteen DB lifecycle/cap tests and DB
typecheck pass. Full recursive typecheck/build pass at exact fc68584,
with the matching server build stamp. The final paired native
feature-cost comparison explicitly uses the **same cap-three policy and
common current DB factory in both variants**; its baseline is
controlled, not the unmodified historical cap-two configuration.
Original gates and workload counts remain unchanged. The one strict
Linux/Daytona case started at 04:40:56 UTC only after both new profiles
passed; it passed on the first attempt with all 49 workflow checks.
Independent cleanup confirms all three owned sandboxes absent, owned
Postgres/controllers/observer gone, VM terminated, root volume absent
and ephemeral SSH key/agent removed.

**Final controlled cap-three qualification: PASS.** Baseline native
implementation `59da15066a` and candidate `fc68584ce6` use the same
current DB factory and checkpoint policy (request/liveness/checkpoint =
10/2/3). Eight 65-second ABBA arms cover 300 controllers and 10,000
historical tasks. All correctness, absolute and 10% relative gates pass;
original cap-two failures remain unchanged.

| Mode | DB CPU µs/op, baseline → candidate | WAL B/op, baseline →
candidate | DB-request p99 ms, baseline → candidate | Maximum drain |
| --- | --- | --- | --- | --- |
| Steady | 542.40 → 554.83 (+2.29%) | 1509.01 → 1558.27 (+3.26%) |
107.11 → 111.90 (+4.48%) | 26.35 ms |
| Burst | 1036.36 → 1063.11 (+2.58%) | 4153.12 → 4317.64 (+3.96%) |
103.736 → 103.841 (+0.10%) | 930.14 ms |

Every matched arm preserves operation/statement/transaction counts,
including 3,900 PRP and 3,900 snapshot transactions. All three actual
pools are included; there are zero lost owners, missed lease deadlines,
dropped operations or errors. Peak Postgres CPU is 18.31%/9.47% of two
cores and memory 614.29/684.42 MiB (steady/burst). The benchmark uses
one Node process simulating 300 controllers; it does not qualify
two-webserver overlap, HTTP latency or hosted capacity. Its observer SQL
contributes to measured database CPU. The original unmodified cap-two
baseline is separately retained as failed; this cap-three comparison is
explicitly controlled.

The exact final-source 8/512/8,192-chunk test also passes: zero recovery
SQL, transactions or changed rows in each text-only interval, zero idle
SQL, seven lifecycle transactions and twelve changed rows per sample.
Existing lease traffic remains separate. The final ordinary managed
application replay passes active recovery, completion, native-cache
removal and C continuation; its UNKNOWN local debug-daemon provenance
and partial instruction-observer limits remain recorded. The live case
uses the attested Linux producer instead.


**Live qualification: PASS, one attempt, no retries.** Campaign
`native-controller-retained-fix-fc68584ce6-ec2-20261031T044055Z` started
04:40:56 UTC on the exact final Linux webserver source; suite definition
hash `12f9bb97823c1537e43f0398e7cddf39890f77bce7b413ca4dfd944f4a30e5d5`.
The existing immutable image and attested Bookworm Runner artifact are
unchanged.

- A was SIGKILLed and its native cache discarded. B started after
300,149 ms, naturally acquired generation 2, authenticated at 316,624 ms
and resumed at 317,682 ms within the original 420,000 ms grace. The same
active run, Runner, provider PID/start fingerprint and one
submitted/started/accepted turn were preserved; one final
result/terminal and workspace finalization completed, with no
retry/recovery action.
- After active and queued completion, B was SIGKILLed and its native
cache discarded again. C received exactly one ordinary task-comment API
follow-up, completed a new tool and produced a useful reply while
preserving native/provider/Runner identities. This closes the earlier
failed post-C gate. A separate C OS-process PID field was not exported;
the physical-PID proof is specifically A→B.
- Four distinct results were accepted and published; all four
finalizations committed at attempt 1, all four accounting-v2 projections
settled, and the task-session pointer advanced to C. The coordinator
independently checked these facts and all 49 canonical checks against
the sanitized receipts.
- Canonical cleanup passed and independent checks confirmed three exact
sandbox 404s plus complete inventory/zero owned remnants, no owned PG or
controller/observer processes, EC2 terminated, root volume absent, and
the ephemeral SSH key/agent removed by 04:54:25 UTC.

Live scope is classic remote Codex warm continuity with non-native
storage dependencies available. Prepared-task completed B→C, full
per-turn application disk loss, browser reconnection, two-server load
and overall Railway/HTTP continuity were not live-qualified here.
Earlier failed campaigns and the original cap-two burst failure remain
failed historical evidence. A report-renderer cache setup failure was
preserved; the normal sanitizer subsequently succeeded with an explicit
cache path and a new output directory, without changing validators.


The repair keeps retained events under their original six-field
authority until persistence and accounting accept them. Unknown semantic
effects stay unresolved. A compact staged target survives source
bootstrap, cached-command delivery and authenticated handoff, including
an uncertain commit response. Direct and same-guest-listener recovery
require the verified original Runner and the exact endpoint. Target
snapshot publication marks readiness in memory, with retirement
coalesced into an existing command write. Headers contain only the
staged descriptor/digest; a prequeue template is written once as a
changed checkpoint part and then recovered from the existing
command/receipt. No second session snapshot transaction, streaming SQL
path, new pool or recurring idle scan is added.

| Current repair gate | Evidence and scope |
| --- | --- |
| Source-tail regression | Real Runner protocol RED reproduced
attachment rejection. Frozen core/transport pass five source-tail cases:
source usage attribution, held persistence/no premature ACK, lost commit
response plus cache loss, unknown semantic hold and accounting rejection
hold. |
| Handoff loss windows | Direct and listener modes pass
queued/prepared/activated/completed boundaries (eight cells), preserving
the actual Runner/provider process. Missing adoption proof and changed
endpoints reject recovery. |
| Compact storage | A 31 KiB attachment grant writes its separate part
once, leaves it unchanged across source snapshots and removes it at
queueing. Headers do not duplicate the grant. |
| Accounting | 21 real-Postgres usage tests and 19 source-binding tests
pass. Recovery hydrates immutable source receipts without resetting
their producer or billing classification; conflicting/incomplete usage
holds. |
| Admission and malformed evidence | Driver 16, staged evidence 42,
restart classifier 27 and source usage 19 focused cases pass. Four
malformed-file regressions pass at d6a7, whose executor is
byte-identical at 484, and server typecheck passes. Full executor file:
990 pass; 16 embedded-Postgres `initdb` setup failures before
assertions, retained separately. |
| Required checks | Exact fc68584 recursive typecheck and build pass
locally, with matching server stamp. Forced full CI succeeds (48 jobs
pass, one intended skip); all PR checks are clean. Maps/token gates pass
for their unchanged source surfaces. Fresh exact-head Greptile is 5/5,
all six threads resolved. |
| Ordinary application path | Exact fc68584 managed component passes
active recovery after natural expiry, completion, native-cache loss and
ordinary C continuation. Production bytes match before/after; temporary
fixtures restored and owned DB/services cleaned. UNKNOWN local
debug-daemon/partial instruction-observer limits remain; the separate
live case uses the attested Linux producer. |
| Producer | Locked Bookworm build from exact 5b7 Rust tree
d03e494217, equal to final fc68584 Rust
source. SHA-256
65b8f8cfdd6099e9ad24f1ca154e8a260777269cc82f6e23c6e6a180c531eaaf;
38,515,328 bytes. Actual Debian 12 execution and live admission pass.
Controller/admission/DB changes do not relabel the producer source. |
| Historical cap-two comparison | Exact 59da baseline vs 5b7 candidate,
300 controllers/10,000 history, ABBA 4×65 s per mode, all three actual
pools and real snapshots. Steady CPU/op 541.36→554.64 µs (+2.45%);
WAL/op 1512.13→1560.99 B (+3.23%); DB-request p99 102.78→105.99 ms
(+3.12%): PASS. Burst CPU/op +4.71%, WAL/op +5.29%, p99 −0.10% pass
relative gates, but absolute drains above fail. Identical operation/SQL
counts, zero lost leases/drops/errors. One Node process simulates 300
controllers; this does not qualify two-webserver overlap or HTTP
capacity. |
| Live acceptance | Strict final-source Linux classic Codex case passes
1/1 on attempt 1, all 49 checks. Active recovery within grace and useful
retained C continuation succeed; four
accepted/published/committed/settled outcomes. Independent provider,
PG/process, VM/root and SSH cleanup complete. Prepared/task and broader
rollout limits are explicit above. |

**Historical qualification snapshot through `9eb9ac6`.** The head at
this snapshot was `9eb9ac6f9e312f3ebe8040f5a7565e88ae8824f3`. Server and
Runner package sources are byte-identical to `283e05ffaa`; the two later
commits change the explicitly scoped qualification and its documentation
only. It includes master through the storage-owner fix
`45a3a92e6dc52d20923b9aa34a22fee2ec5443f9` (#18357). At that snapshot,
CI/review passed but the strict post-C native recovery gate failed; the
PR stayed draft and required another native repair. Prior `39a4343` CI
and review passed, but its full live workflow failed; that grade remains
unchanged. The first repaired `435ccd2` live attempt failed before any
injected outage. It exposed a legitimate terminal snapshot before result
acceptance; no accepted result ID or publication receipt was observed.
Its canonical report remains 0/1 `candidate_failure`, with cleanup
passed and independent provider absence confirmed. `0862062` defers
publication until result acceptance and fixes the fresh review's
coordinator/issue lock-order finding. Both failures have deterministic
real-Postgres regressions. The `0862062` live attempt then proved
accepted-result publication but failed active recovery: a resumed-lease
preparation guard demanded suspension after the exact live Runner was
already verified for adoption. `64c0b746` preserves that verified
active/idle-warm process, revalidates its proof/liveness, and keeps
ordinary suspension checks and later PRP authentication. Its live run
passed preparation but exposed a second active-warm boundary: all
durable binding fields matched except opaque turn/item attachment IDs,
which the server had regenerated. `7a203274` carries the complete
verified six-field binding and keeps the strict Runner mismatch guard.
Its cloud case then passed active adoption, completion, publication and
accounting. The queued continuation exposed accepted-result retirement
writes and an implicit-home fingerprint drift; both are fixed with
deterministic regressions in `39a4343` before this fresh full
qualification.

The prior live case exposed a real publication gap. A goal-less native
turn could persist its accepted result and terminal snapshot before
provider cleanup, while the task-session pointer was still empty. Normal
later admission then selected a fresh session. The fix publishes that
pointer and a private bound receipt in the **existing post-acceptance
terminal checkpoint transaction**, before cleanup. Earlier terminal
observations persist without a handled receipt or continuation pointer.
No result-acceptance or snapshot transaction was added. The late
completion path recognizes the receipt. This also prevents a late
callback from restoring a pointer after a user reset. Goal/control
publication keeps its existing owner.

The dedicated Daytona fixture now uses the existing warm/reusable
environment. Its sandbox, Runner and provider storage must survive. The
former generic per-turn fixture disposed the sandbox after completion
and kept provider-history backup bytes under the discarded directory.
Whole per-turn disk-loss acceptance therefore has a separate
provider-backup storage dependency. It remains distinct from the passing
native per-turn component checks with backup storage available. The
60-second lease, 300-second outage and 420-second grace are unchanged.

| Gate | Current repair evidence |
| --- | --- |
| Assembled regression | Baseline fails with an accepted result and
terminal snapshot but a NULL task pointer while real provider cleanup is
held. The fix passes normal continuation and reset cases, with no
pointer seeding. Exact merged `92b7447` passed 2/2 twice, including
after normal package TypeScript output refresh. Production files are
byte-identical at frozen `435ccd2`. |
| Persistence and concurrency | 19 helper/transaction cases and all 59
existing control-port cases pass on `0862062`. They cover bound result
IDs, reset/newer epochs, Stop, lock waits and expiry, receipt replay,
missing stubs, foreign bindings and atomic 4 KiB rollback. Existing
snapshot transaction count stays one. The new tests reproduce the exact
live missing-result-ID error and the inverse replay/status lock order
against `435ccd2`, then pass after the correction. Replayed results
still reject stale generations. |
| Completion | All 32 completion-boundary tests pass on merged
`92b7447`; the source is unchanged at `435ccd2`. Native result-shaped
session metadata is preserved. |
| Qualification fixture | A valid review report reproduced three
rejected correct replies. Exact token-boundary matching fixes them. 140
focused E2E/catalog cases and Product E2E typecheck pass. Wrong tokens,
marker prefixes/suffixes and wrong authors remain rejected;
retained-identity checks are unchanged. |
| Required checks | Full recursive typecheck and standard build passed
at `435ccd2`. Both exact commands also passed on `0862062`, with its
exact server build stamp; both also pass on `7a203274`, with its exact
server build stamp. Fresh `0862062` Greptile passed 5/5, with zero
unresolved threads. The `64c0b746` normal CI then passed all 49 jobs (48
success, one expected skip) and Greptile passed 5/5 with zero unresolved
findings. The `7a203274` normal CI also passed 48 jobs with one expected
skip, with fresh Greptile 5/5 and zero unresolved findings. Current
`39a4343` recursive typecheck and standard build both pass with
persisted exit codes and its exact build stamp; fresh Greptile is 5/5
with zero unresolved findings. Its normal CI completed successfully: 48
jobs passed and one expected job skipped. The fresh live result remains
separately failed. The `0862062` normal CI had 46 passing jobs, one
expected skip, one worker communication failure and its aggregate
failure; GitHub states the self-hosted runner lost communication. No
failed assertion or specific OOM/spot cause was established, and no
blind retry was used. |
| PostgreSQL supplement | One corrected all-pool pair at the preceding
repair passed 300-controller/10,000-history correctness and absolute
gates. SQL 76,590→75,990; transactions 9,013→8,713; UPDATE rows 11,613
each; 300 correct pointer writes each. PG CPU 31.99→30.46 s, WAL
70.36→67.87 MB, request p99 100.98→103.12 ms. This is one pair, not a
repeated improvement claim or an exact `0862062` CPU measurement. Its
two later changes defer premature publication and reorder existing
locks; they add no transaction, stream/idle writer, pool or timer. The
first collapsed-pool attempt remains invalid and retained. This is not
an exact final-head CPU/WAL measurement; later fixes change ordering,
identity selection and in-memory skill replacement without adding
snapshot transactions, pools, stream writes or idle SQL. |
| Live publication proof (`0862062`) | Baseline run
`06fcc806-f186-4190-9b57-087b61d75533` accepted result
`58f912dc-94d9-4b4d-bd3d-d5f689c83fd3`; the publication receipt names
the same result, the task pointer has parameters and `lastRunId`
matches, and the normalized native identity matches the following active
run. Accounting v2 settled. |
| Active recovery (`0862062`) | **FAIL.** SIGKILL/cache discard at
23:07:28.666 UTC; hold 300,000.859 ms; B ready at 308,896.486 ms;
natural generation-2 ownership at 312,906 ms. At about 319 s,
preparation failed `runner_harness_state_mismatch:
resumed_not_suspended`. No authenticated after-receipt passed within 420
s. Exact original Runner PID 668/start/fingerprint was independently
still alive; provider auto-stop was 15 minutes. No retained C follow-up
qualified. Canonical 0/1 `cleanup_failure`/cleanup-failed report remains
failed; subsequent independent cleanup deleted all three owned resources
with exact 404s and confirmed full intended-account inventory with zero
owned resources. |
| Verified adoption preparation (`64c0b746`) | 91 focused cases pass,
including actual preparation for four active providers and a distinct
prior/current warm proof. Unproved resumed state, dead process and stale
warm owner remain rejected. The earlier options-only fixture missed this
gate. No spawn, backup restore or PRP-authentication bypass was added.
Server typecheck passes. |
| Active warm identity (`64c0b746`) | **FAIL.** B claimed generation 2
at 312,995 ms. Preparation passed, then strict PRP recovery rejected the
generated attachment IDs at about 319 s. The saved and desired Runner,
lease, normalized session and run IDs matched; only
`turn_attachment_…`/`item_attachment_…` differed from generated
`turn-<run>`/`item-<run>`. Canonical 0/1 cleanup-failed report remains
failed; independent cleanup confirmed deletion/404s and zero owned
provider resources. |
| Six-field identity repair (`7a203274`) | Actual server-factory
regression RED4→GREEN38. A real Rust Runner test passes healthy first
run → random-ID warm handoff → held active second run → restored
controller. It verifies a new live authenticated connection, exact
identity, same Runner/provider OS identities, no duplicate launch/turn,
and no redispatch or invented receipt for an uncertain tool. The
synthetic wrong pair still fails the unchanged guard. This protocol case
does not claim business completion. Server and Runner typechecks pass. |
| Active recovery/completion (`7a203274`) | **PASS for the active
component.** Full five-minute SIGKILL/cache-loss hold. B ready at
308,936.651 ms; natural generation 2 observed 312,979 ms; authenticated
at 317,720 ms and resumed processing at 318,527 ms. Same run
`1e1ccb00-0728-4935-97da-57562d81cb42`, Runner
`246f73f7-a689-4177-a9c2-7ce711d63aed`, remote PID 669/start
23:48:27.470, provider lease/session and one submitted turn. Accepted
result `49097529-5310-4ac3-af95-5bf9b91d19f4`, publication, finalization
and accounting v2 all settled. |
| Full workflow (`7a203274`) | **FAIL, preserved.** Queued
late-requirement run `5e61f38d-c7d1-4fc1-bcbf-bce9d46ce4eb` failed
`runner did not durably suspend before checkpoint` before outage 2.
Canonical 0/1 candidate failure, cleanup passed, evidence valid. C
retention was not exercised. Independent provider absence,
controllers/PG cleanup, EC2 termination and root-volume absence all
confirmed; closed receipt SHA-256
`676a3b15eb9937b6bf995c36fb8666c88964d509e5ceb9109fb02da0aa2bc909`. |
| Retirement and stable configuration (`39a4343`) | Real Runner +
Postgres RED reproduced the exact suspension failure and nine owner
rejections. `turn.stop` is now lifecycle checkpoint maintenance under
unchanged owner/result/generation fences. GREEN25/25 proves actual
persisted suspension, exact identity and completed stop/drain/suspend;
unaccepted, foreign, stale and new-work negatives remain rejected. Two
default-home fingerprint REDs become GREEN9/9; explicit mode changes
still invalidate. No global matching exemption or historical fingerprint
rewrite. |
| Fresh full Daytona warm cell (`39a4343`) | **FAIL, preserved.**
Launched 00:15:29 UTC with automatic retries disabled. Active adoption
passed: actual offline hold 300,000.848 ms; B ready 309,922.155 ms;
natural generation 2 at 311,821 ms; authenticated 317,989 ms; resumed
330,440 ms. Same run `3e126819-e786-4b3b-b923-c975cf4f5b79`, Runner
`4079138a-8d7c-44a4-8558-07a5411d1f55`, PID 668/start and
provider/session identities. Accepted result
`08e504af-4dec-48a8-bf05-bf0a32280559` and publication passed. Queued
run `31d168c9-7ab9-4700-abb6-fafb90912237` failed the same strict
suspension error before outage 2. Its row selected a new native
identity, but no new third provider process or invocation was observed.
Recovery-time configuration still reset only `adapterConfig`; native
provider, context and workspace subtrees were unchanged. Canonical 0/1
candidate failure, evidence valid, normal cleanup passed; all campaign
provider/PG/process resources independently absent. Closed campaign
receipt SHA-256
`5e7d6281c13c812f3a946667f97f3c7d339b5ad86a3a819bd185b2fa73bc4a19`;
EC2/SSH remain temporarily held for the concrete follow-up repair.
Original 60/300/420-second limits and strict queued/second-replacement
checks remain unchanged. Direct-process path; no new live prepared-task
claim. |
| Native retirement authority (`283e05ffaa`) | Exact `39a4343` facade
regression RED3→GREEN9. The successor now uses its existing production
coordinator claim and prompt renewal before strict retirement. A real
Runner/Postgres case starts in the actual native-observed/unclaimed
phase, claims through production code, then closes and durably suspends
the predecessor in 1.657 s. All 25 real-Postgres cases pass;
unclaimed/foreign writers remain rejected. Shared attachment staging
stays after retirement; Stop and denied claims neither close nor stage.
No new claim or SQL path was added. |
| Configuration continuity (`283e05ffaa`) | Real shared/project-primary
environment selection → configuration with four skill versions
reproduces `reset: true`, changed category exactly `adapterConfig`,
after a Postgres JSONB read. Pinned restoration had reordered the
generated preference while actual runtime skills stayed identical.
Replacing existing pinned rows in place preserves the original fresh
name/key order and fingerprint. The assembled regression passes, as do
three existing pinned-version/rename/alias cases and server typecheck.
Genuine model/skill-selection changes still invalidate. No
fingerprint-format change or broad compatibility exemption. |
| Final production repair (`283e05ffaa`) | Full recursive
typecheck/build and [normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41151145106)
PASS: 48 jobs plus one expected skip. Fresh review 5/5, zero unresolved
threads. Live baseline, recovered run and queued continuation all
succeeded, each accepted/published/committed once with accounting v2
settled. Their native/provider identities and configuration fingerprint
match; both recovery and continuation report no reset, no changed
categories and task-session reuse. Active outage/authentication remains
within original timing. |
| Browser-dependent continuation (`283e05ffaa`, definition 6) | **Whole
cell FAIL, preserved.** After B was killed at 00:46:38.928 and its cache
discarded, C became ready in 9,151.410 ms. `openParent()` then hit a
30-second `page.goto`/DOMContentLoaded timeout before posting the final
request. No fourth native run or C continuation was exercised. Cause of
navigation timeout is unknown. C task had no execution lock, queued
comments, handoff or recovery blocker; all three prior runs were
committed/accounted. Normal cleanup passed; independent provider
absence, controller/PG cleanup, EC2 termination and exact root-volume
absence confirmed. Sanitized report bundle SHA-256
`2a3b841d62efb2ae30ef14f96a431ac4fa7fe3a782f1144831512f78edc65022`. |
| Native API continuation qualification (`9eb9ac6f9e`, definition 7) |
**FAIL, preserved.** The real post-C API request dispatched run
`c71fbb18-f979-4fe4-93ce-9c4be46dbf21`, which failed
`native_session_interrupted` / `native_session_recovery_failed` at
01:10:00.640 UTC. It selected the same normalized native identity,
Runner and checkpoint-provider identity as the three succeeded turns. No
C result was accepted; pending accounting remains explicit. Active
recovery passed: actual offline hold 300,001.098 ms, B ready 308,335.019
ms, natural generation 2 observed 311,818 ms, authenticated 315,540 ms,
resumed 316,612 ms. Queued continuation succeeded/accepted/published. B
was then killed/cache discarded at 01:09:34.500 and C ready in 8,783.803
ms. Original classifier `transient_infrastructure`, whole-case failure
and cleanup passed are unchanged; that classifier does not establish the
root cause. All owned provider/PG/process resources, EC2, root volume
and SSH agent independently absent. Closed receipt SHA-256
`7f21ac4a22efc07b849fa7e17866083b6127b3321af12671b4fc054ce57039a9`;
sanitized bundle
`71ffef14a30a553e008320a31840a6d6c0de85c1bfdb60f783fdb53a6aae812a`.
Exact backend detail is unavailable in approved structured evidence. |
| Current required checks (`9eb9ac6f9e`) | Recursive typecheck, standard
build and exact build stamp PASS. [Normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41152908417)
PASS: 48 jobs plus one expected skip. Fresh Greptile 5/5, latest edited
summary checked, zero unresolved threads. Exact8/512/8,192 stream probes
again report zero recovery SQL/transactions/rows and settled-idle SQL;
seven control transactions/twelve changed rows each, with 36 separate
legitimate renewals in the longest case. CI success does not supersede
the failed live C gate. |

The completion supplement observed secondary request/liveness/checkpoint
transaction p99 increases of 23.0%/45.7%/15.6% in its single pair.
Renewal p99 stayed below 0.865 s, with 900 successful renewals per arm
and no lost leases. The named pool caps stayed 10+2+2. An aggregate
activity sample peaked at 15 non-observer entries without backend-type
classification, so no claim is made that all were client connections. No
sampled lock episodes does not establish zero lock waits.

Twelve pure reader/selector cases also pass with the new adjacent
receipt. This is not another executed process rollback. Development
reader `5cf7aead` can parse the recovery state, but its old run-detail
redactor can expose receipt IDs/outcome to an authorized run reader. The
first production release containing recovery readers, complete
attachment-identity handling and both private-field redactions is the
rollback floor. The implicit-home fix prevents new recovery-time
fingerprint drift; it does not rewrite previously stored fingerprints
from failed candidates. The development probe is not a published
supported rollback target.

Latest actual `0862062` Runner tests independently passed 8/512/8,192
text chunks with zero recovery SQL, transactions and changed rows, plus
zero settled-idle SQL. All three observed seven setup/control
transactions and twelve changed rows. The 8,192-chunk case separately
recorded 31 legitimate lease-renewal statements, with no lost lease.
These are component SQL observations, not a production CPU/WAL claim.
The new adoption and identity branches affect recovery preparation only.
The `64c0b746`, `7a203274` and exact `39a4343` CI independently passed
all three stream sizes with the same zero-SQL/idle result; each latest
8,192-chunk case recorded 36 separate renewal statements. Exact
`39a4343` Runner 2/2 passed 133 files / 2,317 tests plus two skips; its
separate native group passed 150 tests plus eight skips.

The first API fixture revision omitted the mandatory final screenshot;
this was caught before paid launch and allocated no provider/model work.
The correction captures the actual retained Chromium page with an
explicit unqualified-browser label, without changing the standard report
validator.

Automatic approval review rejected exporting raw private API/workflow
snapshots because of sensitive payloads and insufficient
destination-specific authorization. Sanitized reports and
bound/hash-only receipts are retained; raw snapshots were not copied
through another channel. This does not change the failed grade.

Repair characterization also preserves its failed attempts. An
unconditional staging relocation broke three existing same-scope
scripted cases; the final implementation defers staging only for an
actual superseded warm scope, and all nine order/race cases pass. One
baseline transform overlapped an edit and is explicitly invalid; the
actual immutable `39a4343` loader then produced the three genuine REDs.
The independent task-directory hypothesis produced a RED but was
excluded by the live shared-workspace evidence; no directory policy
change was landed. A shared-workspace fixture authorization omission was
corrected separately from the four-skill semantic RED.

Setup failures are retained separately from semantic outcomes. These
include an old host port after the owned Postgres fixture restarted and
a sandbox-denied TSX IPC pipe. The supported external-Postgres run
passed, and the final exact recursive typecheck passes with normal
process permissions. Seven stale declaration errors after the master
merge cleared after compiling the actual merged SDK/Runner TypeScript
sources. No declarations were edited by hand.

<details>
<summary>Earlier qualification measurements and failed attempts
(historical, before this repair)</summary>

**Historical blocked gate (before the repair below).** Final native
implementation is `abed921e12d543b62f57a56435419c0dafb0bc33`; current
branch head is `af9e2d157dcc349e55374ad186c9be2d6cb96bf9`. The two later
commits change only qualification fixtures, their matcher and
documentation; `server/` and `packages/` are unchanged. It includes
master through `fd008979f5794fda3d5f54151d36acd31d0de140`.

The live Daytona measurement proves five-minute active recovery,
once-only completion and accounting, but **fails retained-session
identity after the second replacement**. The saved checkpoint and
recovery metadata exist. The task-session pointer remains empty, and
ordinary admission creates new native/provider sessions. The exact
skipped publication boundary remains unproven. A useful saved-file
result alone does not satisfy retained-session continuation. This
application-path gap blocks merge independently of CI.

A separate new-master CI failure is owned by the storage workstream (fix
in #18357). No storage guard, session-identity oracle, lease/grace limit
or provider result was relaxed.

| Gate | Observed result and exact scope |
| --- | --- |
| Distinct source recovery and compatible rollback | PASS at A
`5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → B
`5ea45f8d17e33c9a62bd79982dcf0375b07733d0` → fresh B → A. Four
controller processes, natural 60-second lease expiry, absent native
projections before replacements, three succeeded/committed unique
results, one provider thread, unchanged Runner identity, and strict idle
cleanup. 177.782 seconds. |
| Immutable installed native component | PASS at installed A
`5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → installed B
`eaef0d56552c087aaa4fa68a8655282ffbd12667` → fresh B → A. Each
controller imports its contained compiled server and DB. Three unique
committed results, stable Runner/thread, no extra provider restart on
same-contract rollback, strict cleanup. 106.672 seconds. This is
compiled component proof, separate from full CLI-native application
acceptance. |
| Normal public CLI startup | PASS for B
`eaef0d56552c087aaa4fa68a8655282ffbd12667`, Node 24.22.1, darwin-arm64:
fresh external DB, 415 migrations, health ready, root HTML 200, company
POST 201 plus GET readback. The installed tree remained unchanged after
execution. |
| Five-minute component outage | Actual controller absence 300,168 ms.
Natural generation-2 claim at 407.196 seconds after crash, same provider
attempt and Runner, original run succeeded/committed. The larger
540-second case timed out before its retained follow-ups, so it is an
active-adoption pass, not a complete four-controller pass. The complete
short source/installed cases above independently cover follow-up and
rollback. |
| Drain/resume review regression | Fixed in
`226f03e736d1ee99fea1762ab32ea3a8a9241c20`. Five facade-level cases
pass: overdue normal/restart waits, future timer cancellation, another
facade resuming the original owner, and transient retry. Against exact
prior scheduler `5ea45f8d17e33c9a62bd79982dcf0375b07733d0`, four fail
and one passes. The later head changes documentation only. |
| Final corrected-fixture Daytona | **PARTIAL: active recovery PASS;
retained identity FAIL**, exact live source `0b6afc499f`. A was
SIGKILLed and native cache discarded; actual offline hold 300,000.375
ms. B was ready at 308,326.961 ms, naturally claimed generation 2 at
312,059 ms, authenticated at 315,996 ms and resumed processing at
316,519 ms, within the original 420-second boundary. The same run,
Runner PID/start, session, lease and event prefix survived; one turn
submitted. Original execution succeeded and settled accounting. After
completion, B was killed and its native cache discarded; C was ready in
9,077.779 ms. C executed the saved project and returned correct
`queue_r`, with all four runs succeeded/accounted. However, native and
true provider-session identities changed: the unchanged retained oracle
fails. No whole-workflow pass is claimed. |
| Prior matched-producer Daytona | FAILED at exact `abed921e12`. The
matching current-source Runner passed normal admission, completed the
first task and settled its accounting. During the active revision, A was
SIGKILLed and its native cache discarded. Actual offline hold was
300,000.727 ms; B was ready at 308,304 ms and naturally claimed
generation 2 at 309,651 ms. At about 318 seconds, the remote process
check reported `process_not_running` for the original admitted Runner.
No authenticated after-receipt or retained follow-up passed. The current
failure is preserved; no automatic retry, lease/grace relaxation, debt
edit or provider replay is used. Bounded forensic read found preserved
exact-bound state with no recoverable failure, a transport-reset
diagnostic and no terminal delivery; this supports external process loss
without proving the exact cause. The sandbox had a five-minute auto-stop
policy, conflicting with the required surviving-sandbox assumption.
Owned cleanup precedes the separate corrected-fixture case. |

**Live retained-session boundary.** Prior completed run `3806…` and C
follow-up `8adf1df3-6744-40bf-b35d-100451f29d9f` both have database
session snapshots and `nativeControllerRecovery`. Their tool
fingerprint, provider configuration, workspace descriptor,
runtime-context digest, driver and work mode match. There is no recorded
force-fresh/workspace-selection change, and this is an ordinary task
rather than a conversation-generation reset. The same issue/agent
task-session row has `session_params_json: null` and `last_run_id: null`
from baseline time. Both native normalized UUID and actual provider
session/thread hashes differ after C. Full application admission
therefore did not select the retained source. No fallback to file
execution or relaxed identity comparison is accepted as proof of
continuity.

**Independent cleanup completed.** Full current-account provider
inventory confirms no resources owned by either final fixture, with
exact-ID absence checks. Owned controllers and campaign Postgres are
stopped/removed. EC2 `i-01551c09af098dde5` is terminated, its
delete-on-termination root volume returned `InvalidVolume.NotFound` at
22:14:08 UTC, and the temporary SSH key/agent were removed. The
canonical failed report and immutable oracle input are retained. Closed
qualification receipt SHA-256:
`b4b5fefce287ebfdbf2cc2238efaad62d8ad485aed0e9c0628ae95dbc18a8575`.

**Original measurement and grader repair remain separate.** The C reply
contains the requested marker followed by a newline and the correct
output. The old workflow waited for one literal space, so its normal
receipt collection stalled; that original attempt was closed and its
canonical `cleanup_failure`/interrupted result is preserved. Commit
`af9e2d157d` repairs only this reply predicate, with 144 focused cases,
E2E typecheck, discovery and diff checks passing. Ordinary string
matching stays literal. The corrected predicate accepts the saved reply,
but the unchanged retained-identity oracle still fails. Pure
active-recovery grading of the captured measurement passes 7/7; this is
not a rewritten passing campaign.

The sandbox idle-window conflict was corrected only for this explicit
fixture (15 minutes). Actual baseline and later sandbox metadata confirm
that policy; the original active box was disposed before a post-gate
policy read, so its 15-minute setting is established by the normal
create/environment configuration rather than an after-read of that box.
Production native lifecycle and the 60/300/420-second constants remain
unchanged.

The development compatibility reader A is a genuine backport commit
based on `59ff89a18fe1312b5f0bb908439bbdb1237b0252`, with the required
idle-goal reader fixes. It is not a previously published release. Its
older executor differs from B. The rollback floor must include this
reader capability. The source component cases admitted the Linux ARM64
Runner packaged at `59ff89a18fe1312b5f0bb908439bbdb1237b0252`; the
installed component case admitted the `5cf7aead` package. Their Runner
bytes are identical: SHA-256
`b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8`. Each
admitted artifact remained unchanged throughout its case.

[Final public package
build](https://github.com/paperclipai/paperclip/actions/runs/41121460635)
and [reader-floor
build](https://github.com/paperclipai/paperclip/actions/runs/41122170506)
both passed all four daemon builds, package assembly and both Darwin
clean consumers. Both Linux ARM consumers failed before provider work
with the same existing local Codex permission-profile error: bubblewrap
attempts `activity.fifo/.git` and reports “Not a directory.” This
matched baseline failure remains open; it is not a native remote
continuity pass. Full post-run tree and build-stamp revalidation also
passed after the compiled native case. The installed component trees are
A `c29114c1bf71c5af00f761aeb9e3d84ed3d3075b9f5600dee3ef2543eb2b6ec5` and
B `eda575505373a3bdc630fa4919862b359791fe3a7ef637961a25b73b38efe4b0`.

**Controlled PostgreSQL qualification passed.**
Baseline/candidate/candidate/baseline workloads used 300 controllers,
10,000 historical tasks, real snapshot/encrypted PRP writes and
unchanged pools. PostgreSQL and client were each limited to two CPUs and
1536 MiB on an isolated eight-vCPU Linux host.

| Workload | PG CPU per operation | WAL per operation | Database request
p99 |
| --- | ---: | ---: | ---: |
| Steady | +1.18% | +1.81% | +1.91% |
| Aligned burst | +3.40% | +2.56% | +3.81% |
| Two logical servers | +3.11% | +2.56% | +9.68% |

All arms passed absolute/correctness gates with unchanged control
transaction counts, no dropped operations and no owner loss. Overlap p99
has little margin below the relative threshold. It uses two clients in
one Node process, not two HTTP servers. [Closed measurements and
limitations](doc/plans/2026-10-30-native-controller-qualification-evidence.json)
and exact-source reproduction patches are retained. [Earlier
contended-host
failures](doc/plans/2026-10-30-native-controller-load-evidence.json)
remain unchanged. The production DB-claim probe claimed 300 owners in
2.84 seconds and restored/committed controls and snapshots in 8.74
seconds, with zero owner loss or settled-idle SQL; its provider
references were synthetic. The measured checkpoint/snapshot/renewal
paths retain the same persistence behavior after integration; later
accounting and business writes have separate owners.

Actual Runner stream tests at `da8c7f6e1837ddd896dfe83a7524812fee813487`
passed 8/512/8,192 chunks: zero recovery SQL, transactions and changed
rows during text intervals; zero settled-idle SQL. Each retained seven
control transactions and twelve changed records. Existing renewal SQL
was measured separately. The guard/scheduler fixes add no stream path,
pool or recurring idle scan.

The live fixture used current-source producer SHA-256
`f0000c90cb83880191479da639201e659aa6ce0fc7d67c859faa172ec977849e`, a
normal locked exact-head build with pinned Rust 1.97.1 in Bookworm.
Actual network-isolated, read-only ABI execution passed (GLIBC
requirement 2.34 against runtime 2.36). The immutable base image
remained
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:36069788f53c51f228764fa36180575443dfb5d1f84a342d7c0daceb28dc61cc`,
source `a0ef3ce7`. This was an unpublished development producer,
explicitly distinct from the base image and admitted before the fresh
fixture began. The earlier host build failed ABI preflight before
admission. The preceding old-producer fixture completed its initial task
but lacked the usage-basis marker required by current master; it
remained pending/unpriced and correctly paused. It was independently
cleaned without debt or budget changes. Two preceding local Linux setup
attempts OOM-killed before provider allocation, and their cleanup
passed. These attempts remain separate from the final recovery failure.

**Failures retained and resolved separately.** The earlier `98f05fc`
probe rejected a valid settled PRP v2 goal suffix; `eaef0d5` fixed that
reader and the fresh follow-up passed. Later component cleanup failures
were traced by a scoped exception observer to the fixture adapter
rejecting directory downloads. A bounded owned-directory transfer with
exact-byte/mode preflight corrected the fixture; source and installed
strict-cleanup cases then passed. No production guard was relaxed. The
initial local integrated control-port run hit its 30-second embedded
Postgres startup hook before executing 59 cases; 18 other focused cases
passed. A bounded rerun used a fresh owned external Postgres fixture,
applied normal migrations before Vitest, and passed all 59 cases with
the original timeouts unchanged. Its container and data were removed;
the failed attempt remains retained.

The additional opt-in full Product E2E Vitest run is not green: 5,313
passed, seven failed and three skipped. An actual `abed921e12` archive
with the same dependencies, Node and IPC permissions reproduces all
seven failures with the same reasons. Five are unchanged
workflow-policy/profile-pin assertions, including Pi profile version 37
versus the test pin 35. Two database suites failed during embedded
Postgres startup before their assertions; semaphore exhaustion is
confirmed in the observer log. The observer passes its supported
external-Postgres rerun with original timeouts and source unchanged. The
controller-database client-kill/postmaster assertions remain unexecuted
after `initdb` bootstrap failure; no external override is supported. The
unit launcher stopped before its Node-test phase, which was then run
independently and passed 181/181. The original failed full run is
preserved. These results remain distinct from the passing focused checks
and normal repository CI. The first local Product E2E typecheck also
encountered stale compiled Runner declarations; a genuine package-local
TypeScript compile refreshed them and the unchanged check passed,
without a dependency install or producer rebuild.

Earlier [Daytona campaign
41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900)
failed at sandbox creation before any agent, with the underlying
provider error unavailable. Its [sanitized
report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-41108162900-2/index.html)
remains a `cleanup_failure`. Both old creation names are absent in the
current authenticated account, but the old CI account identity was not
retained, so account equivalence is unproven. The subsequent exact
`eaef0d5` source Product E2E attempt passed image/agent setup and the
initial project, then held the controller offline for 300,003.7 ms. Its
Mac supervisor failed owned-process inspection before B readiness; no
adoption passed. Full current-company lease/label checks, provider
deletion, exact-ID 404s and complete inventory confirmed cleanup of that
attempt. Its local server and owned Postgres stopped. The prior AWS
campaigns and database qualification VM were terminated with root
volumes removed.

The old admitted test Runner lacks the newer usage-basis marker
introduced on master. The integrated component B turns retained
unpriced, incomplete accounting obligations with `costAccountingPending:
true` and no settled charge. The compatibility-reader A turn used its
legacy receipt interpretation but likewise retained pending accounting
and no settlement timestamp. This proves obligation preservation, not a
whole-run accounting migration. Runner/receiver release ordering remains
governed by the usage owner's contract.

At native implementation `abed921e12d543b62f57a56435419c0dafb0bc33`,
Greptile was 5/5 with zero unresolved threads and CI completed with 54
successful checks and five expected skips. At `0b6afc499f`, Greptile
again passed 5/5 with zero unresolved threads, but CI ended with 51
successful checks, five skips and three failures. Server 9/12 fails a
deterministic managed-skills maintenance assertion newly inherited from
master `198218bf` in merge `c80e00ad`; that test is absent from the PR
head, and the storage owner has the explicit-StorageService fix in
#18357. Runner 2/2 passed 2,283 unit cases but its runner received a
shutdown signal during the long stream probe; its cancelled step and
aggregate failure are preserved. No blind retry was used. Current
`af9e2d157d` checks are pending; the matcher-only change passes 144
focused cases and Product E2E typecheck. Earlier scheduling-sensitive CI
failures and the interrupted unsharded local test run remain recorded in
the dated plan. CI runs the repository's normal test partitions. Final
verification does not claim Railway rollout, uninterrupted HTTP service
or whole-webserver filesystem loss.

</details>

## Risks

- The task-pointer repair passes its assembled regression. The first
live repair exposed pre-acceptance ordering and the fresh review exposed
a replay lock-order issue; both are fixed and pass real Postgres tests
at `0862062`. The following live attempt exposed the
verified-running-versus-suspended preparation gate. Its narrow fix
passes 91 focused cases at `64c0b746`. The next live case exposed
synthetic attachment IDs; `7a203274` fixes that field selection and
passes real Runner authentication. The subsequent queued continuation
exposed strict retirement maintenance and a false default-home change;
`39a4343` fixes both with real Runner/Postgres and fingerprint
regressions. The `39a4343` live continuation failed again despite green
CI. `283e05ffaa` repairs the actual native-observed admission ordering
and reproduces/fixes the JSONB skill-order fingerprint drift. Both pass
targeted real-Postgres regressions and the live active plus queued
paths. The definition 6 post-C browser navigation timed out before
admission. Explicit definition 7 now verifies native continuation
through the ordinary comment API; its strict post-C native gate failed
after a real dispatch and remains historical failed evidence. The final
fc68584 repair now passes its fresh strict 49-check Linux/Daytona
campaign; the prior failed grades remain unchanged. Browser reconnection
is handed to its own workstream with the original failed evidence. No
prior failed campaign is reclassified as passing.
- Incorrect checkpoint/source binding or stale ownership could duplicate
external work. Missing evidence remains a hold; existing receipt and
finalization owners keep uncertain effects unresolved.
- Recovery must retain maintenance writes after accepted results without
allowing a stale business turn to write. Real lock-wait and generation
tests cover that boundary.
- The current two-connection paired burst profile fails its absolute
drain gate in both baseline and candidate, despite passing relative load
gates and all correctness checks. A causal capacity diagnostic selected
three slots; the final controlled cap-three profiles pass unchanged
thresholds. Full application replacement and two-server overlap capacity
remain separate. There is no new streaming writer, pool or recurring
idle scan.
- The remote sandbox and Runner must survive the recovery window.
Provider auto-stop is a separate lifetime bound from reconnect grace;
the explicit qualification fixture uses a longer idle window.
- Workspace and provider backup availability remain external
dependencies. This PR does not certify whole-webserver filesystem loss,
uninterrupted HTTP service or a Railway rollout.

## Model Used

Implementation and initial verification used OpenAI GPT-6 through Codex;
the initial hosted snapshot was not exposed. The follow-up qualification
used three OpenAI `gpt-6.1-sol` workers coordinated through Codex, with
repository tools, code execution, CI and scoped test infrastructure.
Context-window size is not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused and exact-source
application/cost checks pass; full forced CI passes; earlier local
embedded-PG setup failures remain recorded)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (exact fc68584 full CI and
auxiliary checks pass; forced complete test partitions)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(exact fc68584 check and edited summary pass; all six threads resolved)
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-31 07:33:52 -05:00
DottaandPaperclip 9f524ec1a7 fix: persist native Runner usage before transport acknowledgement (#18354)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent execution must preserve usage so Paperclip can price work and
enforce budgets.
> - Remote Runner execution can outlive its web receiver.
> - The receiver previously acknowledged Runner events before its later
usage recorder saved the accounting receipt.
> - This pull request saves required native usage receipts before that
transport acknowledgement.
> - A replacement receiver can accept the same retained report and
create one ledger charge.

## Linked Issues or Issue Description

**What happened?**

Native remote Runner usage could leave its durable outbox before the web
receiver saved its accounting receipt. The later recorder required a
local receipt directory. A replacement web receiver could lose the only
remaining report when the database write failed.

**Expected behavior**

The Runner retains unaccepted structured usage. Postgres owns accepted
receipts. Repeated delivery preserves one receipt for each source report
and one charge for the run. Missing usage keeps accounting pending.

**Steps to reproduce**

1. Start a supported built-in native remote Runner.
2. Make the receipt database insert fail after the provider reports
usage.
3. Replace the web receiver and its local directory.
4. Reconnect the same Runner and inspect the saved receipt and ledger.

**Paperclip version or commit**

The initial investigation used
`ebd05e9f4c88c66af7473db86bbaacfd1544aa0a`.

**Deployment mode**

A web server with a remote native Runner and shared Postgres.

Related work: #18327 owns output logs and provider traces. This change
has no dependency on its S3 export path. #18031 covers interrupted Codex
receipt recovery. #18005 covers legacy spool reads. #17516 covers Codex
stop-boundary retention.

## What Changed

- Add native producer receipt delivery through the existing Runner
callback before transport acknowledgement.
- Reuse `run_usage_receipts`, stable source sequences, and deterministic
report identities. Reject conflicting replay and retry a lost commit
response with the same receipt.
- Restore per-turn usage from structured receipt facts. Preserve
provider turns that share one PRP envelope turn.
- Bind retained warm transport callbacks to the current run authority
epoch, allowing old attachment control acknowledgements while preserving
accounting ownership.
- Persist Codex session baselines and cumulative totals in the existing
provider checkpoint. Mark qualified whole-run deltas, preserve them
across resume/restart/attachment, and leave old unmarked reports
incomplete and unpriced.
- Normalize the bounded final usage tail before the terminal event.
Retain later increases for explicit reconciliation, block further
attachment, and fail an affected successor without changing its
baseline.
- Complete runs from the accepted immutable receipt and its exact saved
rate-card amount; preserve existing live-budget enforcement.
- Skip the local receipt recorder for supported built-in native remote
Runner execution. Keep local, legacy, and external Dot/Muse behavior
unchanged.
- Keep pricing, reservations, ledger settlement, source fencing, and
budget holds with their existing owners.
- Put the actual Runner integration test in the existing Rust-cached
native test lane. Build before opening Postgres and give cold
compilation its own bounded budget.

## Verification

- Fresh Greptile review is 5/5 on
`e7376a50c8162770da1d581ecf25c8cffac64520`, with no unresolved findings.
Parent and independent peer review approved the final changes.
- Real PostgreSQL 18.6 and a real Rust Runner exercise failed receipt
insertion, receiver-directory loss, replay from the same producer, and
exactly one 25-cent ledger charge. The actual accounting spool path
stays absent. Optional logs/traces are disabled. The provider is
controlled; no paid request is made and the fixture never pre-saves the
expected receipt.
- Producer checks passed 32 usage/provider tests,37 terminal controls,
corrected completion, all 3 previously failing exit cases, and both full
transport goal-continuation cases. Rejected active-goal completion
retains its 120-token tail before failure. Baseline 100→successor150→33
duplicate old snapshots→old120 retains explicit incomplete
reconciliation evidence and positive budget exposure.
- Heartbeat executor checks passed 982/982. PostgreSQL tests preserve
exact 1.002 USD pricing, the 100.2-cent ledger amount, budget holds,
source takeover, partial/final receipts, cost-only corrections, equal
rate-card replay, and stale-completion interleaving. Newer incomplete
usage remains pending rather than settling an older snapshot.
- Runtime replay and strict usage-schema checks passed 2/2; generated
contracts and Rust formatting pass. Test-partition contracts passed
32/32.
- [Final combined-head
evidence](https://github.com/paperclipai/paperclip/pull/18354#issuecomment-6531526158)
records full workspace typecheck/build and 41/41 accounting tests
against all five cloud PR heads, including this final accounting commit.
It includes commands and evidence limits.
- Exact-head CI is green: 53successful checks, 6intentional skips, and
the legacy Snyk status passed. Typecheck, build, Runner, server,
browser, and aggregate gates pass. Runner Vitest1/2 initially failed an
unchanged mocked Antigravity login fixture; its focused case passed
locally, then one unchanged job retry passed 131 files/1995 tests,
including all 11 Antigravity tests. The initial failed attempt remains
recorded in CI.

Initial local attempts exposed the host SysV semaphore limit and
corrected harness path, IPC readiness, and queue-counter defects. A
temporary test loader selected an existing POSIX-semaphore PostgreSQL
18.6 build without changing repository dependencies. An optional
expanded debug provider run was interrupted at its unrelated 8192-turn
rollover stress case; no full-file local pass is claimed. Required
failure cases were run directly, and the full Runner CI remains
required.

## Risks

- The guarantee covers built-in native remote Runner execution. Local
native execution, legacy adapters, external Dot/Muse execution, and
server-originated model calls retain their existing contracts.
- This does not provide traffic switching or general web-server
failover. The existing controller identity, checkpoint, and generation
gates still apply.
- The receipt JSON gains optional structured producer facts. No schema
migration or accounting S3 spool is added. Text output and raw trace
payloads are excluded.
- Codex accounting authority requires a Runner artifact built from this
provider baseline implementation and qualified by digest. Update
receivers and strict usage-schema readers before releasing that Runner;
PRP does not negotiate this payload field. Existing unmarked Codex
reports remain pending/unpriced. No deployed Runner artifact is claimed
fixed.
- Usage arriving beyond the bounded terminal tail may require manual
reconciliation or a fresh session. Its evidence remains in the existing
producer checkpoint/outbox; no adjustment ledger or recurring scan is
added.

## Model Used

OpenAI `gpt-6.1-sol` assisted with the implementation, tests, and
documentation through code execution and repository tools. The context
window size is unavailable.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-30 15:20:42 -05:00
ec3b558277 fix(server): schedule accepted native response recovery (#17696)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs store accepted results and their finalization decisions.
> - A completed run can still owe its public answer after an
interruption.
> - Periodic recovery scans for that answer even when no new work
exists.
> - This pull request wakes recovery from committed changes and saved
retry deadlines.
> - The existing owner still verifies the result, source, and current
publication authority.
> - Settled or dormant response work no longer needs its own periodic
scan.

## Linked Issues or Issue Description

Refs #17690 (merged). This is slice 48 of the sleep migration stack, now
based on master. Related: #17661 handles GitHub terminal publication;
this change handles native accepted-response presentation.
Governed-review presentation remains a separate duty.

**What happened?**

Native finalization recovery repeatedly checks already accepted results
for a missing answer. Removing that check without complete notifications
would lose responses when evidence arrives late, an agent resumes, or
admission settles after execution.

**Expected behavior**

Recover the answer from its saved result when a supported change makes
it eligible. Retain real retry deadlines. Stop querying when the answer
is selected or cannot currently be published. Reconstruct work after
restart.

**Steps to reproduce**

1. Commit a native accepted result with no selected response comment.
2. Run finalization recovery while its source is unavailable or locked.
3. Restore that source or release the lock. The answer must appear once
without repeating provider execution.

## What Changed

- Honor recorded non-create decisions and skip-comment policy in both
discovery and recovery. Apply durable conversation/result/continuation
guards before recovery locks. Preserve genuine unanswered and legacy
recovery.
- Scope routine response wakes to the affected issue/run. Coalesced
input names its active run; pending-effect settlement names its issue.
Sixteen existing global authority/configuration hooks use one explicit
company invalidation boundary. No new transaction wrappers or tables are
added.
- Add an accepted-response worker using the existing delivery
coordinator and scoped deadline helper. Preserve saved retry deadlines
and pending-effect fences under the owner's lock.
- Share the finalization notification boundary. Wake on finalization
settlement, exact accepted-result/final-message evidence, agent
availability, retry admission, and existing chat authority changes.
- Keep lifecycle policy in services. A narrow pre-write status hook lets
lifecycle retain ownership of its transaction and status transitions.
- Register the worker during enabled maintenance startup. Remove its
authority subscription on shutdown. Periodic replay skips this duty on
both committed and fresh-commit paths; ordinary finalization and
explicit recovery retain their behavior.
- Add a partial index and expression statistics for unpresented native
runs. Migration 0412 supports concurrent replay and adds no queue table.
- Add real PostgreSQL and lifecycle coverage. Preserve the
implementation and producer audit in the plan document.

## Verification

- Additional real Photon callback acceptance on the published production
code disproves the security warning: ten active assertions cause zero
response-recovery hints/SELECTs, while three concurrent assertions
during an actual attention-to-active recovery cause exactly one
wake/SELECT. The existing locked status guard already enforces this
boundary. No production change was needed; the resolved finding now has
a successful current-head security check. The regression will be
retained with the next slice.
- Eligibility repair source `892f36e5f66ba1af123034936a7160c64c3ab883`
passes **1489 distinct server cases**: 1139 in six complete
recovery/finalizer/monitor/summary/workspace suites (81.02 seconds),
plus 350 selected chat cases (297.68s). Another 1463 chat cases are
outside the pattern; the complete 1813-case file is not claimed. All
eleven new regressions are included in that count. Five dedicated fresh
PostgreSQL 17 databases migrate through unchanged 0412, with no
fixture-source adaptations. Database build, server types/build,
boundaries and full-SHA migration ordering pass. All seven test source
hashes match.
- The exact published baseline first fails eight behavioral assertions
while the three genuine-answer cases pass. The same corrected test
source passes all eleven after repair: recorded none/reuse remain
suppressed; five ineligible history families take zero recovery row
locks across repeated issue hints; missing/legacy/create-null answers
still publish once; fresh periodic finalization defers presentation to
its worker. Earlier fixture setup and variable/compiler failures remain
explained in the audit, rather than erased by later success.
- Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821
server cases**: 1496 in 13 complete producer/lifecycle/authority suites
(110.63 seconds), plus 325 selected chat cases (278.99 seconds).
Coverage includes all native-session cases, queue admission, status
decisions, membership/board access, the complete GitHub workflow,
accepted-response lifecycle and real authority restoration. The other
1477 chat cases are outside this selection; the complete 1802-case file
is not claimed for this repair run.
- Actual PostgreSQL regressions first reproduce a coalesced-input hint
missing its run scope and two routine hint paths producing 12
response-history SELECTs each after three notifications. The same
assertions pass with run-scoped coalescing and zero response-history
SELECTs for unscoped routine hints. Actual membership restoration still
publishes the waiting response once. Assertion time limits are
unchanged; this measures real driver SQL and persisted producer work.
- Eleven dedicated fresh PostgreSQL 17 databases migrate through 0412.
One embedded-only wake-queue fixture import temporarily uses its own
external database and restores byte for byte. All 14 selected source
files match pre-run hashes. This qualifies external-database behavior
rather than reproducing embedded CI locally.
- Master advanced independently with #18126's Pi archive retry fix.
Normal merge `2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both
complete binary patches. All accepted server/database/UI/test/CLI trees
stay identical; the three independent runner installer/manifest files
are retained. All 13 installer tests pass, including stalled header/body
cancellation. Combined server types/build, boundaries and full-SHA
migration order pass. A subsequent normal merge of #18128 at
`e8b78cce5bb557b086bee9ca930eeaab1d09b244` retains its three test-only
file changes with both complete patches preserved. The changed HTTPS
backfill case passes on a fresh PostgreSQL fixture (1 selected case; 178
outside the pattern), and combined types/build, boundaries and migration
order pass again. All 14 repair source hashes remain unchanged. The
independent browser fixture changes are preserved; no local browser run
is claimed.
- The earlier source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passed
the 4171-case full baseline before this scope repair. That scope is
preserved in the audit rather than represented as a current full rerun.
All 0412 database bytes remain unchanged: original concurrent
index/statistics/ANALYZE, prior executable history, journal entries and
retained snapshots. Five snapshots 0408 through 0412 remain. The
populated 0411-to0412 upgrade preserved 410 prior receipts and 38
companies, appended one receipt and none on replay; index/statistic
checks passed.
- The earlier P2 security failure, Greptile's former conditional
dependency on #17702, and the subsequent actual Ponytail eligibility
findings are recorded in
`doc/plans/2026-10-25-native-response-presentation-sleep-audit.md`. This
PR now includes the issue-scoped pending-effect notification fix itself,
so it is independently correct before #17702. Fresh published-head CI,
Greptile 5/5 and a clean security scan remain required. No
deployed-sleep, external-wake or fleet-savings claim is made.


## Risks

- The writer contract is process-local. Independent SQL writers and
other replicas are not covered by these notifications.
- Unpresented backlog reconstruction still has memory and time cost
proportional to candidate count. Pages contain at most 100 rows;
execution uses the shared fair batch. Selected history is excluded.
Health-only signals do not trigger reconstruction.
- Publication policy remains with its current owner. Digest, company/run
binding, source authorization, selected-answer suppression, and
ownership fences remain required.
- The rest of native recovery and governed-review presentation still run
separately. This PR does not relax Cloud sleep eligibility or add
external wake coverage.
- The stack parent has merged and its actual ancestry is reconciled.
This publication still needs fresh current-head CI and review.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution, and PostgreSQL tests. The runtime does not expose a more
specific model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 02:31:20 -05:00
DottaandPaperclip ba5bba86dc fix(server): wake GitHub acknowledgements from committed changes (#17656)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/206/changes/paperclip/206
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip coordinates agent work and presents its progress in chat.
> - GitHub deliveries save acknowledgement actions and provider
receipts.
> - Periodic maintenance currently searches for unfinished
acknowledgements even when idle.
> - Committed delivery, lease, response and presentation changes
identify when work can advance.
> - The existing owner should wake on those changes and recover only at
saved deadlines.
> - Completed history should become quiet without losing unfinished
provider effects.

## Linked Issues or Issue Description

Refs #17653, which is merged. This existing GitHub acknowledgement slice
targets master.

**Problem:** A saved GitHub acknowledgement can wait for a publication
lease, fast-response result or native presentation decision. Polling
empty or unchanged work keeps the tenant active. Removing maintenance
without covering these producers can strand the acknowledgement.

**Expected behavior:** Committed changes wake the scoped owner. It
reconstructs existing actions on startup, retries at saved deadlines,
parks unchanged presentation waits and becomes quiet when settled.

## What Changed

- Register GitHub acknowledgement work with the shared scoped deadline
coordinator. Reuse existing action, delivery, lease and fast-response
rows.
- Add transaction-aware response-availability notifications at supported
writes. Keep GitHub protocol details in its platform owner and shared
notifications limited to IDs.
- Share the bounded latest-request/run projection between scheduling and
direct response authorization. An absent native presentation decision
parks the action until its real writer commits.
- Retain publication leases, source/principal authority, version
fencing, response markers and final receipt precedence. Check admission
before each new provider request; let in-flight work settle. Stop does
not consume the provider retry budget.
- Remove only recurring acknowledgement recovery and keep its manual
path. Add no work table or scheduler service.
- Regenerate the three original indexes as
`0400_absent_phantom_reporter.sql`: pending acknowledgement IDs,
fast-response delivery lookup and publication comment/destination
lookup. Each uses concurrent/idempotent creation and the existing
top-level migration runner. Preserve all landed SQL/journal entries and
the five newest snapshots.

## Verification

Published head: `09ca00b9f029838434ec042cda691345d67b493e`, on landed
continuation parent `063cfd43813cabf18417605fb9eefb9aacf95c8d`.

- **87 distinct current cases pass:** 18 real PostgreSQL acknowledgement
lifecycle/presentation cases, all 26 fast-response service cases, all 30
heartbeat completion cases and all 13 startup cases. Database build,
server typechecking/build, module boundaries and three fresh migrations
through 0400 pass. The current Runner bundle rebuild also passes after
upstream provider-event changes.
- The lifecycle matrix covers empty/history quiet, lease release,
restart, drain/standby, fast-response failure/expiry/publication/unknown
outcome, streaming quarantine, outer rollback/commit, saved provider
retry, stop between requests and receipt state. The real presentation
writer proves quiet while pending, rollback, committed reactivation and
final quiet.
- The earlier **305-case** matrix (236 full GitHub workflow cases, 26
fast-response service cases, 30 completion cases and 13 startup cases)
passes at `337f272853d4f8a51b5b9300900a46eb4e13f698`. It retains its
prepared-parent scope. Repeated cases are not added to current 87.
- A populated 0399 database upgrades to 0400 with all 398 prior
migration receipts unchanged, one new receipt and no additional receipt
on replay. All three indexes are valid/ready on fresh and upgraded
databases. Their SQL definitions match the original draft after
whitespace normalization; migration history and retained snapshot bytes
are preserved.
- Normal integration `9b3607e159153e00ccdfe538d52b591bd38d6499`
preserves the full 28-path owned binary patch and all 20 unrelated
master paths, including native retry, provider-event and evaluation
workflow changes. Both normal merge parents and the exact tree are
checked. No force push or rebase occurs.
- [The owner
audit](https://github.com/paperclipai/paperclip/blob/09ca00b9f029838434ec042cda691345d67b493e/doc/plans/2026-10-25-github-chat-maintenance-sleep-audit.md#acceptance-on-the-actual-landed-continuation-parent)
retains the original missing-notification, JSON precedence, receipt
lifecycle, external-database fixture and migration-registry failures.
Earlier 69/228/broad counts keep their original scope. Skips and failed
revisions are not relabelled as current passes.
- The final audit-only edit leaves all owned source/test/migration
hashes unchanged. **Fresh exact-head CI and Greptile 5/5 with no
actionable findings remain required.** No full local repository, live
GitHub tenant or deployed sleep result is claimed.

## Risks

- Process-local transaction hints cover supported application writers
and shared pools. Independent SQL writers and other replicas do not gain
a new wake guarantee.
- Provider effects remain guarded by the existing publication lease and
current source/principal authority. Unknown outcomes retain their
existing recovery semantics.
- Presentation waits depend on their supported milestone/transaction
producers. A missing producer can defer recovery until another supported
change or restart.
- Index creation can take time on large histories and runs concurrently
outside a transaction. Existing migration recovery handles replay; prior
SQL is unchanged.
- Worker quiet is distinct from host/database suspension. Cloud wake
coverage and sleep admission remain unchanged; later maintenance duties
are separate slices.

## Model Used

OpenAI Codex, using reasoning, repository inspection, code execution and
real PostgreSQL tests. The exact model identifier and context-window
size are not exposed in this session.

## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 16:06:42 -05:00
DottaandPaperclip 5f0da562db fix(server): wake chat lifecycle milestones from committed changes (#17606)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected chats show safe notices when a run starts, waits,
finishes, or fails.
> - A recurring scan projected these notices even when no eligible work
existed.
> - Eligibility depends on committed run state, conversation lineage,
and current authority.
> - Existing database rows can reconstruct that work after restart.
> - This PR wakes a scoped worker from those writes and removes
lifecycle polling.
> - Native progress events keep their separate existing owner until the
next migration.

## Linked Issues or Issue Description

**What happened?**

The chat maintenance timer repeatedly scanned run lifecycle state. Empty
and settled instances still queried the database. A run-status event
alone cannot replace the scan: prompt receipts, continuation receipts,
identity changes, and restored authority can make a run eligible later.

**What did you expect?**

Committed changes wake projection. Startup reconstructs eligible work. A
completed pass stops until another relevant change arrives. Failed work
retains recovery without scanning settled history forever.

**Steps to reproduce**

Configure an automatic chat endpoint. Finish its work, or leave it
unused. Observe recurring lifecycle projection queries from the chat
maintenance callback.

Stacked on #17592. Related foundation: #15625. This PR owns lifecycle
milestones only.

## What Changed

- Preserve upstream coalesced Cloud membership writes together with
locked membership restoration and its committed chat notification.

- Signal milestone eligibility from the real Slack account-link
transaction.

- Register the lifecycle worker with the existing delivery coordinator.
Remove lifecycle projection from the periodic chat callback. Keep native
progress in a separately named lane and preserve the explicit combined
API.
- Reconstruct company/issue scopes from the existing selector. Drain
bounded batches with company fairness. Retry failures, retain writes
through drain/resume, and disarm settled work.
- Signal from supported run, presentation, interaction, receipt,
lineage, conversation, recovery, and authority writers. Preserve their
transaction ownership, including nested savepoints and uncertain
settlement. No new table is added.
- Preserve provider authorization, authored-final suppression,
responsible-user continuity, deduplication, the 24-hour selection
window, and raw-output restrictions. Keep provider interpretation in its
owner.
- Preserve PostgreSQL timestamp precision in lifecycle pagination.
Rejected submillisecond candidates must not repeat the same page and
starve later notices.
- Document producer coverage and alternate-writer dispositions in
`doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md`.

## Verification

Published head: `1d056f68c5b1005933b9e12de8745bf1c4cbeeca`, targeting
master after #17592 merged at
`7d75ae0569e41786640cf09647ae8849411d2f22`. A normal merge integrates
the actual landed parent. The entire 77-path milestone patch is
binary-identical to the locally verified preparation across that
ancestry reconciliation; both unrelated master paths are preserved. The
four earlier producer/test integration conflicts retain scoped milestone
and voice notices, current onboarding completion admission, and expanded
identity query/read-scope coverage.

- **330 distinct selected cases pass on the verified unchanged patch:**
305 across 14 complete suites (all five milestone worker suites, run
identity, board claim, Cloud provisioning, auth session routes, Cloud
actor authentication, stale-lock recovery, GitHub credentials, project
Git authorization, and app admission wiring), plus 25 selected real chat
producer cases. The remaining 1,590 chat integration cases were skipped,
not passed.
- The real producer cases cover saved milestone discovery, GitHub and
Slack identity changes, account-link rollback/reconfirmation, ordinary
ingress reactivation, post-close action/delivery receipt settlement, all
ten authority-restoration paths including Cloud synchronization, and
changed native review metadata.
- Worker coverage verifies outer commit, rollback, uncertain-write
recovery, restart, bounded batching/fairness, timestamp precision,
retries, drain/resume, and settled SQL quiet. Authorization and existing
publication deduplication remain with their owning services.
- Server production typechecking, build, and module-boundary checks
pass. The tests above were executed separately. Production typechecking
excludes `src/__tests__` and does not replace those runs. Each database
owner uses its own newly created and app-migrated PostgreSQL fixture.
- [Current-parent acceptance and preserved failure
ledger](https://github.com/paperclipai/paperclip/blob/1d056f68c5b1005933b9e12de8745bf1c4cbeeca/doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md#current-parent-integration-and-acceptance-october-28)
records commands, counts, producer scope, conflict dispositions, and
limits. Earlier combined-stack counts and failures keep their original
revision scope; local acceptance does not erase old CI failures.
- No new migration, queue table, dependency, workflow, or shared
scheduler contract is introduced. Parent migration 0397, bounded consent
recovery, voice cleanup/retention repairs, and runner checkpoint
restoration remain present.
- **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No
Ponytail review is present, which is an explicitly authorized
nonblocker; no failing review is waived. Older reviews and parent checks
are not current passes. Actual failing Ponytail findings must be fixed;
an absent/stopped/paused turn without a verdict is an authorized
nonblocker. Active unfinished Ponytail uses its original request/start
timer, with a 25-minute monitor cutoff respecting the user's 30-minute
maximum.
- No complete local repository matrix, live provider tenant, external
alarm, physical host/database suspension, or fleet savings claim is
made. Native progress remains a separate following slice.

## Risks

- A future eligibility writer must notify the existing projection
boundary. Independent SQL writers or another active server process are
outside this process-local signal contract.
- Authority and lineage remain sensitive. The worker uses the existing
selectors and publication policy; it does not grant execution or
disclosure authority.
- Native progress and other publication maintenance still have recurring
work. This PR removes one reason, not the entire chat maintenance timer.
- Worker quiet does not prove database suspension, host sleep, or
external alarm coverage. Cloud sleep blockers remain unchanged.
- Merge eligibility requires passing current-head CI and Greptile 5/5
after this actual-master reconciliation. Subsequent stack slices retain
their own gates.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window size are not exposed. Used repository inspection,
reasoning, shell execution, PostgreSQL tests, browser E2Es and GitHub
tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details available in this session)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the documented focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 11:07:54 -05:00
DottaandPaperclip d0c8f0b090 fix: explain project access denials and add an authorized repair path (#17509)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Low-trust agents may read only the projects in their allowed scope.
> - Workspace setup correctly stops a task outside that scope.
> - The task previously showed a generic failure, and agent settings did
not offer an additive repair.
> - This pull request explains the denial and lets authorized operators
add projects while preserving other restrictions.
> - The saved task then offers an explicit retry. Saving access does not
start work.

## Linked Issues or Issue Description

**What happened?**
A project access denial appeared as an ordinary failed attempt.
Operators could not add a second allowed project from agent settings.

**Expected behavior**
The task should explain the denied access and link to a working
permission editor. The editor should preserve existing scope and
restrictions, enforce operator authority, and retain drafts after a
failed save.

**Steps to reproduce**
1. Assign a low-trust agent to a task whose project is outside its
allowed projects.
2. Let workspace setup deny project access.
3. Open the task, follow Manage project access, add the project, save,
reload, and return to the task.
4. Confirm that the saved message remains and Retry task is a separate
action.

## What Changed

- Record a typed, non-retryable project denial before provider dispatch.
Project denials do not enter automatic continuation or provider comment
policy.
- Project current and historical setup denials into a safe task notice.
Keep unrelated failures, prior assignees, and approval/pause/budget
gates separate. Recheck current project authority, including standard
trust and root-issue grants. Accepted retries keep their working state.
- Add board-only project-access reads and additive saves under the agent
row lock. Enforce company scope, agent-management authority,
pending-approval restrictions, and an audit record for new additions.
- Add the production notice and searchable multi-project editor,
including read-only, saving, cancellation, and failed-save states.
Preserve the return-to-task query through canonical agent redirects.
- Register board-only project-access routes in OpenAPI with a strict
request schema.
- Add production Storybook examples and API/persistence/concurrency
regressions. Retain the approved design snapshot and update the behavior
maps and implementation contract.

## Verification

- Passed: `pnpm -r typecheck`, `pnpm build`, `pnpm build-storybook`,
`pnpm check:token-gates`, `node scripts/check-maps.mjs`, and `node
scripts/storybook-catalog.mjs`.
- Passed: 11 permission/projection tests and 30 run-completion tests
against an isolated PostgreSQL database. These cover historical
failures, concurrent additions, idempotent audit, preserved
restrictions, foreign projects, read-only users, agent self-grants, and
no automatic wakeup.
- Passed: all 27 OpenAPI route tests, including route coverage and the
new board-only request/response contract.
- Browser: the real app displayed a historical denial; Manage project
access opened the editor; saving retained the original project and
survived reload; Return to task showed the ready notice. Saving created
one audit event and zero runs. Explicit Retry returned 202 and reached
the separate sandbox requirement, which remained enforced. No external
provider was used.
- All five production Storybook interaction checks pass: blocked task,
save, cancellation, failed-save recovery, and read-only settings. These
use fixture callbacks; durable authorization is covered separately
above.
- `pnpm test:run` was attempted locally but stopped after repeated
embedded PostgreSQL startup failures caused by macOS semaphore
exhaustion. Full-suite CI is required before merge. The separate UI
suite passed all 9,095 tests. The 41 focused server regressions also
passed after rebasing onto the latest master. A further 26
recovery-classification tests pass, including a red-to-green regression
for automatic continuation after a project denial.

## Risks

- No schema migration. The new endpoint only adds same-company project
IDs and preserves the rest of the stored policy.
- The historical notice recognizes one exact setup denial message. It
does not display arbitrary run errors or disclose a different source
project's identity.
- The retry still passes all current admission and isolation checks. The
notice does not promise that other execution requirements are satisfied.
- Related open work: #13309 covers inherited secret restrictions; #11234
covers the broader repository catalog. This change does not replace
either policy surface.

## Model Used

OpenAI Codex (GPT-6), with code execution and browser tools. The precise
runtime model ID and context-window size were not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused server and full UI
suites; full-suite infrastructure limit above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-24 16:16:29 -07:00
DottaandPaperclip dff63551fe fix: resume unfinished work after repository preparation (#17047)
## Thinking Path

> - Paperclip manages agents that perform work for a company.
> - Task repositories are prepared at normal run admission.
> - A running native task can stage a repository while its work remains
unfinished.
> - The controller did not offer that saved setup as a continuation.
> - The agent could add a timer or wait for a user restart.
> - This change uses the existing same-agent continuation and status
ledger.
> - The accepted explanation and proven conversation remain available
after setup.

## Linked Issues or Issue Description

Refs #17045. That prerequisite has merged. This PR targets master.

**What happened?**

A projectless native task can request a repository for the next
admission. An unfinished task has no server-selected repository
continuation. An explicit same-agent continuation also loses chat
provenance.

**Expected behavior**

An unfinished standard task can yield and resume through the existing
admission queue. Its explanation and final result retain the authorized
conversation. A task that only stages a repository can complete without
another run.

**Steps to reproduce**

1. Start a native standard task in a task directory.
2. Request an authorized repository through prepare_repository.
3. Yield with unfinished work and no explicit continuation.
4. Observe that the former driver rejects the yield, although the
pending repository is saved.

## What Changed

- Offer a saved same-agent continuation for the current task's pending
repository.
- Accept that current server continuation in the shared provider driver.
- Resolve chat provenance through applied status decisions, delivered
effects, and admitted wakes.
- Preserve accepted setup explanations without repeating an owned
question card.
- Document optional reproduction details and configured repository
discovery.
- Extend database regressions and workspace access maps.
- Preserve Copilot profile v52 and bind the shared continuation source
in v53. Include the active identity in the QA image fingerprint.

## Verification

- Five focused server suites: 100 checks pass.
- Shared continuation driver: 12 checks pass.
- Server typecheck passes.
- Map contract and whitespace checks pass.
- Existing API and driver assertions fail on the former implementation.
- Real dispatcher and queue cohort: 43 checks pass. The original
reply-loss regression fails before the dispatch receipt fix.
- Continuation driver and installation identity cohort: 62 checks pass.
- Regenerated capability inventory and contract checks pass.
- Final response precedence cohort: 109 checks pass. The stale-comment
case fails on the former implementation.
- All required CI checks, build, and both source reviews pass on
b9c8677635.
- Live provider journeys and deployed acceptance remain pending. No live
qualification is claimed.

## Risks

A pending repository now offers a next admission when unfinished
standard work yields. Incorrect eligibility could create unwanted work.
The lookup requires the current workspace and task. Explicit completion
does not use it. Existing Stop, admission, approval, and transport
authorization remain authoritative. No schema or public task-status
change is included. The shared continuation change requires a new
Copilot source identity. This PR preserves v52 history and does not
claim fresh Copilot live qualification.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The runtime does not expose its exact deployment ID or context-window
size. Those values are not inferred.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 07:17:50 -05:00
DottaandPaperclip c2daf906d9 fix: preserve task continuation and question delivery (#17045)
## Thinking Path

> - Paperclip manages agent work and delivers its results.
> - One task can need several runs and a saved workspace.
> - A run can succeed while the task waits for its next run.
> - Chat fallback treated that success as completion and dropped useful
status.
> - Monitor runs also lost their conversation origin, and text-question
cards lost their prompt.
> - This change preserves the task outcome and exact delivery authority
across those boundaries.

## Linked Issues or Issue Description

**What happened?**

A Slack task requested a repository, yielded to a scheduled
continuation, and showed only a generic completion message. The next run
could not publish through the original conversation. A later text
question showed its title and Respond button without the question
context. Workspace recovery could commit the saved monitor result
without restoring its explanation.

**Expected behavior**

The original conversation shows the accepted continuation explanation,
receives the resumed result, and displays the full question when input
is needed. A successful provider run does not claim that unfinished work
is complete. Recovery delivers saved output once without another
provider result.

**Steps to reproduce**

Start an authorized chat task, prepare a repository for a later
admission, save a monitor wait, and let that monitor resume the task.
Repeat with workspace finalization interrupted and with a free-text
question. Inspect the originating conversation and the persisted output.

Related merged work: #16806 and #16975. The open monitor and recovery
changes found during duplicate search address different status and
liveness policies.

## What Changed

- Keep committed monitor summaries visible through the shared board and
chat response resolver. Preserve interaction ownership and comment
reuse.
- Recover a saved monitor explanation after finalization, with no
repeated provider result.
- Prove monitor publication origin from the admitted wake and matching
scheduling receipt. Match company, task, agent, requester, and schedule;
reject ambiguous sources.
- Include monitor continuations in the existing publication scan and use
the canonical task binding. Keep current transport authorization.
- Suppress completion fallback for yielded results. Explain missing
final output without claiming completion.
- Include question summaries, prompts, and help text in modal-backed
cards. Keep answered text private.
- Update execution semantics, capability maps, and the bounded
reliability plan.

## Verification

- Original source fails four composed PostgreSQL regressions: monitor
origin, yielded completion, question body, and saved-summary recovery.
The shared response resolver also fails before the repair.
- Repaired focused cohort: 148 tests pass across publication, question
privacy, scheduler, and native monitor suites. These use real disposable
PostgreSQL; they do not qualify live Slack.
- Full workspace typecheck and production build pass locally. Map checks
and whitespace checks pass.
- All CI gates pass at 1701c48f17.
Greptile reports 5/5. Both review findings are resolved. The local
full-suite run remains separate.
- The added 21-admission lineage and cycle regressions pass with the
complete 11-test database suite.
- The larger live Codex/Claude qualification and deployed staging
acceptance remain in progress. This PR does not claim that milestone is
complete.

## Risks

- Incorrect lineage could disclose output to another conversation. The
resolver requires one exact receipt-backed source and the worker still
checks current delivery authority.
- Status recovery could duplicate output or compete with a question. The
source-run presentation marker, saved-comment reuse, interaction
arbitration, and replay tests protect those boundaries.
- No database migration or public task-status change is required.

## Model Used

OpenAI Codex, GPT-6 family, with repository editing, terminal tests, and
browser inspection. The runtime does not expose an exact deployment
model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 20:43:17 -05:00
DottaandPaperclip 505e1a5582 feat(cases): add persistent workflows with same-task agent handoffs (#16959)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - An agent workflow needs persistent facts and a safe way to transfer
work.
> - Creating a task for each phase splits the objective, discussion,
workspace, and review policy.
> - A case can retain the facts while one primary task moves between
agents.
> - Transfers must survive process failure and preserve private access,
budgets, and completion gates.
> - This pull request adds durable case handoffs, native case tools, and
read-only case pages.
> - Prompts describe the workflow. Existing task controls decide when
work can run and finish.

## Linked Issues or Issue Description

**Problem or motivation**

Prompt-driven agent teams need a persistent record that can move through
triage, implementation, verification, and correction. Ordinary
reassignment does not provide a durable instruction, safe release
barrier, or recoverable delivery receipt for an agent handing off its
own running task.

**Proposed solution**

Extend standalone cases with one primary task, a descriptive phase,
retained access sources, and a handoff ledger. Apply assignments after
source execution and workspace settlement. Deliver each transfer through
the existing durable wake queue. Provide `create_case`, `read_case`,
`update_case`, and `handoff_case` in the native runner. Creation
requires kickoff and explicit workflow text. An explicit handoff to
`done` closes the case when normal task completion is accepted.

**Alternatives considered**

Do not add a workflow engine, filesystem queue, configured state graph,
or one task per phase. The workflow document combines transition prose
with an optional Mermaid `stateDiagram-v2` map. The server does not
execute the diagram. The Pipelines experiment removed on master stays
removed.

**Roadmap alignment**

This extends Agent Reviews and Approvals, Enforced Outcomes, Bounded Run
Recovery, and Activity Log attribution. Existing approval, review,
budget, privacy, workspace, and responsible-user rules remain
authoritative.

Related work: [ordinary handoff wake
recovery](https://github.com/paperclipai/paperclip/pull/15019). This
feature adds a separate persisted case-handoff receipt and release
barrier.

## What Changed

- Add case routing fields, retained access sources, durable handoff
history, delivery leases, and replay-safe migrations.
- Keep the same primary task across agent phases. Preserve each
returning agent's own session and the task's workspace.
- Fence outgoing mutations, wait for provider and workspace release, and
recover delivery without duplicate task comments or runs. Recheck
current assignment permissions as the original caller before applying a
delayed transfer. Scope a run Stop to that run’s own handoff intents, so
repeated old Stop requests preserve newer work. Cancelling terminal
intent without a source run also requires fenced current task ownership,
including native retries and preparation with an old checkout. For Cloud
callers, revalidate current stack membership with an uncached Cloud RPC
and derive owner elevation from the current managed setting.
- Keep case reads, documents, assets, Audit, and live events inside
retained task/project permissions.
- Add native case creation/read/update/handoff tools. Store required
workflow text and initial documents on the first task. Page oversized
case facts and referenced document bodies through the dedicated tool
with bounded UTF-8 JSON, current routing versions, snapshot conflicts,
and current source authorization on every page.
- Add read-only case lists and detail pages, task-style list controls,
compact dotted field tables, linked handoff tasks, and case filters in
Audit.
- Add native/legacy integration tests, migration/restart tests, browser
coverage, and bounded live receipt fixtures. Extend the trusted sandbox
condition to cover the new subscription fixture profile.
- Hide obsolete paused-agent warnings on completed or cancelled task
threads.
- Synchronize the company-switch settings test with React effects after
a CI failure. Preserve its security assertions and production remount
behavior.

## Verification

The branch directly includes master `9ef434fe79`. The required `ci /
verify` and `ci / e2e` gates pass on head `765fccd67c`. [Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/39464348614)
passes typecheck, build, tests, and all eight browser shards. Security
checks pass. [Greptile gives this exact head
5/5](https://github.com/paperclipai/paperclip/pull/16959#issuecomment-6291156549),
with no unresolved review threads. The optional Paperclip Review check
remains queued because its assigned agent is paused. It is not a
required branch check. No check result was overridden.

Local workspace typecheck and build pass after the master merge, along
with database migration checks, token gates, map checks, Storybook
catalog checks, and the Copilot profile identity check. Standard builds
refresh Runner vendor output. Focused checks pass 2,008
API/paging/context tests and 7 strict case protocol tests. The final
commit changes one test fixture only; its ten tests, UI typecheck, and
token gates pass. No production source changed in that commit.

On current head `765fccd67c`, all **194 critical workflow tests**
executed and passed with no skipped tests in those files:

- 38 Stop/ownership tests, including board completion intent, native
retries, stale Stop, and preparation with an old checkout.
- 2 actual legacy process/HTTP handoff tests, including scratch failure,
a closed-stdio child that continues writing, and recovery through a
fresh worker after the process group stops.
- 5 migration tests, including SQL column metadata, preserved null
authority context, and all six prior-numbering receipts.
- 3 process-settlement and 5 scratch tests.
- 88 handoff-service, 18 heartbeat handoff, and 21 native case-tool
integration tests.
- 7 bounded case/document-read projection tests and 7 strict
protocol-contract tests.

The current CI checkout was `fed65d49`, merging `765fccd67c` into master
`9ef434fe79`. Its new document regressions reconstruct exact
Unicode/escaped bodies and revisions through real native authority and
authenticated HTTP with generic API tools disabled. They cover
case-owned, primary-task, and linked-task documents; routing on every
page; stale document snapshots; access loss and unlinking;
unrelated/foreign/private sources; system documents; and quarantined
content. Active-task document tools retain their existing behavior.
These are current-head test results from the first CI attempt; the
browser-only retry did not rerun these suites.

The full `64fdd3403f` CI failed one separate settings test: after a
company switch, it observed an unsettled API-key draft. The production
component and test were unchanged from prior green source. Production
switches remount both keyed settings layers; the test directly reused an
unkeyed view and used a timer tick instead of waiting for React effects.
The final commit replaces that fixture synchronization with React `act`.
The exact secret-clearing assertions and all timeouts remain unchanged.
The original assertion failure remains recorded; no unchanged retry was
used. The narrow unchanged baseline and fixed fixture each pass 10
tests.

The upstream result-paging merge preserves case capture namespaces and
current asset privacy. Its API integration suite executed 39 of 41 tests
successfully, with two skipped; the optional assigned-result PRP
integration test was skipped. Neither skipped suite is counted in the
194 critical tests or as live provider proof.

All 361 upstream migration entries and SQL bodies remain unchanged. Six
case migrations follow master as 0363–0368 with identical SQL hashes.
Official Drizzle generators produced the combined metadata and five
retained snapshots. Generation reports no schema changes, and the
snapshot drift test passes. The earlier `ce361ae5d0` CI passed on
attempt 1, including its then-current 179 critical tests. Fresh review
later identified unusable document references with generic API tools
disabled; the bounded document reads and ten additional authenticated
HTTP tests address that finding.

The full local `pnpm test:run` attempt was cancelled after this Mac
exhausted PostgreSQL semaphore capacity; it is **not** counted as a
pass. Later database attempts also stopped before assertions or skipped,
including one sandbox socket denial. Two original stale-Stop regressions
did reproduce the bug against the old source. A REST serialization
assertion passed after two preserved database-startup failures. No test
timeout was increased. All 194 assertions listed above executed and
passed in current-head CI. The complete 385-file, 3,963-test UI shard
also passed after the fixture repair.

A configured-provider receipt workflow passed 13 backend checks across
triage → implementation → verification → correction → done, using one
task and four work runs. The saved receipt is revision 2 with total 6
USD; case and task both closed. The original canonical attempt remains
failed because its browser assertion confused the terminal destination
with completed-task tooltips. The corrected full browser verifier passed
on the same retained database and original runs, with no additional
provider calls. The last clean-build browser readback is on
`6a86c54a53`; later changes affect recovery, authorization, bounded
reads, and settlement. Accounted work cost is a $1.010574 rate-card
estimate; provider billing and setup-probe usage are unavailable.

Copilot profile 49 passed the unchanged seven-fixture private protocol
pack (46/46 checks, no retries) and the product file-edit canary after
master's completion-replay change. All 60 bound implementation hashes
remain unchanged on the current source. Historical profile-47/48
identities and results remain intact. See the [profile-49 compatibility
record](https://github.com/paperclipai/paperclip/blob/codex/persistent-case-handoffs/doc/plans/2026-10-20-copilot-completion-recovery-compatibility-v49.json).
This is bounded compatibility evidence, not a fresh full matrix,
installed-package product test, or Daytona rollout.

Preserved CI failures and retries:

- On `20662930ac`, Slack managed setup failed a five-second success
assertion after reload. Its test/application path was unchanged from a
prior green head. One unchanged-source retry passed all 25 tests in that
shard; no source or timeout changed.
- On `9402b4f17a`, two CI workers received shutdown signals at 15:24:19
UTC. Neither log had an assertion failure before interruption. One
unchanged-source retry of each passed all Runner static checks and all
32 tests in browser shard 6; both aggregates passed. The next head,
`7b4374edaf`, passed all CI without retries.
- On `295f289b5a`, serialized shard 9 timed out during its first
device-login test, then the next test found an active login lease.
Direct login/test paths were unchanged from a prior green head. Cold
import plus a timed-out async body using the next test's mutable
actor/store is the supported fixture-race explanation; no request trace
proves that sequence. The original failure remains recorded. One
unchanged-source retry passed all 294 tests in 19 files, including all
41 device-login tests. No login source or timeout changed. The
subsequent `ce361ae5d0` CI passed without retries.
- On `64fdd3403f`, one company-switch settings assertion observed an
unsettled draft. The final test-only commit uses React `act` without
changing assertions, timeouts, or production code. On `765fccd67c`, all
ten settings tests and the complete 385-file, 3,963-test UI shard
passed.
- On `765fccd67c`, browser shard 3 completed the split-handoff
assertions, then cleanup received HTTP 500 from run cancellation after a
one-second PostgreSQL row-lock timeout. The extra chat run was reporting
completed work while teardown began. Cleanup stopped before restoring a
global experiment flag, causing a later assertion to fail. Relevant
production and test paths are unchanged from an earlier green commit.
Stop's lock order is consistent, and no nested self-lock was found; the
exact PostgreSQL blocker was not recorded. One unchanged-shard retry
passed. The original failure, trace, and downstream flag failure remain
recorded. No Stop implementation, cleanup assertion, or timeout changed.
The retry supports intermittent cleanup contention; it does not identify
the original blocking transaction.


## Risks

- Local process transfers require host-owned PID/group absence. Missing
or uncertain termination evidence keeps delivery pending, independently
of scratch setup.
- Handoff admission and task cancellation share execution ownership. New
regression tests cover temporary holds, late source writes, and
uncertain provider termination.
- Retained private sources use an intersection of existing permissions.
This intentionally fails closed when a source is missing or an existing
task cannot preserve that boundary.
- Migrations add tables and indexes. They preserve master history and
can replay over prior feature installs. Migration 0368 adds nullable
internal authorization provenance with no default or authority backfill.
Historical transfers still awaiting release without this context fail
closed; an operator must cancel them and request a new handoff.
- Delayed Cloud handoffs depend on the current Cloud membership RPC. An
unavailable, ambiguous, expired, or mismatched response holds the
handoff for retry; it does not reuse cached authority or grant access.
- Bounded reads reload and reauthorize the selected projection per page.
Underlying work grows with the data size and number of pages. Changed
data or routing requires restarting from offset zero without the old
snapshot.
- Phase names and workflow diagrams remain advisory. They do not satisfy
review or approval gates.
- This is a large, cross-cutting PR. The maintainer requested one PR for
the whole feature; the commits separate data, runtime, UI, and
qualification work.

## Model Used

OpenAI Codex, GPT-6 family, with tool use, code execution, and delegated
code review. The exact deployment model ID and context-window size were
not exposed in the session. Live fixture agents used the explicitly
configured provider model recorded in their test evidence.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 12:46:14 -05:00
DottaandPaperclip cccde7122b Trace execution preparation and durable response persistence (#16419)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server prepares an execution environment before it starts an
agent.
> - Operators need to distinguish preparation, provider wait, and
response persistence.
> - Existing preparation phases start separate traces, and the native
session trace ends before the final response is saved.
> - This pull request adds an execution-attempt trace and detailed spans
at the existing work boundaries.
> - The benefit is a measurable startup and settlement path without
changing execution policy.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Operator-configured OpenTelemetry tracing for agent preparation,
startup, and durable response storage.

**Subsystem affected**

Server orchestration, native Runner startup, and the Honeycomb run link.

**Current behavior**

Preparation phases start independent roots. Workspace lock waits,
authorization groups, runtime asset work, provider-pack fallback, and
the gap after provider completion are hard to attribute.

**Proposed behavior**

Use a server-owned async context for each execution attempt. Parent
nested work to that attempt. Link the separately sampled native session
because its ingress and queue history can begin earlier. Record
transaction entry, lock waits, grouped authorization, asset preparation,
artifact fallback, copyback, and response commit.

**Reason and benefit**

Operators can identify expensive work from recorded boundaries instead
of inferring the cause from elapsed time. Nested and concurrent
durations remain visible and must not be added together.

**Breaking changes**

No API or execution-policy change. New Observability span names and
closed count attributes are documented. The existing local
run-performance schema also records the additional provider-pack and
harness measurements.

Refs: #16178, #16205, #16330.

## What Changed

- Add `task.execution_attempt` with same-run async ownership,
independent root sampling, and native-session links.
- Measure workspace transaction entry and settlement, advisory and row
waits, source scans, reader groups, notifications, and writes.
- Measure runtime selection and each concurrent instructions, skills,
and MCP branch. Separate manifest, cache, materialization, and
verification work.
- Measure provider-pack discovery, verified reuse or upload, final
verification, and remote harness state checks.
- Measure workspace settlement, accounting, liveness, response
authorization, comment commit, and publication enqueue.
- Preserve the silent endpoint-off path, including when another library
has registered a tracer. Bound repeated detail spans to 16 per name per
attempt.
- Extend the Honeycomb run link and document the actual coverage and
remaining gaps.

## Verification

- After resolving the conflict with retained-reader batching on
`b98cdaea531c8447637cd4dcc19c61b3f5f41844`, all 61 targeted query,
authorization, workspace-preparation and trace-boundary tests passed.
The server typecheck passed. Reader tests still bound query count, avoid
loading retained run payloads and enforce fresh project grants on every
publication attempt.
- Before that base integration, the tracing/exporter group passed 855
tests with 2 optional-SDK skips across 10 files and no inherited OTLP
endpoint. It covers real PostgreSQL waits/rollback, durable response
commit, assembled heartbeat preparation, concurrent parentage, sampling
and tracer failure isolation. Native workspace/history tests passed 23
cases. These results remain supporting evidence for the unchanged
tracing paths.
- Full workspace typecheck and build passed during implementation. Full
sharded CI, release validation, E2E and Apex 5/5 passed on prior head
`8cebad80f3856dbcff6f3a6d5d34880ff77a20f2`. Fresh CI, security and Apex
review are required on the current head before merge.
- `pnpm check:token-gates` passed. No styles changed.
- Earlier CI runs exposed exact-source-expression assertions changed by
wrappers, missing run hashes on transaction-boundary spans, and sampling
fixtures without an enabled endpoint. The fixes preserve the calls, add
the declared hash and span cap, and enable tracing explicitly in the
sampling fixtures. Focused repairs passed; the full prior-head CI passed
after those repairs. Earlier failed run:
https://github.com/paperclipai/paperclip/actions/runs/38729131366.
- The full local `pnpm test:run` was stopped after the base advanced
during that process. Its mixed-version results are not final-head
evidence. A full local-suite pass is not claimed.
- The old-head Superagent scan stalled at 35%. The current head has a
fresh scan; no security pass is inferred from the old pending result.
- No deployed Honeycomb acceptance is claimed by this code change.

## Risks

- Enabled tracing creates more spans. Repeated detail spans are bounded,
and inventories use group spans.
- The attempt and native session remain linked traces. Historical queue
time can precede the attempt, and nested intervals overlap.
- A committed comment proves durable server storage. Publication enqueue
and realtime dispatch do not prove external delivery or browser paint.
- New custom attributes use only the existing run hash and validated
file/byte counts. No SQL, paths, permissions, content, or exception text
is added.

## Model Used

OpenAI GPT-6 through Codex. The serving model ID and context window size
are not exposed in this session. The agent used reasoning, tools, code
editing, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-15 08:13:53 -05:00
76164b53cb fix(heartbeat): retry a legacy run that lost its lease before dispatch (#16358)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat runner executes legacy agent runs. A controller lease
watchdog keeps durable ownership of each legacy run, and it aborts the
run when it loses the lease.
> - #16084 records a lease loss as a `controller_lease` cancellation
when the adapter returns or throws. It does not cover a lease loss
during setup, before the adapter starts.
> - In setup, the abort reason goes to the setup catch. The catch
records an opaque `setup_failed` and sends the raw
`LegacyControllerLeaseLostError` to Sentry with no cause label.
> - Issue continuation recovery treats `setup_failed` as non-retryable.
The task moves to `blocked`, but no provider work started.
> - This pull request records a lease loss before dispatch as
`controller_lease_lost`, with the same `controller_lease` evidence.
Recovery retries it as transient infrastructure, and so does the
accepted-interaction (plan approval) continuation retry.
> - The benefit is that a host lease loss during setup no longer strands
the task, and the next Sentry event shows which renewal failure caused
the loss.

## Linked Issues or Issue Description

Refs #16084
Refs #16355
Refs #16359

**What happened?**
A staging server reported `LegacyControllerLeaseLostError: Legacy
controller lease lost` from the heartbeat. The run failed during setup
with `setup_failed`. Continuation recovery then said "non-retryable
failure on this issue's continuation run (`setup_failed`)" and moved the
task to `blocked`. A scanner had to move the task back to `in_progress`
by hand.

**Expected behavior**
A lease loss before adapter dispatch is a host infrastructure stop. The
run records the `controller_lease` cause, like the adapter-phase path.
The run keeps `providerWorkStarted: false`. Recovery retries the
continuation with its bounded transient budget, and it does not block
the task.

**Steps to reproduce**
1. Start a legacy run that has a controller lease.
2. Before the adapter dispatch, make the lease renewal fail: the update
matches no row, the query errors, or the lease deadline passes.
3. `controllerLease.assertOwned()` throws the abort reason in workspace
preparation or before dispatch.
4. Before this change: the run is `failed`/`setup_failed`, `resultJson`
has no `cancellation`, and continuation recovery blocks the task. After
this change: the run is `failed`/`controller_lease_lost` with
`cancellation.source: "controller_lease"` and a closed cause, and
continuation recovery classifies it as `transient_infra`.

**Additional context**
#16355 handles a different path: a restart orphan that the next boot
reaps as `process_lost`. This change handles a live controller that
loses its own lease during setup. This change does not find the root
cause of the lease loss. The closed cause label in the next Sentry event
gives that fact.

## What Changed

- `server/src/services/heartbeat/run-completion.ts`: `failRunSetup`
receives the execution abort signal. When the abort reason is a
`LegacyControllerLeaseLostError`, it writes error code
`controller_lease_lost`, adds the `controller_lease` cancellation
evidence, and keeps the existing bootstrap recovery facts. The agent
goes back to idle, not `error`, because the stop is a host fault. This
applies only before provider dispatch: `failRunSetup` also receives
`providerDispatchStarted`, and a setup-path error after a legacy adapter
or native dispatch began keeps `setup_failed`, so started work is never
retried.
- `server/src/services/legacy-controller-lease.ts`: add the
`CONTROLLER_LEASE_LOST_FAILURE_CODE` constant.
- Merged with #16359 (lease-loss timings): the setup failure keeps its
`resultJson.controllerLeaseLoss` timings and adds this PR's
`controller_lease` cancellation from the abort signal. #16359's setup
test now aborts the signal first, as `assertOwned()` does, and expects
`controller_lease_lost`.
- `server/src/services/heartbeat.ts`: pass the execution abort signal
and `providerDispatchStarted` (`legacyAdapterEntered ||
nativeDispatchStarted`) to `failRunSetup`.
- `server/src/services/recovery/service.ts`: add `controller_lease_lost`
to the transient-infrastructure continuation codes (3 bounded attempts
with backoff).
- `server/src/services/heartbeat/retries.ts`: add
`controller_lease_lost` to the accepted-interaction continuation retry
allowlist, next to `process_lost`. Without it, a plan-approval
continuation that lost its lease before dispatch was escalated and
blocked before generic recovery could see it.
- `server/src/services/heartbeat/run-lifecycle.ts`: add
`controller_lease_lost` to the pre-adapter setup codes, so the comment
policy does not queue a missing-comment retry.
- `doc/run-log-events.md`: document the new error code.
- Tests: setup lease loss when the abort reason is thrown, and when a
setup step throws a different error after the abort. A unit test shows a
lease loss after dispatch keeps `setup_failed`. A classifier test shows
`controller_lease_lost` is transient and `setup_failed` stays
non-retryable. Two assembled heartbeat tests
(`heartbeat-process-recovery.test.ts`) lose the real lease during
workspace preparation: one for a plain issue continuation (one
`transient_failure` retry) and one for an accepted plan-approval
interaction continuation (one `interaction_continuation_infra_retry`
retry). Both check the saved run, no adapter dispatch, and the issue
still `in_progress`.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts`: 25 passed (embedded
Postgres), including the new post-dispatch test.
- `cd server && npx vitest run src/services/run-cancellation.test.ts
src/services/recovery/provider-failure-classification.test.ts`: passed
on the first head (29 with run-completion; 25 classifier).
- `cd server && npx vitest run
src/services/legacy-controller-lease.test.ts`: 13 passed (needs `pnpm
--filter @paperclipai/plugin-sdk... build` first).
- After merging master (head `2148b27`): `run-completion`,
`legacy-controller-lease`, `run-cancellation`,
`provider-failure-classification` and `run-failure-report` suites, 310
passed; `heartbeat-process-recovery.test.ts -t "lease|dispatch"`, 31
passed.
- Head `c52825c`: `retries`, `run-completion`,
`heartbeat-retry-scheduling`, `legacy-controller-lease` and
`provider-failure-classification` suites, 134 passed;
`heartbeat-process-recovery.test.ts -t "lease|dispatch|interaction"`, 41
passed.
- Negative controls: with the lease-loss branch in `run-completion.ts`
disabled, the plain-continuation assembled test fails (`setup_failed`
instead of `controller_lease_lost`). With the `retries.ts` allowlist
change reverted, the accepted-interaction assembled test fails because
no retry is scheduled.
- Not run locally: full `tsc` and `pnpm build`. The sandbox has a 4 GB
memory limit and the OOM killer stops `tsc`. CI is the typecheck and
build gate.

## Risks

- Low risk. The change applies only when the execution abort reason is a
`LegacyControllerLeaseLostError` in the setup catch before dispatch.
Other setup failures keep their current codes.
- The retry uses the existing continuation recovery budgets and
admission checks (agent state, budget, pause). The
interaction-continuation retry keeps its existing 2-attempt budget. The
retry is a new run. The revoked controller cannot dispatch, because its
renewal compare-and-set fails.
- A run that the reaper settles first keeps the reaper outcome.
`setRunStatusIfRunning` makes the first terminal write win.
- Sentry still gets this unexpected stop, with the closed
`controllerLeaseLossCause` label. If the lease loss is frequent, the
event volume does not change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass (targeted suites pass;
typecheck/build not possible in this sandbox, see Verification)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 21:31:42 -07:00
ccc77c5b5e fix(server): record legacy controller lease-loss timings (#16359)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat runner executes legacy agent runs, and a controller
lease watchdog keeps durable ownership of each run
> - The watchdog renews the lease every 10 seconds and stops the run
when a renewal does not finish within the 60-second lease
> - A staging run failed in setup with `LegacyControllerLeaseLostError`
and the `renewal_deadline` cause, after about eight minutes of
preparation and with no server restart
> - The setup failure path did not record the cause, and no evidence
showed why the renewal did not finish: a stalled database query or a
blocked event loop
> - This pull request records bounded watchdog timings with each lease
loss, on the setup failure path and on the cancellation path
> - The benefit is that the next event shows which of the two causes
stops renewal, which is the missing fact to fix the root cause

## Linked Issues or Issue Description

Refs #16084 (added the closed lease-loss cause for cancelled runs)
Refs #16355 (retries a run that a restart stopped before dispatch; a
different path)

**What happened?**
A legacy `paperclip_runner` run on staging failed with `setup_failed`
and the message "Legacy controller lease lost". The stack shows the
watchdog deadline timer (`renewal_deadline`). The run was in
`executionStage: "preparing"` for about 467 seconds. The server started
about 11 minutes before the run, so this was not a restart. The run
record and the Sentry event do not show the lease-loss cause, because
the setup failure path does not read it. Nothing shows if a renewal
query was in flight at the deadline, or if the deadline callback ran
late.

**Expected behavior**
When the watchdog loses the lease, the run records the closed cause and
the watchdog timings. This applies when the loss fails setup and when it
stops the adapter. The failure report contains only the closed cause and
bounded numbers.

**Steps to reproduce**
1. Start a legacy run with a controller lease.
2. During setup, make a lease renewal query hang for more than 60
seconds, or block the event loop past the deadline.
3. Let the watchdog abort the run.
4. Read `resultJson` on the failed run. Before this change, it has no
lease-loss evidence.

## What Changed

- `legacy-controller-lease.ts`: The watchdog tracks the last successful
renewal, the renewals in flight, the scheduled deadline, and the renewal
count. `LegacyControllerLeaseLostError` now carries `leaseLossTiming`:
`sinceRenewalMs`, `renewalPendingMs` (or `null`), `deadlineLateMs` (only
for `renewal_deadline`), and `renewalCount`. Ownership behavior does not
change.
- `run-cancellation.ts`: Add the `LegacyControllerLeaseLossTiming` type
and `readControllerLeaseLoss()`. The reader accepts only a closed cause
and non-negative finite numbers, and clamps each value to 24 hours.
- `heartbeat/run-completion.ts`: Persist
`resultJson.controllerLeaseLoss` with the cause and timings. This
applies to a lease loss that stops the adapter and to a lease loss that
fails setup (the thrown abort reason).
- `run-failure-diagnostics.ts`: Report `controllerLeaseLossCause` and
the four bounded timing fields from `controllerLeaseLoss`. A setup
failure now gets the cause label too.
- `doc/run-log-events.md`: Document the new record and how to read the
timings.
- Tests: watchdog timings for a stalled query, a blocked event loop, and
a failing query; completion records the evidence for an adapter stop and
for a setup failure; the failure report clamps values, drops unknown
keys, and drops a record without a closed cause.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/legacy-controller-lease.test.ts
src/services/heartbeat/run-completion.test.ts
src/services/__tests__/run-failure-report.test.ts
src/services/run-cancellation.test.ts`
- I could not run these tests locally. The sandbox has a 4 GB memory
limit, and `pnpm install` was stopped by the out-of-memory killer during
lockfile resolution. CI is the test run for this change.
- After deployment, a new lease loss shows
`controllerLeaseRenewalPendingMs` and `controllerLeaseDeadlineLateMs` in
the Sentry execution context. A pending time near 50 seconds or more
points at the database. A large deadline lateness points at the event
loop.

## Risks

- Low risk. This change adds diagnostics. It does not change when the
watchdog stops a run, and it does not prevent the lease loss.
- The new `resultJson.controllerLeaseLoss` field is server-written.
Readers ignore unknown values. Old rows do not have it.
- The failure report gets five closed or numeric fields. It gets no
reason text, IDs, or user content.
- The event-loop test blocks the loop for about 80 ms to make the
deadline late. It uses real timers, so a very slow CI host makes the
lateness larger, not smaller.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, Sentry and GitHub connectors). The exact
context-window size is not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass (not possible in this
sandbox; see Verification)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Devin Foley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 19:52:23 -07:00
a64969b288 fix(server): attribute legacy runs stopped by controller lease loss (#16084)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat runner executes legacy agent runs. A controller lease
watchdog keeps durable ownership of each legacy run.
> - When the watchdog loses the lease, it aborts the adapter. The abort
reason was a plain `Error`, and nothing recorded why the run stopped.
> - Completion then saved the cancellation as source `unknown`. Sentry
got an unexpected cancel that it could not tell apart from other
unrecorded stops.
> - We saw this on a sandboxed Grok run. The system cancelled it after
about 22 minutes with `cancellationSource: unknown`. Workspace copy-back
then failed. Every `cancelRun` path records a stop request, so an
unrecorded stop this late points to the lease watchdog.
> - This pull request records a lease loss as its own cancellation
source, with a closed cause label.
> - The benefit is that the next event shows whether the lease loss
caused the stop, and which renewal failure caused it. That is the
missing fact to fix the root cause.

## Linked Issues or Issue Description

Refs #15790
Refs #15920

**What happened?**
A legacy sandbox run was stopped by the system while it was working. The
run record and the Sentry diagnostics show `cancellationSource:
unknown`, with no initiator. The controller lease watchdog aborts the
adapter with a plain `Error("Legacy controller lease lost")`. Completion
does not read the abort reason, so it records the stop as `unknown`.

**Expected behavior**
When the host stops its own adapter because it lost the controller
lease, the run records that cause. Diagnostics name the cause with a
closed label. A stop that a user or the control plane requested earlier
keeps its recorded source.

**Steps to reproduce**
1. Start a legacy run that has a controller lease.
2. Make lease renewal fail: the renewal query errors, the update matches
no row, or the lease deadline passes.
3. Let the adapter return its cancelled result.
4. Read `resultJson.cancellation` on the run. Before this change,
`source` is `unknown`.

**Agent adapter(s) involved**
Not adapter-specific (core bug). We saw it with `grok_local` in a
sandbox.

**Additional context**
This change adds diagnostics. It does not prevent the lease loss. #15920
retries a transient copy-back failure. It does not help here, because
the stopped sandbox rejects every retry.

## What Changed

- `legacy-controller-lease.ts`: The watchdog aborts with a new
`LegacyControllerLeaseLostError`. The error carries a closed cause:
`renewal_rejected`, `renewal_deadline`, or `renewal_error`.
- `run-cancellation.ts`: Add the `controller_lease` cancellation source.
Add `controllerLeaseLossCancellation()` and
`readControllerLeaseLossCause()`. The reader accepts only the
server-written reason text.
- `heartbeat/run-completion.ts`: When the outcome is `cancelled` and the
abort reason is a lease loss, record a `controller_lease` cancellation
(`expected: false`). This covers both an adapter that returns a
cancelled result (`completeRun`) and one that throws after the abort,
for example through `signal.throwIfAborted()` (`failRun`). A stop
request recorded earlier still takes precedence.
- `run-failure-diagnostics.ts`: Add `controllerLeaseLossCause` to the
execution diagnostics. Only the closed label goes to Sentry, never the
reason text.
- `heartbeat/run-state.ts`: preserve `controller_lease` in the bounded
SQL projection when saved results exceed 64 KB. A database regression
verifies the default run-detail read.
- `canContinueCancelledRun` stays unchanged. It does not accept
`controller_lease`, so a lease loss does not authorize automatic
continuation.
- `doc/run-log-events.md`: Document the new source and cause labels.
- Tests: unit tests for the cause on each watchdog path, completion
tests that record the source when the adapter returns and when it throws
the abort reason (both fail without the change), tests that a recorded
stop request takes precedence, and a test that the Sentry report
contains the closed label and not the reason text.

## Verification

- `cd server && npx vitest run src/services/run-cancellation.test.ts
src/services/legacy-controller-lease.test.ts
src/services/__tests__/run-failure-report.test.ts
src/services/heartbeat/run-completion.test.ts`: 238 passed, including
the embedded Postgres suites.
- I reverted only `run-completion.ts` and ran the new completion test.
It failed with `source: "unknown"`. With the change, it records
`controller_lease`.
- I removed only the `failRun` attribution and ran the new thrown-error
test. It failed without the `controller_lease` record, and passes with
it.
- Final full repository `pnpm -r typecheck` and `pnpm build` pass.

- Final local regression verification: all 247 tests pass across
run-state, cancellation, watchdog, failure reporting, and completion
suites. The large-result test fails before the SQL allowlist change. All
latest-head CI checks pass (54 successes and two intentional skips).
Fresh Apex is 5/5 on `708856c568d07d7b446a4a4af2edc6d9eec7d155` with no
unresolved findings.

## Risks

- Low risk. The change adds a new source value to server-owned JSON. Old
rows keep their values. The readers accept the new value, and no UI code
switches on this field.
- A lease-loss cancel stays `expected: false`, so it is still reported
as unexpected, as before. Now it has a more specific source.
- The Sentry payload gets one new closed label
(`controllerLeaseLossCause`). It does not include reason text or IDs.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, test runs).

OpenAI GPT-6 (`gpt-6`) through Codex added the large-result regression
and SQL projection fix, with shell, file-edit, and test tools. The exact
context-window size is not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-13 14:50:40 -07:00
DottaandPaperclip c27ee928b0 refactor(server): extract heartbeat run completion (#15732)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Heartbeat execution connects provider runs to task state,
accounting, and recovery.
> - Earlier refactors moved scheduling and lifecycle services into the
heartbeat folder.
> - The remaining executor still contains result persistence and two
large failure handlers.
> - These handlers must preserve the outcome chosen by Stop or native
recovery.
> - This pull request moves them into a typed run-completion module at
the same call and catch boundaries.
> - Reviewers can now inspect completion separately from provider
dispatch and resource cleanup.

## Linked Issues or Issue Description

Refs: #15717. Related behavior work: #13857 (agent status convergence).

**What existing behavior does this improve?**

Maintainability of heartbeat execution and terminal result handling.

**Subsystem affected**

server/ — orchestration services.

**Current behavior**

The main heartbeat service has 9,335 lines. Result persistence,
accounting, presentation, session updates, and execution/setup failure
handling are embedded in `executeRun`.

**Proposed behavior**

Move those steps into `server/src/services/heartbeat/run-completion.ts`.
Keep the existing decisions and ordering. The main service becomes 8,125
lines, a reduction of 1,210 lines.

**Reason and benefit**

This creates a substantial boundary for the next execution refactor. It
also gives terminal handling direct tests without starting a provider.

**Breaking changes**

None intended. Public heartbeat exports and method signatures remain
compatible.

## What Changed

- Add `createHeartbeatRunCompletion` with `completeRun`, `failRun`, and
`failRunSetup` handlers.
- Move terminal-result persistence, accounting, response presentation,
retry decisions, and session updates with their failure helpers.
- Keep Stop barriers and native ownership/recovery exceptions in the
executor. Keep the outer cleanup in its original location.
- Pass live output getters and an immediate trace-finalization callback
so cleanup sees state even if a later completion step throws.
- Add 19 boundary tests, update the existing publication-order source
guard to read the extracted module, and document the module in the
developer guide.

## Verification

- Passed `pnpm -r typecheck` and `pnpm build`.
- Structural AST comparison confirms all 74 top-level statements in the
three extracted blocks, the moved helper bodies, and all remaining
service statements are unchanged after normalizing the explicit boundary
parameters. All 140 public exports are preserved.
- Passed 304 existing tests across seven adjacent lifecycle, accounting,
session, Stop, and presentation suites. After rebase, all 19 new
boundary tests and all 12 publication tests pass together. Total focused
coverage: 335 passing tests.
- Full local `pnpm test:run` was stopped after more than 28 minutes
without completing. Additional route-suite diagnosis hit local database
startup timeouts. The complete CI run passed on the final head,
including all test shards, build, typecheck, browser tests, and canary
install checks.
- Rebased onto the workspace-repair fix in #15726. Preserve its
`defaultBranch` metadata in the moved helper and add a regression test.
All 31 completion and publication tests and the server typecheck pass on
the rebased head.
- Greptile reviewed current head
`65e3f8d8ac165dfec4508c467b1b2af8f2510f1a`: 5/5 with no actionable
findings.
- Final-head [CI
run](https://github.com/paperclipai/paperclip/actions/runs/37991844034)
passed. All 53 completed check runs succeeded; two optional Storybook
checks were skipped. The legacy security status also passed. No
unresolved review comments or merge conflicts remain.

## Risks

Terminal handling is sensitive to Stop races, late native results, and
failures during persistence. The tests exercise these paths with real
database status writes, accounting, and task sessions. Native recovery
exceptions and resource release stay at their existing boundaries. There
are no schema or API changes.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, code editing, and command
execution. The session does not expose the exact model variant or
context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 16:29:54 -05:00