Commit Graph
18 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 2a7fe6c585 fix(server): withhold a queued assignment run once its task moves to backlog (#21095)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Assigning a ready task to an agent queues an assignment run.
Paperclip does not queue one for a backlog task
> - An agent created a ready task, which queued an assignment run, and
then moved the task to backlog. The agent was busy, so the run stayed
queued
> - When the run started, the queued-run gate did not check for backlog,
and the run checked out and did the backlog task
> - This pull request makes the queued-run gate withhold a never-started
assignment run whose task is now in backlog
> - The benefit is that moving a task to backlog keeps it parked, as
when it is created in backlog

## Linked Issues or Issue Description

**What happened?**
An agent created a task without a status, so it was created ready with a
queued assignment run. The agent then moved the task to backlog. The
queued assignment run started two minutes later and completed the
backlog task.

**Expected behavior**
A queued assignment run that has not started does not execute a task
that has since moved to backlog. The run is cancelled with a clear
reason.

**Steps to reproduce**
Assign a ready task to a busy agent so that its assignment run waits in
the queue. Move the task to backlog. Let the agent become free. Before
this change, the assignment run started and checked out the task.

## What Changed

- `run-dispatch/domain/policy.ts` (`decideQueuedRunStaleness`): a queued
run whose wake reason is `issue_assigned` is stale when its task is in
`backlog`. The new error code is `issue_in_backlog`, with the reason
"Cancelled because the task moved to backlog before its assignment run
could start".
- `pre-dispatch-review-wait.ts`: lists `issue_in_backlog` as a
never-started dispatch cancellation, like the other queued-run gate
codes.
- `run-dispatch/adapters/postgres.test.ts`: a new test moves the task to
backlog after the run is queued and checks that `cancelStaleQueuedRun`
cancels it before it starts. A control test checks that a comment wake
on a backlog task still starts.

## Verification

- The new test failed before the change (`not_stale`) and passes after
it.
- `pnpm vitest run src/modules/run-dispatch/
src/services/pre-dispatch-review-wait`: 185 passed.
- Product evidence: a paid chat-eval run of
`questions-and-approvals.accept-one-proposal` on a branch that combines
this change with three related fixes. The result is in a PR comment.

## Risks

- Low risk. The rule applies only to assignment wakes. Comment,
interaction and recovery wakes on a backlog task keep their current
behavior.
- To start the task, a person or agent moves it out of backlog. That
wakes the assignee with `issue_status_changed`, as today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:19:27 -06:00
DottaandClaude Fable 5.2 39442400b7 fix(native): let a saved governed wait settle on its own same-run resume (#20838)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs can save a governed wait (a question or confirmation for
a person) and park the provider turn
> - If shutdown of the retained execution cannot be confirmed in time,
the run keeps its checkpoint, records
`native_governed_wait_settlement_pending`, and schedules a same-run
resume that settles the saved wait (#20336, #20494, #20519)
> - That failure writes an active recovery action for the run. Its cause
is in `EXECUTION_RECONCILIATION_CAUSES`, and its wake policy resumes
that same run
> - When the run is a resolved-interaction continuation, its dispatch
gate checks execution blockers. The gate read the run's own resume
action as a hold and cancelled the resume with
`execution_reconciliation_required`
> - The task then stayed blocked with no Retry and no continue control,
and the saved question could not wake the agent
> - This pull request makes the gate skip only the run's own
scheduled-resume action
> - The benefit is that a saved governed wait settles by itself, with no
person involved

## Linked Issues or Issue Description

Refs #20494, #20519, #20809

**What happened?**
A staging Codex run on a remote sandbox was a continuation after an
answered question. The agent saved a second governed wait (a
confirmation). Shutdown confirmation of the retained execution was still
pending, so the run recorded `native_governed_wait_settlement_pending`
and scheduled its exact-checkpoint resume. The resume reattached to the
remote runner and started. 32 s later it ended `cancelled` with
`execution_reconciliation_required`. No person cancelled it. The
lifecycle event of the cancel carried the run's own recovery action id.
The task showed `blocked` with `canRetry: false` and `canContinue:
false`, and no later run started.

**Expected behavior**
The same-run resume settles the saved governed wait. The task waits for
the person's answer. No run is cancelled and no recovery action needs a
person.

**Steps to reproduce**
1. A native run starts as a resolved-interaction continuation (context
`mutation: interaction`, `interactionStatus: answered`).
2. The run saves a governed wait, and shutdown confirmation does not
arrive in time (`native_governed_wait_settlement_pending`).
3. The coordinator schedules the same-run resume. At dispatch,
`decideCurrentRunStaleness` calls `getExecutionBlocker`, which returns
the run's own active action, and `cancelStaleRunInTx` cancels the run.

## What Changed

- `server/src/services/execution-blocker.ts`: `getExecutionBlocker`
takes `resumingNativeRunId`. For that run it skips only the active
action whose wake policy is `resume_native_run` for the same run and
whose evidence names that run. Workspace-restore holds and all other
actions still block.
- `server/src/modules/run-dispatch/adapters/postgres.ts`: the dispatch
staleness decision passes the native run id.
- `server/src/modules/run-dispatch/adapters/postgres.test.ts`: the
failing case and two controls (an action for another run, an action
without a resume policy still cancel).
- `tests/restart-chaos`: new opt-in workload Q4 (a second saved question
asked by the first answer's continuation run). Provider-fault rules take
`occurrence`; Q4 rules use 2, so PV1/PV2 hit the second governed wait.
Codex shim and ACPX tap both honor it. README documents Q4.
- `doc/execution-semantics.md`: states that the run's own scheduled
resume is not a hold against it.

## Verification

- Red on master: `postgres.test.ts -t "saved governed wait"` fails with
`{ dispatched: false, status: "cancelled", errorCode:
"execution_reconciliation_required" }` (the staging error code).
- Green: the same test passes; the two controls still cancel.
- Focused: `postgres.test.ts`, `explicit-native-continuation.test.ts`,
`native-preclaim-continuation.test.ts`,
`workspace-recovery-preparation-gates.test.ts`; server typecheck.
- Harness: `question-handoff.test.ts`, `provider-faults.test.ts`,
`acpx-provider-faults.test.ts`, `journeys.test.ts`; harness typecheck.
- Chaos: `--adapters codex --workloads Q4 --faults PV2 --tasks 1 --seed
111` on master and on this branch (results in a comment below).

## Risks

- Low. The exemption is exact: same run id in the wake policy and the
evidence, status `active`, only for a native run being dispatched. A
recovery action that names another run, has no resume policy, or is
resolved keeps its current effect.
- Saved state from older versions: actions already written have the same
shape, so a resume that is still scheduled settles after deploy.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-11 09:59:22 -06:00
51a65af9b8 fix(recovery): continue preserved preparation after process loss (#19464)
## Thinking Path

> - Paperclip manages AI agents and their execution state.
> - A server restart can stop workspace preparation before provider
execution.
> - Workspace recovery must preserve files before the task can continue.
> - Recovery handled interrupted preparations but skipped failed
process-loss records with the same no-provider certificate.
> - A withheld first retry could also hide the remaining retry budget.
> - This change resumes certified preparations through normal admission
after file recovery.

## Linked Issues or Issue Description

**What happened?**
A process-loss attempt remained failed after workspace recovery
completed. Its first retry had been cancelled before dispatch by the
workspace hold. The task had no running attempt or automatic
continuation.

**Expected behavior**
Continue once after the server proves that provider work did not start,
file preservation completed, and the exact leases stopped. Retain the
existing failure budget and all task admission gates.

**Steps to reproduce**
1. Stop a Runner preparation before native runtime selection. Record the
restart recovery bootstrap certificate.
2. Retain the workspace and withhold the first automatic retry before
dispatch.
3. Complete file preservation and restart the recovery worker.
4. Observe that the old code schedules no successor. The regression test
reproduces this with PostgreSQL and real filesystem operations.

**Paperclip version or commit**
Reproduced against master at `2f35eb1080`. Related prior work: #19422
continues saved user input after verified copy-back. This change handles
automatic continuation of certified process-loss preparation.

## What Changed

- Accept failed process-loss preparation only with the durable
no-provider bootstrap certificate and existing execution, lease and file
checks.
- Discover and settle the recovered source through the existing
preparation recovery path.
- Continue from a verified withheld retry without rewriting its history
or resetting its failure count. Adopt it only as the sole retry child;
multiple-child history reuses the newest existing child
deterministically.
- Exercise automatic dispatch, preserved-file readback, one provider
turn, accepted result and task completion in the local native-session
fixture.
- Keep a durable continuation receipt after file-hold settlement until a
successor is recorded. Known gates wait for existing eligibility events.
Unresolved delivery keeps its shared recovery deadline.
- Document the recovery and retry-budget rules.

## Verification

- Red: both new PostgreSQL/filesystem regressions failed with zero
scheduled successors where one was required.
- Green: 54 focused recovery, automatic-dispatch, interruption,
multiple-child history and admission safety cases pass on `13c27c2c2f`.
The full PostgreSQL admission adapter suite passes all 83 tests; server
source declaration typecheck also passes. Local environment: Darwin
arm64, Node 26.4.0, PostgreSQL 18.1, Vitest 5.0.3.
- Additional red/green: interruption after committed settlement,
pause/resume, failed enqueue after cursor advancement, and absence of an
earlier retry all failed before their repairs. Restart before enqueue,
restart after the successor claim, and pause/resume now preserve one
continuation. A rolled-back eligibility change cannot start execution.
Deferred and delivered records have no retry deadline.
- Local acceptance uses the real heartbeat dispatcher, isolated files
and PostgreSQL, with a simulated native provider. It reads the preserved
file, writes continued work, accepts the structured result and completes
the task once.
- Negative cases retain no-certificate, provider-evidence, live-owner,
Stop, missing-stop-receipt, wrong-snapshot, human-block, pause,
reassignment, approval and exhaustion fences.
- CI fixture regression: 4 error-propagation/concurrent-resolution cases
pass after adding the new obligation read to their transaction stubs;
existing assertions remain unchanged.
- Existing idle-drain and notification contract regressions: 4 tests
pass after restricting recovery wakes to actual durable obligations.
Late approvals block scheduled, queued and final dispatch.
- Additional existing retry-scheduling suite: 62 cases pass locally. The
pre-provider quota case exceeds its existing 5-second terminal wait on
this loaded Darwin host, including when tested alone and with the
preceding retry implementation. It remains a local timing limitation;
the assertion and timeout were not changed. All Linux CI server shards
pass on the combined head, including this unchanged suite.
- Existing run-control, legacy-recovery and shared execution-control
lifecycle suites: 128 tests pass with the review repair. Required CI
supplies final-revision broad coverage.
- Real-Codex acceptance was not run: automatic approval review rejected
authenticated provider execution because it could incur cost/external
effects and required explicit approval. No provider call ran; the
controlled-provider acceptance covers the causal restart and durable
scheduling behavior.
- Hosted acceptance and recovery of existing tasks require rollout. They
have not been claimed from the local provider fixture.
- Final-head review: Greptile 5/5 on `13c27c2c2f`, no outstanding
findings. Another contributor added the multiple-child history safety
repair in `13c27c2c2f` while CI was running. The delta was inspected and
the 54 focused recovery cases rerun successfully. Required CI passes on
that exact combined head:
https://github.com/paperclipai/paperclip/actions/runs/42241794918. All
55 successful checks include aggregate `ci / verify`, typecheck, build,
canary, all server and E2E shards, and Runner package/Rust/static
checks. The separate Paperclip Review service check remains queued
waiting for its assigned agent; no validated assessment has been
returned.

## Risks

A malformed preparation certificate or a false no-execution receipt
could permit duplicate work. The existing checks still reject provider
execution records, native coordinators, live controllers, unsettled
leases and unverified files. The pending continuation must survive a
restart without polling known admission blockers. A withheld retry
retains its original row and consumes its remaining failure budget.
Normal admission checks current ownership, approvals, pause and budgets.

## Model Used

OpenAI GPT-6 family via Codex. The exact model ID and context window are
unavailable in session metadata. Used reasoning, code editing and tool
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal task id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>
2026-11-06 09:36:46 -06:00
b929ad43de fix: keep dependency waits neutral and explain GitHub access failures (#19315)
## Thinking Path

> - Paperclip manages agent work and shows users what can move it
forward.
> - Task status and tool approval cards describe different parts of that
work.
> - A queued dependency wait has no provider attempt to retry.
> - GitHub can reject an approved action while other connection tools
still work.
> - These cases showed a false Retry banner and an unhelpful provider
error.
> - This pull request keeps dependency waits neutral and gives GitHub
access failures a direct repair path.

## Linked Issues or Issue Description

Related: #19268 preserves pending Retry preparation and current failure
causes.

**What happened?**

A queued wake cancelled by the dependency gate appeared as a failed
attempt with Retry. A GitHub MCP permission denial appeared as raw HTTP
error text after an approved repository action.

**Expected behavior**

Keep the task's dependency state. Show GitHub access failures with a
clear explanation and a link to the selected connection's Permissions
page. Preserve actual failures, provider details, and approval gates.

**Steps to reproduce**

1. Queue a wake, then leave its task dependent on unfinished work before
dispatch.
2. Open the task after the dependency gate cancels that wake.
3. Separately approve a GitHub action whose MCP result contains `403
Resource not accessible by integration`.
4. Reopen the saved tool approval.

## What Changed

- Recognize both queues' unstarted dependency-wait receipts in
cancellation provenance and task projections. No dispatch or recovery
gate changes.
- Classify GitHub's specific permission error after MCP result
normalization. Keep result validation, the raw diagnostic, and working
tools available.
- Apply the same display code to both historical GitHub error codes.
Identify the provider from company-scoped connection metadata, including
renamed apps, without rewriting stored decisions.
- Show a token-based inline access card with **Manage GitHub access**
and collapsed **Error details**.
- Give agents trusted recovery steps after an approved GitHub action
fails. Keep provider text in the untrusted result and prevent automatic
replay.
- Add regressions, a production-component Storybook case, and
execution/connector documentation.

## Verification

- Red before the fix: cancellation provenance, dependency-wait
projection, the real queue cancellation, approved MCP failure
classification, saved interaction projection, and the error card
assertions failed at the expected behavior.
- Green focused tests: queue cancellation, one ready-task executor,
approval-before-dispatch, one failed GitHub request, connection reuse,
retained provider diagnostics, saved interaction reads, task list/detail
projection and redaction.
- UI: 106 component tests passed. Token gates and Storybook catalog
checks passed.
- `node scripts/typecheck-low-memory.mjs`: passed.
- Local source instance: verified a saved dependency wait stays neutral
after reload; a saved GitHub error shows repair guidance, expands its
original error, and opens the correct connection Permissions page.
PostgreSQL and HTTP APIs were real. The GitHub denial was a controlled
MCP fixture; no account or provider permission changed.
- `pnpm build`: passed for all workspace packages, the Runner binary,
server, CLI and UI. The server build and all server typecheck layers
also passed after the review repair.
- Additional safety checks passed for realistic repository arguments,
active recovery, failed Retry preparation, and bounded provider
diagnostics.
- Review repair: 45 pure tests and 7 PostgreSQL cases passed. These
cover both historical error codes, renamed apps, and
company/provider/application identity boundaries. A second local browser
check verified the renamed app, retained error detail, and correct
Permissions destination.
- Final review regressions: the dispatcher and metadata cases failed
before the repair. Afterward, 25 focused cases and 59
projection/identity cases passed, including connection-key precedence.
The native wait and app-only GitHub source also passed a local browser
check.
- The historical read adds at most one connection query per interaction
read batch. It selects IDs and the provider key only and runs only for
candidate permission failures. It adds no provider requests.
- Full local `pnpm test:run` stopped at the chat suite's `afterAll`
cleanup timeout. All 268 assertions in that group and the three prior
chat groups passed. The failure is retained here; it is not counted as a
full-suite pass. CI passed on `e8a692e1e2` after one workspace-startup
rerun. Fresh CI is pending for the combined head. Its first workspace-a
attempt timed out in the CLI company import/export test's child-server
startup wait; 4,968 UI tests and 287 CLI tests passed in that job. The
same shard passed on the preceding commit. The workspace shard passed on
its one rerun. All other test, build and browser jobs passed. The [first
failure](https://github.com/paperclipai/paperclip/actions/runs/42093823928/job/127478869941)
is retained in the Actions history. Greptile reviewed the preceding head
at 5/5. Fresh review is pending for the combined head.

## Risks

The GitHub classifier matches the documented permission-denial phrase,
not every 403. Rate limits, missing resources and other providers retain
their existing error behavior. Historical normalization affects display
only. A dependency cancellation needs an unstarted run and a supported
queue receipt; current user gates, Retry preparation and recovery remain
authoritative. No schema migration or credential changes.

## Model Used

OpenAI GPT-6, through the Codex agent. The exact runtime model ID and
context-window size are not exposed in this session. Used reasoning,
repository tools, TypeScript, PostgreSQL, MCP fixtures and browser
verification. The parallel agent-recovery commit records Anthropic
Claude Opus 5.5 as co-author.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with version and capability
details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the bug template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change
- [x] I have run the focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open findings
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 20:55:34 +00:00
DottaandPaperclip 0eaabe5c0b refactor(accounting): remove company serialization from paid settlement (#18917)
Settle paid operations through operation-scoped transactions. Preserve exact ledger evidence and replay while moving aggregate budget decisions to durable deferred work.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 07:29:28 -06:00
DottaandPaperclip 451d06e2fc fix: resume interrupted Runner preparation and confirm failure report delivery (#18853)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for work.
> - The server owns run recovery and task continuation after a restart.
> - A shutdown can interrupt Runner preparation before any provider work starts.
> - Workspace recovery can save the files but leave that task behind a legacy execution hold.
> - Retry then accepts a request without starting a successor. Startup also leaves the task blocked.
> - This pull request verifies the interrupted preparation, preserves its files, and admits one bounded continuation.
> - It also keeps a dropped Sentry capture eligible for a later report.

## Linked Issues or Issue Description

**What happened?**

A hosted task remained blocked after a graceful server shutdown interrupted Runner preparation. Its remote workspace was recovered. Retry returned an accepted request, but no successor started. Startup did not resume it. The task also had no matching Sentry event.

The recovery cause is verified. The exact cause of the missing historical Sentry event is unknown. A separate regression with the real Sentry SDK proves that processing backpressure can drop a capture while the task cache marks it as reported forever.

**Expected behavior**

Retry must continue eligible interrupted work from its saved workspace. Startup must schedule one bounded retry when durable evidence proves that no provider started. A later task read must be able to retry an unconfirmed Sentry report.

**Steps to reproduce**

1. Interrupt Runner preparation during a server shutdown before native runtime selection.
2. Retain and recover its remote workspace under the legacy recovery hold.
3. Press Retry, or restart after file recovery has completed.
4. Observe that the task remains blocked without a successor.

For the reporting case, hold the first Sentry event in its asynchronous enrichment step. Fill the SDK processing buffer. Capture a second task failure, release the first event, and read the second failed task again later.

**Paperclip version or commit**

Reproduced against source commit `4d31588dea`. The hosted incident served `6ef37398c8d799c5142a3a6590ecc70ad352b663`.

**Deployment mode**

Hosted Linux server with remote workspaces and external PostgreSQL. Local regressions use the source checkout on macOS arm64, Node 26.4.0, PostgreSQL 17.10, and the audited optional Sentry SDK 10.71.0.

Related public changes: #18541 added task-state failure reports. #18678 retained remote exports before Retry. #18803 fenced remote workspace writers before continuation.

The small backend-port fixture correction follows #18739. It is included here because the same existing setup defect blocked CI.

## What Changed

- Verify pre-native shutdown interruption against the immutable adapter claim, controller ownership, run events, native records, stopped leases, and exact workspace recovery receipts.
- Retire only the server's unchanged bootstrap failure block. Admit explicit Retry through the existing continuation path.
- Discover completed workspace recovery at startup and schedule one retry under the existing failure budget. Let saved input own its continuation.
- Apply the existing safe replacement dispatch checks to preparation retries. A later human block, competing execution, dependency, pause, approval, or budget gate must still prevent dispatch.
- Keep execution evidence available through the bounded run projection.
- Confirm Sentry task-report delivery from a successful SDK HTTP response. Keep unconfirmed captures pending for 60 seconds before a later task read can retry them. Both caches remain bounded and start no timer.
- Document recovery and report delivery semantics.
- Align existing handoff/budget assertions with the guarded preparation retry reason. Keep the owned-runtime fixture outside reserved proxy ports, with bounded allocation attempts.

## Verification

- Red before the recovery fix: all three new Retry/startup cases failed because no successor was created. Green after the fix: all three passed with real saved workspace archives and PostgreSQL.
- The pre-merge revision passes all 384 tests in the six affected recovery, run-control, run-state, retry, and dispatch suites.
- The late-block regression failed before its repair. Its four focused recovery/dispatch cases then passed, including final dispatch cancellation.
- Review found a second stop check that still rejected verified local and lease-free preparation. Both new cases failed before that repair. All 17 interrupted-preparation regressions passed on the pre-merge revision.
- The Sentry regression failed with the real SDK before the fix. All 91 affected reporting tests then passed. The transport stays in memory. No test sends events to Sentry.
- Full repository typecheck passed with `node scripts/typecheck-low-memory.mjs`. The full `pnpm build` and module boundary check passed.
- CI found untyped delivery-hook parameters when the optional Sentry SDK was absent. Explicit callback types fix that build. The SDK-absent server build and all 91 SDK-present reporting tests pass.
- Server typecheck passes with the SDK installed and the real Sentry SDK contract CI job passes. Greptile reviewed the pre-merge commit `103e175c44` at 5/5 with no open findings.
- The local full-suite attempt initially overlapped the rebase and stopped at test-line selection. Its unchanged-head rerun passed the first 309-test chat shard. I stopped the old-head run before applying review fixes. Current-head CI owns the remaining full-suite verification.
- CI exposed two old retry-reason expectations. The handoff and retry-budget checks pass with the new guarded reason. A separate existing process-adoption fixture chose port 42953 in the reserved 42000–42999 proxy range. The fixture now rejects reserved ports, following #18739. Its real process/adoption/termination test passes locally.
- The final Runner package job [initially failed](https://github.com/paperclipai/paperclip/actions/runs/41718507919/job/126205912033) one unchanged Codex credential case after it could not acquire its port quorum, then timed out. The complete unchanged 42-case credential suite passes locally. One unchanged CI retry passed; the original competing port owner was not identified. No Runner implementation or credential test was changed.
- Required CI passes on `103e175c4490e1bdf804c64a8635b0e4568545ae`, including `ci / verify` and `ci / e2e`. That revision has Greptile 5/5 with no open findings. The optional Paperclip Review check on that earlier head remained queued, waiting for its assigned agent.
- Merged current `master` without replacing published commits. The retry gate retains both shutdown interruption checks and the new workspace preparation timeout policy. All 311 tests in the five combined recovery, retry scheduling, completion, and native startup suites passed after the conflict resolution. Server typecheck passed. Final head `005d380b00d45a2393c4a2a190eab27611168d3c` includes the next conflict-free master update. All 17 interrupted-preparation regressions passed again on that final head. Greptile reviewed that exact head at 5/5 with no new findings. Required CI passes on that exact head, including `ci / verify`, `ci / e2e`, typecheck, build, the full test shards, and the real Sentry SDK isolation check. The optional Paperclip Review remains queued, waiting for its assigned agent; it is not a required branch check.
- On the final head, six test jobs were cancelled after their spot runners received shutdown signals at the same time. Another server shard passed 2,419 tests, but its unchanged 103-record cleanup backlog case exceeded the 15-second limit. The unchanged full cleanup suite then passed all 29 tests locally. One unchanged retry of only the seven unsuccessful jobs passed. The [original cancellation and timeout logs](https://github.com/paperclipai/paperclip/actions/runs/41724898038/attempts/1) remain in the workflow history. No cleanup implementation or test was changed.
- The signed-in hosted UI reproduced the failed Retry before the fix. The fixed path is covered through the real HTTP Retry route and assembled heartbeat/recovery services. This PR has not been deployed to the hosted incident. A live repaired Retry remains unverified until rollout.

## Risks

- Automatic continuation must not repeat provider work. It requires positive preparation evidence and rejects retained provider, process-launch, coordinator, result, live-controller, unsafe lease, or mismatched workspace evidence.
- Repeated recovery calls must create one successor. Tests cover concurrent requests, repeated startup visits, pending Retry, retry exhaustion, and changed task ownership.
- Sentry delivery state stays local to the process. A restart or cache eviction can report a failure again. An unconfirmed event needs a later task read to retry.
- No schema or API contract changes are required.

## Model Used

OpenAI GPT-6 through Codex. This session does not expose a more specific deployment ID or context window size. Reasoning, code editing, test execution, and browser diagnostics assisted the change.

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before requesting merge


Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 16:10:49 -06:00
DottaandPaperclip 16643a6e03 fix(runner): recover pending Stop cleanup after controller restart (#18682)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server owns execution, Stop delivery, and usage settlement.
> - A Stop can remain pending when the controller restarts.
> - Recovery must reach the existing session without starting more paid
work.
> - Startup and budget fences currently block that cleanup, and
interrupted runs leave the restart selection.
> - This pull request gives exact audited Stop cleanup a shared
authority check and records its real acknowledgement.
> - The benefit is safe cleanup while usage debt remains honest.

## Linked Issues or Issue Description

**What happened?**

A pending native run-only Stop can retain an unfinished execution owner
after controller loss. Its startup cancellation fence prevents recovery
admission. If teardown marks the run interrupted and clears its task
lock, restart selection also omits it. A recovered session can receive
cancellation without the durable acknowledgement that the normal Stop
request would save.

**Expected behavior**

Recover only the existing remote session for the audited pending Stop.
Keep fresh provider work blocked, including during budget pause. Keep
incomplete usage pending until a real complete receipt arrives.

**Steps to reproduce**

1. Start a native run with a held reservation and an authenticated
remote checkpoint.
2. Persist a run-only Stop and its startup and execution cancellation
fences before acknowledgement.
3. Lose the controller. Leave the coordinator unfinished, mark the run
interrupted, and clear its task lock.
4. Attempt restart recovery and observe the missing claim or
cancellation admission rejection.

Related work: #14733 binds caller Stop requests. #18658 handles plugin
startup cleanup obligations. #18035 proposes broader accounting changes.
This change repairs existing-session Stop recovery and preserves current
accounting guards.

## What Changed

- Add one shared check for the exact run, company, issue, pending Stop,
intent audit, live recovery lease, generation, and remote reattachment
disposition.
- Include eligible interrupted Stop owners in restart selection. Keep
generic interruptions and another task execution owner fenced.
- Reuse only the original held reservation for cleanup. Keep budget
pause effective for fresh, bootstrap, dead-runner, and unproven recovery
work.
- Apply the same cleanup authority at heartbeat runtime selection and
native entry. Skip attachment staging and warm-owner retirement only for
that exact cleanup. Revalidate at session publication and fail closed if
the proof changed.
- Send the existing Stop through the audited cancellation producer
before turn submission. Close the coordinator only after
acknowledgement, including system-owned run-only Stops.
- Give verified cleanup a transport purpose that retains pending
semantic inputs without replay, dispatch, or a semantic reply. Add an
explicit Core retain-only intake mode; preserve authenticated identity,
schema, digest, sequence and replay checks. Ordinary execution still
requires its dispatcher.
- Add PostgreSQL, native protocol, and startup-fence regressions. Update
the execution contract and recovery maps.

## Verification

- The combined Stop regression fails on unchanged production at the
cancellation reservation guard. Interrupted restart selection also fails
before the repair while its running sibling passes.
- An assembled `heartbeatService.recoverNativeRunsAfterRestart`
regression exposes the earlier runtime-selection fence with the rest of
the repair present. The fixed path reaches selector/claim, runtime
selection, original held-budget reuse during pause, dispatch, real
native protocol, audited cancellation, and heartbeat settlement. It
starts no provider turn, acquires no new sandbox, and does not reclaim
the task lock.
- Fail-if-called attachment staging and turn boundaries remain
untouched. A retained warm-owner regression prevents retirement.
Invalidation at session publication fails closed. Actual backend/driver
constructor tests verify cleanup-purpose forwarding, absent default
options, and denial under foreign ownership.
- Missing usage retains accounting debt and the hold. A later complete
receipt records one 25-cent charge and settles that same hold.
Concurrent duplicate cleanup is denied. Both user-owned and system-owned
Stops pass.
- Real transport tests: 9 pass on Node 24.22.1. They preserve ordinary
saved-tool and completion recovery, prove the authority-denial race
before cancellation, and qualify cleanup with pending
ordinary/completion inputs and either handler ordering.
- Real authenticated Core intake: 11 pass. Callback-present and
callback-absent retain-only cases persist and acknowledge unanswered
input without dispatch, recovery or result. Default missing-dispatcher
rejection and digest-integrity/suspension guards remain effective.
Omitting the callback alone first failed by disconnecting; the explicit
retain-only mode fixes that defect.
- Repository `pnpm -r typecheck`, the server/runner-vendor build, map
checks, and module boundary checks passed locally before integrating the
latest retained-startup change. That integration passes 11 native
startup/cleanup compatibility cases and 3 assembled recovery/accounting
cases. The final broad affected-suite attempt was stopped without a
verdict; it is incomplete. Full repository tests and build are delegated
to current-head GitHub CI.
- Review exposed an additional real-process close failure with
unanswered input. The corrected path requires a fresh exact
provider-stop receipt confirming exit, followed by durable event drain
and exact runner suspension. All 11 focused transport cases pass,
including missing/false exit-proof negatives and ordinary recovery
siblings. Runner TypeScript and map validation pass after this
correction.
- The assembled test controls provider, workspace-transfer and GitHub
bootstrap I/O. It does not qualify hosted credentials, physical provider
termination, workspace preservation, or historical receipt recovery.
Local socket-permission setup failures are separate from semantic
regression results. No hosted replay has run on this repair.

- CI exposed an ordinary factory-call compatibility regression and an
obsolete authority-read test fixture. The repair preserves
single-argument ordinary calls and updates the race fixture without
changing its Stop assertions. The complete native executor suite now
passes all 1,049 tests; the ordinary factory boundary and all five
Stop-accounting cases pass. A separate local environment-default case
twice exceeded its five-second wait while other work ran; its prior-head
CI case passed. The same unchanged case passed an isolated diagnostic
without changing its assertion or timeout; the precise contention source
remains unproven.

- The final fixture edit initially used the wrong overloaded promise
type. CI rejected it; the annotation-only correction passes the exact
`pnpm run typecheck:build-gaps` entry point locally, including server
service tests and remaining workspaces. No runtime assertion changed.

- Final exact head `007fd4d04a2d6338c6dc073e97ab470a277fc3d0`: required
`ci / verify` and `ci / e2e`, build, typecheck and test groups pass.
Greptile is 5/5 with both findings resolved. Three expected checks are
skipped; the optional native check is neutral, while its final
current-head comment gives 5/5 from source inspection without a checkout
or test execution. The first server-shard attempt lost its self-hosted
runner connection; one scoped rerun passed. Its original annotation and
unavailable-log evidence are retained.

## Risks

- This changes execution authority. Cleanup requires an audited, exactly
bound pending Stop, an unfinished coordinator, a live remote
reattachment claim, and the original held reservation. It never
authorizes a replacement provider turn.
- Missing checkpoint, original held reservation, termination, or usage
evidence can keep cleanup or accounting pending. The repair does not
manufacture acknowledgement, replies, receipts, or zero cost.
- Retain-only intake is an internal trusted constructor option. It
changes no agent tool permission. Ordinary callback requirements and
semantic dispatch remain unchanged.
- No schema, public API, provider profile, or deployment setting changes
are required.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source
inspection, code execution, and regression testing. The context-window
capacity is not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the bug report template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket ID
- [x] I have run affected tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open findings
- [x] I will address all reviewer comments before merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 18:22:40 -06:00
DottaandPaperclip 27270e9bf8 fix(server): wake interrupted retry cancellation from committed intent (#17667)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A parked scheduled retry has no executor to finish an interrupted
Stop request.
> - Its saved cancellation fence already records the work that remains.
> - The orphan reaper currently scans for that work on every heartbeat,
including idle periods.
> - This PR registers the existing records with the shared delivery
scheduler and wakes recovery after committed writes.
> - Recovery keeps the existing cancellation guards and settlement
behavior, then becomes quiet.

## Linked Issues or Issue Description

Refs #17666, the merged preceding PR in the recurring-server-work
migration stack. This PR targets master. Related cancellation work found
in the dedup search: #12301 and #7340. This PR changes the trigger for
interrupted parked-retry cancellation; it does not replace their
cancellation or retry policies.

**What happened?**

The periodic orphan reaper queries scheduled retries with cancellation
fences even when no cancellation needs recovery. Its first-20-row scan
can also stop behind malformed evidence.

**Expected behavior**

Committed cancellation intent wakes bounded recovery. Failed work
retains a retry. Startup reconstructs durable work. Empty or settled
work performs no recurring SQL for this duty.

**Steps to reproduce**

1. Start a server with heartbeat scheduling enabled and no interrupted
retry cancellations.
2. Observe the orphan reaper repeatedly query for cancellation fences.
3. Cancel a scheduled retry and fail the final status write after its
fence commits. Recovery must complete the existing Stop request without
running a provider or dropping its metadata.

**Paperclip version or commit**

Current parent squash `7afee8f0622e273d10506b0febfb68f143d168b1`. Source
acceptance `66107ae0a7ad85fd0476f91c4f8f98ae74a7a048`.

**Deployment mode**

Self-hosted server with the existing single-process transaction signal
model. Local evidence uses PostgreSQL 17.

## What Changed

- Add one domain-owned recovery worker over existing heartbeat run rows.
No new table. It delegates to run control and preserves original
cancellation, budget, and operator metadata.
- Signal cancellation fences, lifecycle transitions, dispatch
deferral/promotion/suppression, relevant settings changes, and deletion
within their owning transactions. Wrap only the ordinary database fence
in a short transaction; provider calls stay outside it.
- Keep startup/manual recovery compatible and remove this duty from the
periodic orphan sweep. Other reaper duties remain for separate PRs.
- Extend the shared scoped worker to retain pending work with a
memory-only admission retry while its owner is paused. Without pending
work, it schedules nothing. Existing GitHub consumers retain their
behavior.
- Add a concurrent idempotent partial index and lifecycle/producer
tests. Enable guarded disposable PostgreSQL URLs in affected test
harnesses.
- Reconstruct one hundred candidates per scope per scheduler pass.
Resume from the saved scope cursor and retain fair twenty-item execution
batches. Manual recovery starts cancellation before reading the full
backlog.
- Require the recovered browser reply to be saved before testing reload.
Keep all original assertions and time limits.
- Record the full producer audit, lifecycle limits, and failed attempts
in `doc/plans/2026-10-25-scheduled-retry-cancellation-sleep-audit.md`.

## Verification

Published head: `405e533394b26215e22d0452e9b0207d3f784a79`. Source
acceptance: `a35fa43b98e2bd378d4a920d2775ce243654838b`. The final commit
changes the audit only.

- All 363 backend cases in ten complete owner and consumer files pass
without skips. This includes actual Stop/retry producers, transaction
settlement, interrupted-write recovery, pause/resume, shutdown, fair
backlog processing and quiet after settlement. Seven separate fresh
PostgreSQL 17 databases prevent fixture interference.
- The two embedded-only backend fixtures temporarily use separate
external databases on this IPC-exhausted host. Their fixture imports and
native client teardown are restored byte for byte afterward. Assertions
and timing limits are unchanged. CI must still pass with the original
embedded fixtures.
- Two paging regressions fail on the prior worker and pass on the fix.
They cover cancellation before full reconstruction, a malformed first
page, fair batches, a newly committed intent behind the cursor, and no
later idle SQL. Per-candidate admission reads remain deliberate safety
checks.
- The repeated CI reload failure is reproduced with a temporary
twelve-second fixture settlement delay. The trace retains the same
five-second missing-reply assertion, even though a cleanup race masked
it in the top-level error. Requiring exactly one saved reply from the
resumed run passes under the identical delay. The fault is removed. All
six browser journeys pass on the final source in 40.5 seconds, with zero
retries and the original 120-second case and five-second reload limits.
- Browser acceptance uses the canonical CLI/server/static-UI path and a
fresh external database. Only fixture database-URI setup is temporarily
adapted and then restored. This is not an unmodified embedded
CI-environment reproduction. The six cases include the controlled case;
repeated runs are not added to the distinct count.
- Actual database build, server typecheck/build, module boundaries and
exact-current migration order pass. The database source tree is
unchanged from the verified 0405-to0406 upgrade: all 404 prior receipts
and 300 companies stay intact, exactly one receipt is added, replay adds
none, and the index is valid on upgraded and fresh databases. The five
snapshots remain 0402 through 0406.
- All earlier failures remain recorded in the audit. Prior head
`48b3fcb83f305616df02a4337a42ec0867bb4b80` had exact-head Greptile 5/5
but failed the repeated browser assertion. It was not retried blindly.
Its completed Ponytail review's backlog concern and old migration
paragraph are addressed here. Old scores do not satisfy new-head gates.
- Current-head CI passes. Exact-head Greptile and Ponytail both give
5/5, with no remaining actionable findings. These checks do not prove
deployed instance or database suspension.

## Risks

- Only supported in-process writers produce local transaction signals.
Independent SQL writers or other replicas need a separate ingress
contract. Startup reconstructs saved obligations.
- A real failure retains bounded retries. Unknown commit outcomes retain
the shared intent hold. Neither is discarded to report quiet.
- Per-owner drain uses a memory-only admission recheck. The shared
helper change affects existing consumers, which have focused regression
coverage.
- The partial index is concurrent and idempotent; it adds index storage.
Each scoped load reads at most one hundred rows. The shared helper
retains discovered pending IDs between passes, so a large unresolved
backlog still uses memory.
- Cloud alarms, idle-safety exemptions, and other reaper duties remain
separate. Merge requires passing current-head CI and review gates.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, shell tools, and local
PostgreSQL tests. The runtime does not expose a more precise model
identifier or context-window size for this turn.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 21:07:55 -05:00
DottaandPaperclip 30b753f614 perf(budgets): scope accounting and deliver enforcement asynchronously (#17046)
## Thinking Path

> - Paperclip manages AI agents and their paid work.
> - Budgets must prevent new work and stop reported overruns.
> - Cost settlement also ran serial lifecycle reconciliation for every
eligible company agent.
> - That made request latency grow with unrelated idle agents and
delayed results on stop delivery.
> - This change separates atomic admission and accounting from durable
asynchronous enforcement.
> - Scoped live usage detects overruns without writing provisional
ledger charges.
> - The result is constant query scope for ordinary settlement and
recoverable stop delivery.

## Linked Issues or Issue Description

**What happened?**

Settlement and budget checks entered company-wide lifecycle
reconciliation. A company with 1,000 unrelated idle agents performed
1,001 accounting transactions for one ordinary cost evaluation. User
requests could also wait for provider termination. Running usage
checkpoints did not enforce reported monetary exposure.

**Expected behavior**

Keep the final reservation gate authoritative. Commit charges, scope
blocks, and enforcement intent before returning. Deliver stop requests
after commit. Read only applicable policies and execution scopes for
ordinary charges. Stop long-running work when supported reported or
priced usage establishes an overrun.

**Steps to reproduce**

1. Create a disposable company with one company budget and idle agents.
2. Evaluate a charge below the budget with 0, 17, and 1,000 unrelated
idle agents.
3. Compare query counts, company accounting transactions, and duration.
4. Use the controlled partial-usage and delayed-stop fixtures to check
enforcement and request responsiveness.

**Paperclip version or commit**

The measured baseline is `6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837` from
`master`.

Related public work: Refs #16976, #2006, #3539, #13763, #14296, #16853,
#13075. These cover the affected service charge, cancellation, and
accounting paths. This change removes their shared reconciliation
overhead.

## What Changed

- Batch applicable policy exposure under the existing company accounting
lock. Preserve exact fractional money and zero-reservation dispatch
fences.
- Commit a durable final budget dispatch claim before provider I/O.
Retain a separate final issue/run ownership transaction through
synchronous handoff, without holding the company accounting lock or
awaiting provider work. Preserve native recovery lease checks and
duplicate handoff fencing. Browser handoff rechecks current policy; it
reuses a consumed rate slot only for the identical policy generation.
- Separate ledger spend, live observed spend, and effective held
capacity. Use the maximum of reservation and live observation for each
unfinished operation.
- Deliver budget cancellation through the existing coordinator.
Revalidate generations and persist queued/running stop fences before
external termination. Bound stop concurrency to four.
- Apply agent holds only to the affected agent. Project budgets leave
other projects running. Project real company transitions in bounded
lifecycle batches.
- Replace periodic broad reconciliation with durable pending work,
startup repair, receipt replay, and concrete retry/window deadlines.
Retain drain and idle-safety checks.
- Enforce accepted cumulative usage checkpoints without provisional
charges. Final accounting atomically replaces live exposure and settles
reservations.
- Add operator-gated budget spans, regression coverage, map updates, and
a real-PostgreSQL comparison harness. No schema or first-party telemetry
change.

## Verification

- Full workspace `pnpm -r typecheck` passes on the final ownership fix,
rebased onto `bbcf5dfe98`. Production `pnpm build` passes on final head
`4c5f4de8b1`. Map, module-boundary, token, and Storybook catalog checks
pass.
- Focused real-PostgreSQL coverage includes the 292-case rebased
accounting/dispatch run, 63 browser cases, 52 run-control/cancellation
cases, 42 policy-generation cases, 77 lifecycle/commentary cases, and
the exact Muse confirmation regression. Native executor unit coverage
passes all 768 cases. Final ownership coverage passes: 74
adapter/use-case cases, nine public regression cases, and the additional
deleted-project race. These exercise competing ownership, budget changes
and charges between phases, zero/nonzero duplicate claims, rollback,
recovery contention, read-only capacity rechecks, and controlled HTTP
startup after commit. Another 74 pure/startup/tracing cases pass. The
complete CI matrix and both 5/5 reviews passed on `1ac9b58`; its
external security scan identified a stale-owner handoff race. The final
fix restores synchronous issue/run ownership fencing after the
accounting transaction commits. Final head `8e84f55ce8` has fresh
Greptile 5/5 with zero new comments, Paperclip review 5/5, and a passing
security scan. The complete latest-head [CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/39535189808)
passes, including budget, browser, Runner, build, and canary jobs. The
unchanged Runner close fixture passed on its sole same-head job retry.
- Broad local `pnpm test:run` invocations were stopped before required
source updates (rebase and review fixes). They are not claimed as full
passes. Final full-suite coverage comes from the complete latest-head CI
matrix; local focused coverage and the incomplete broad invocations are
reported separately.
- Reproduce performance with `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-budget-enforcement.ts
--baseline-ref=6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837`; repeat with
`--reverse`. The harness uses disposable real PostgreSQL, 20 warm
samples per variant/count, and records raw timings, SQL/lock counts,
agent reads, unchanged agent state, and query plans. Final
production-source measurement revision:
`eff713a4f72c7f4b7a4002a701cb481d4bfccd95`; the extended harness is
committed in `4c5f4de8b1`.

| Unrelated idle agents | Baseline median / p95 | New median / p95 |
Baseline SQL / locks | New SQL / locks | New agent reads |
| --- | ---: | ---: | ---: | ---: | ---: |
| 0 | 1.80 / 2.33 ms | 2.99 / 3.88 ms | 5 / 1 | 7 / 1 | 0 |
| 17 | 15.53 / 19.55 ms | 2.24 / 2.60 ms | 90 / 18 | 7 / 1 | 0 |
| 1,000 | 978.35 / 1,790.43 ms | 2.16 / 2.52 ms | 5,005 / 1,001 | 7 / 1
| 0 |

- These measurements use local PostgreSQL 18.1 and Node 26.4.0 arm64.
They isolate the cost-evaluation core and complete admission path,
without hosted-region or provider latency. Prior-revision reversed-order
controls also preserved the constant query counts; the final
measurements above use the final production source.
- Complete reserve → final ownership gate → immediate offline provider
result is measured separately, with one responsible agent and one active
operation at each unrelated-agent count. Admission p95 baseline/new: 0
agents **11.54 / 29.77 ms**; 17 agents **9.31 / 27.62 ms**; 1,000 agents
**24.51 / 74.05 ms**. Baseline admission was already scoped; the
stronger final fence adds bounded overhead. New counts stay constant at
53 Drizzle statements, three transactions, two company-accounting locks,
and five agent-reading statements bound to the responsible agent. Zero
unrelated-agent reads, exactly one provider callback per sample, and no
leftover runs or holds. All admission scenarios stay below 250 ms. No
external provider calls are included.
- The partial-usage/delayed-termination fixture passed 20/20: receipt
persistence p95 13 ms, receipt return to observed durable Stop fence p95
28 ms, complete receipt-start-to-fence p95 50 ms, unrelated database
request p95 1 ms while termination was blocked. After releasing the
controlled termination fixture, acknowledgement p95 was 14 ms. These are
separately measured phases; fixture acknowledgement is not live-provider
termination latency.
- Models and provider deadlines are unchanged. No deployment, live
inference, or staging acceptance is claimed. A quiet 30-minute follow-up
waits for the merged revision on staging, then repeats classifier and
controlled budget-stop acceptance in staging, recording hosted
bookkeeping, queue delay, and actual provider acknowledgement
separately.
- Development failures retained: finalizer notification regressions
(fixed; 27/27 affected cases passed), invalid browser admission fixture
assumptions (fixed; browser coverage now passes 63/63), and first-head
lifecycle/dispatch CI regressions (reproduced and repaired). A
prior-head Cases E2E test hit its unchanged announcement-dismiss handler
race while opening a filter; the same journey passed on `1ac9b58`. The
external security review then found a stale-owner race between
accounting commit and provider entry; final verification covers a
competing run in that gap while retaining provider I/O outside the
accounting lock. Early PostgreSQL startup attempts hit local semaphore
pressure; serial reruns succeeded. The final-head Runner lane had one
failure among 1,823 tests: the unchanged `timeout-attachment` fixture
failed at its first process close after 6.28 seconds. The sole same-head
job retry passed all 1,819 active tests (four skipped), including that
fixture in 1.85 seconds. Three isolated local probes could not run
correctly because the Rust fake-provider fixture binary was absent;
cleanup masked the original startup failure. CI builds that fixture
through its normal lane. These probes are inconclusive and are not
counted as passes.

## Risks

- This changes accounting, lifecycle, and cancellation ordering. Durable
generations, source/sequence checks, ownership leases, and company
locking remain required for correctness.
- Provider stop acknowledgement can be slow or fail. Stop intent remains
pending until existing termination proof succeeds; cancelled row status
alone does not acknowledge delivery.
- Mid-run limits depend on provider usage reports or supported price
calculations. Providers that report only at completion cannot provide
exact mid-call detection.
- Signals are process-local wake hints for the supported
single-active-writer model. Durable state and startup recovery remain
authoritative.
- There are no migrations, new queue tables, or cached balances. Repeat
the classifier and controlled budget-stop acceptance after normal
rollout.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution,
and parallel coding agents. The host does not expose the exact backend
model revision or context-window size, so those values are not asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 07:23:00 -05:00
Devin FoleyandPaperclip 06484b3c41 fix: preserve conversation retries through execution cleanup (#15463)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Recovery schedules bounded retries after a provider disconnects.
> - A stopped run can still hold its environment lease while cleanup
runs.
> - Retrying before that lease is released cancels the new run before it
starts and spends another retry.
> - Restoring the task can also send the worker repair instructions from
an already resolved recovery action.
> - This pull request preserves the waiting retry and removes settled
recovery instructions from later wakes.
> - The task can continue after cleanup without an operator repairing
the same incident again.

## Linked Issues or Issue Description

**What happened?**

A legacy conversation run disconnected while it was doing ordinary work.
Its environment cleanup took longer than the retry delay. Two retries
were cancelled before dispatch with `execution_reconciliation_required`.
Those cancellations exhausted the failure budget. After an operator
restored the task, the wake still told the original worker to repair the
runtime and hand the task back to itself.

**Expected behavior**

Cleanup waits preserve the pending attempt. The same retry can continue
after ownership is released, subject to all current gates. Once a
recovery action is resolved or cancelled, subsequent task wakes omit its
repair instructions.

**Steps to reproduce**

1. Fail a legacy conversation run while its environment lease remains in
`pending_cleanup`.
2. Schedule a bounded retry and run promotion before cleanup releases
that lease.
3. Repeat the scheduler sweep. Before this fix, retries promote and then
cancel without starting.
4. Resolve a stranded-task recovery action and build the restored task
wake with that action ID. Before this fix, the wake still includes the
settled repair instructions.

**Paperclip version or commit**

Reproduced with database regressions against `ceabc3bc880` on master.

Related: #15019 restores a skipped assignment handoff after lease
release. #15235 filters stale handoff evidence in the recovery sweep.
This change preserves an existing scheduled conversation retry and
corrects restored wake content. It does not create a new handoff wake.

## What Changed

- Keep an unstarted legacy conversation retry on the same durable row
while prior execution ownership remains active. Recheck after 30 seconds
without increasing retry accounting.
- Return a queued retry to scheduled state if it encounters that hold at
the claim gate. Retain its issue claim and publish the status change.
- Record one local lifecycle diagnostic per blocking run. Remove that
wait marker on promotion.
- Include recovery action metadata only while the referenced action is
active or escalated.
- Return an explicit `waiting` response and the saved schedule when
Retry now meets cleanup. Show the wait inline without a false success or
disabled button.
- Add database, rendered-prompt, route, and UI regressions. Document the
execution and run-log contracts.

## Verification

- Five cleanup and restored-wake regressions fail against the original
production code. The Retry now route and UI regressions also fail before
their correction.
- Related retry, dispatch, stale-queue, and recovery suites: 321 tests
pass across seven files. All ten focused cleanup/restored-wake cases
pass after rebase. The final dispatch adjustment passes all 46 adapter
tests.
- Retry now routes and affected UI suites: all 46 tests pass. The tests
cover repeated clicks, the saved schedule, unchanged accounting,
promotion after release, and no false success or error state.
- `pnpm -r typecheck`, `pnpm build`, `pnpm build-storybook`, and `pnpm
check:token-gates` pass.
- Full local `pnpm test:run` was started and then stopped after the
final commit passed all GitHub CI test shards. No complete local
full-suite result is claimed; CI supplies the complete test result for
the final commit.
- Final head `6f1058a332c039e33c4f002b296d20a5554e760e`: all 55 GitHub
checks are green or intentionally skipped. Apex review is 5/5 after two
reviews, with no unresolved threads. The PR has no merge conflicts.

## Risks

The wait applies only to unstarted legacy conversation retries. Native
runs and non-conversation execution keep their existing recovery rules.
Cleanup must actually release ownership before execution can resume. The
wait does not fix a cleanup service that never finishes. Promotion and
dispatch still enforce cancellation, reassignment, pause, budget, and
reconciliation gates. No schema change is required. The Retry now
response adds a `waiting` outcome; the shared contract and all three UI
controls handle it.

## Model Used

OpenAI Codex based on GPT-6, with repository analysis, tool use, and
local code execution. The runtime does not expose the precise serving
model ID or context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; complete
final-head test coverage in CI)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-07 11:15:49 -07:00
DottaandPaperclip f2715e02bb fix(native): surface model capacity errors and retry automatically (#15347)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs preserve provider events and results, then decide task
state.
> - Codex can fail a turn because the selected model is temporarily at
capacity.
> - Paperclip displayed a generic native-session failure and left the
task in recovery.
> - A known capacity failure needs a clear message and a durable delayed
retry.
> - This pull request uses committed terminal evidence, the
status-effect ledger, and existing dispatch gates.
> - The task can continue automatically without an unbounded retry loop
or a model switch.

## Linked Issues or Issue Description

Related: #13993 adds launcher capacity deferrals before provider
startup. This change handles a committed native Codex `serverOverloaded`
terminal after provider work has started.

**What happened?**
A native Codex turn ended with `codexErrorInfo: serverOverloaded` and
“Selected model is at capacity. Please try a different model.” Paperclip
saved the result, showed a generic failure, and required recovery
instead of waiting and retrying.

**Expected behavior**
Show the capacity error directly. Schedule bounded retries after a short
delay. Preserve saved work and honor current execution gates.

**Steps to reproduce**
1. Run a native Codex task.
2. Commit a `turn.failed` event with `serverOverloaded`, then accept a
failed result for the same turn.
3. Inspect the task error and its recovery state. The regression test
reproduces this without a paid provider call.

**Paperclip version or commit**
Reproduced against master `16b7db35ffa0f9a95913c8cbdeea3d595435691f`.

## What Changed

- Classify capacity failures from committed runner events and the pinned
execution identity. Preserve accepted results and display the specific
capacity error.
- Atomically persist one scheduled successor with the status decision.
Retry after one minute, then two minutes. Share the existing failure
budget and stop after two automatic retries.
- Reuse dispatch gates for ownership, task holds, dependencies, budget,
and locks. Wait for predecessor execution, finalization, and cleanup
before claiming a retry.
- Preserve pending reviewer authority and suppress retries after
reassignment or a successor claim.
- Consume the failed run's resume receipt and delivered wake input.
Rebuild ordinary continuation from the failed run so explicitly resumed
tasks can retry without borrowing one-run authorization.
- Label scheduled retries “Model at capacity.” Add a Storybook example
and avoid duplicate punctuation in the existing retry card.
- Add regression coverage and document the runtime contract.

## Verification

- Targeted recovery and UI suites: 125 tests passed. Additional final
cleanup and lock regressions passed.
- Latest-head continuation and authorization regressions: 57 tests
passed, including all four capacity integration tests against an
isolated PostgreSQL database.
- Rechecked all four capacity integration tests with the full runner's
isolated `PAPERCLIP_HOME`, config, temporary directory, and serial fork
settings: passed.
- `pnpm -r typecheck`: passed. Final server typecheck passed.
- `pnpm build`: passed.
- `pnpm check:token-gates`: passed.
- Browser: checked the real Storybook card. It shows the capacity
message, automatic retry time, and existing Retry now action.
- `pnpm test:run`: attempted and restarted after an interruption. The
resumed run started before the final review correction and was stopped
after recorded workspace/native test failures and the five-minute Git
streaming timeout already documented on master. Exit 130; no complete
local full-suite pass is claimed. Current-head isolated capacity tests
and the complete CI suite pass.
- A combined local recovery-suite attempt also hit PostgreSQL
initialization failures at this macOS host's global shared-memory limit
(32 slots). The focused recovery suites passed separately; final-head
capacity tests also pass with the full runner's isolated environment
settings.
- Latest-head GitHub checks: all 56 checks green, including general and
serialized test shards, browser E2E, typecheck, build, Runner
verification, and canary dry run. No merge conflicts.
- Greptile: 5/5 on `813a470b8c18c05aeb7e31e63e573c3a3a2f5cac`, with zero
unresolved findings after fixing the resumed-task continuation issue.

## Risks

- Capacity retries can repeat a task turn after partial work. They start
a fresh provider session with task history and wait for predecessor
cleanup. They do not resume the failed turn.
- Retries retain the configured model unless an operator changes
configuration. Persistent overload consumes the existing failure budget
and then needs an explicit retry or model change.
- Only run-bound native Codex `serverOverloaded` failures qualify.
Usage-limit exhaustion, model/account incompatibility, unbound text, and
unknown provider failures retain their current recovery behavior.
- No schema migration is required.

## Model Used

- OpenAI GPT-6 through Codex. The exact model ID and context-window size
are not exposed to this session. Used reasoning, repository tools, code
execution, and browser verification. No subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-06 10:59:18 -05:00
DottaandPaperclip 422287eecd fix: preserve runner recovery, warm sessions, and task outcomes (#13338)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner connects task messages, provider execution, and
task outcomes.
> - First-time user tests exposed gaps in recovery, completion
permissions, message delivery, and Stop behavior.
> - These gaps left usable output hidden, completed work waiting for
bookkeeping, or safe work unable to continue.
> - This pull request fixes the shared lifecycle and receipt paths while
preserving process ownership and action checks.
> - Users can continue work with accurate task state and durable
messages.

## Linked Issues or Issue Description

**What happened?**

A stopped local Codex execution could remain blocked even after its
processes had stopped and its complete transcript proved that no
external action needed replay. Claude under Conservative permissions
could fail to call task completion tools. Recovery could reuse an
assistant item ID and overwrite prior output. A delivered comment could
remain marked uncertain after navigation. Stop could look like Pause or
a new recovery incident. Workspace contention could look like
cancellation. A direct reply reopening Done could enter a clarification
loop.

**Expected behavior**

Recover automatically only with verified termination and complete action
receipts. Preserve answers and messages. Keep task completion available
under Conservative permissions without broad tool access. Show crashes
as Blocked, actual human decisions as In Review, and ordinary workspace
contention as waiting. Stop the current response and allow a new
direction.

**Steps to reproduce**

1. Create ordinary response tasks with local Codex and Claude Code, then
send follow-up messages through the task composer.
2. Interrupt a disposable local Codex runner during text-only work.
Verify automatic continuation and retained output.
3. Stop a response, send a new request, answer a clarification, and
reopen completed work with another message.
4. Navigate or reload while a comment submission is pending. Confirm the
exact persisted request receipt settles it without removing newer draft
text.
5. Run two tasks in a shared Daytona workspace. Confirm waiting does not
appear as failure.

**Paperclip version or commit**

Initial acceptance baseline: `c9021c6721f91e2c74bd9fee9d3fd41c999d17b7`.
Current integration base: `6cef9743c`. Both operator-interruption and
workspace-waiting guards are preserved; native restart and legacy
permission rules remain documented.

**Deployment mode**

An isolated source-built test-drive instance, with real local Codex and
Claude Code providers and disposable Daytona environments.

Related work: #13314, #13316, #13327, #13344, #13239, #13254, #13163.
This PR addresses additional failures from ordinary task journeys,
including controller restart handoff and repeated warm sandbox setup.
Historical task status reconciliation is excluded.

## What Changed

- Persist runner ownership immediately at spawn and resume an explicitly
adopted runner even when the controller crashed before the first driver
checkpoint. Detach the controller safely across graceful restarts,
including session startup. Prevent an old finalizer from suspending or
signaling an adopted runner. Checkpoint idle warm sessions before
shutdown. Preserve the same run and queued follow-up messages.
- Scope saved legacy queue successor checks to the queue owner while
preserving ordinary task locks, operator identity, assignment gates, and
exactly-once delivery.
- Preserve managed Codex credential files when an old session is
detached for restart; normal owned cleanup still copies refreshed auth
back and removes the scoped copy.
- Reuse the bound warm shared sandbox and fully verify an existing
staged provider pack before using it. This avoids repeated uploads when
the pack is already valid.
- Add a narrow local Codex replacement path with stopped-process proof,
a closed transcript inventory, exact completion receipts, and
fresh-session lineage. Preserve no-replay holds when evidence is
incomplete. Recovery may clear only the same run's recorded Blocked
status version; manual re-blocking and dependency changes invalidate
that receipt, while queued comments do not. Later blocks stop scheduled,
queued, and final dispatch; queued/final checks re-read dependencies
even when the task status stays In Progress.
- Permit only task delivery and human-input tools through the isolated
Claude runner's exact task bridge.
- Scope assistant item identity to the provider turn and ignore only
authority-free Codex skill-change notifications during startup.
- Reconcile composer submissions by client request ID across response
loss, navigation, and reload. Retain text typed during delivery.
- Keep acknowledged run-only Stop neutral and show workspace contention
as waiting. Project exhausted native failures as Blocked.
- Restore the guarded task-page retry action for failed legacy runs,
including the server-supported explicit new-attempt path for stopped
conversation adapters. Preserve native/process recovery holds and avoid
promising Retry while a decision or execution gate hides it.
- Refresh delivered artifacts and handle direct user replies that reopen
completed work without a clarification loop.
- Check the embedded PostgreSQL PID, data directory, and actual port
before connecting or migrating.
- Document accepted behavior and add focused regressions at lifecycle,
route, transcript, and UI boundaries.

## Verification

- Final head `fece606ac2` passes the complete GitHub CI matrix: **34
green checks, two expected Storybook skips, no failures or pending
checks**, including `ci / verify`, `ci / e2e`, full runner verification,
typecheck, build, every server/workspace shard, and all browser shards.
[CI
run](https://github.com/paperclipai/paperclip/actions/runs/34727183287).
Greptile is **5/5 with no open findings**. The final two commits only
refine test fixtures; both affected suites pass 24/24 locally and in CI,
with server typecheck green.
- Complete local Vitest coverage uses the canonical groups/shards: all
635 general server suites, all 145 serialized suites, and all workspace
packages. The aggregate began on `0a8001c18` while the final queue fix
arrived: 23,903 passed, five failed, 87 skipped. The five
port/socket/timing failures passed unchanged in follow-ups (60 tests in
the exposure/file suites and 412 tests covering the serialized failures
and unrun tails). The final queue/operator-identity suites separately
passed 52/52. This is aggregate coverage plus explicit reruns, not a
pristine single-command final-head run.
- After integration with current master,
queue/operator-identity/continuation suites passed 162/162 and affected
UI suites passed 140/140. ACP Stop/continuation and legacy
task/Inbox/message browser suites passed 9/9, including both task
recovery Retry and thread Try again, automatic saved-message delivery,
exactly one new run, Done, and retained output after reload. The default
process Stop/Pause/Resume browser case passed (the native-provider case
is opt-in and skipped by default). The complete Board attachment/receipt
browser suite passed 11/11 on a disposable instance, covering both
composers, exact receipts after lost responses, no replay, bound
attachments, and newer drafts after reload.
- Blocking-intent regressions cover pre-existing Blocked, a mismatched
run/cause, an explicit manual re-block, changed dependencies, a queued
comment after failure, and a block arriving between scheduling and
provider dispatch. The negative cases reproduced before the fix. All 478
affected executor/recovery/dispatch tests passed; both database suites
ran separately after availability-probe skips in the first combined
command. The final late-dependency check passed all 143 affected
recovery/dispatch tests (zero skips) after two new negative cases
reproduced the bug.
- Focused runtime regressions cover awaited runner ownership
publication, authenticated adoption before the first checkpoint,
old-finalizer detachment, idle and busy warm-session shutdown, rejected
checkpoint propagation, provider-pack verification, and managed-Codex
credential preservation. Four managed credential detachment cases
reproduced the bug before the fix; normal owned cleanup still succeeds
exactly once.
- Live local Claude: SIGKILL 2.6 seconds into startup recovered the same
run automatically in 53 seconds, then a normal follow-up completed in 24
seconds. SIGTERM 2.5 seconds into startup preserved the same run (54
seconds) and its queued follow-up (21 seconds). Answers remained visible
and the task reached Done.
- Live Claude Daytona: a warm follow-up retained its sandbox and fell
from 121 seconds to 44 seconds. A separate cold turn took 127 seconds;
after controller shutdown and checkpointing, its follow-up completed in
33 seconds with the same sandbox, workspace, native session, and runner.
Both answers remained visible and the task was Done.
- Other live journeys covered task completion and follow-up with local
and Daytona Codex, local Codex crash recovery, Stop then new direction,
clarification response, live artifact refresh, and shared-workspace
waiting.
- Validation limits: the opt-in native composer Stop/Pause→subtree
Resume fixture exposes terminal/result ordering and subtree-cancellation
attribution bugs that can leave a child task blocked; that new finding
is assigned to a separate follow-up and is not claimed fixed here.
Default CI skips this optional native-provider fixture. Managed-Codex
credential handoff and the queue-agent integration use automated
regression evidence. Cold custom provider-pack uploads still add startup
latency.

## Risks

- Automatic replacement remains deliberately narrow: local Codex,
verified stopped identities, unchanged retained state, and a complete
text/completion-only turn. Unknown actions, partial history, or changed
ownership remain blocked.
- Claude completion permission handling changes an upstream package
patch. The exact isolated task bridge must remain pinned; unrelated
tools keep their existing permissions.
- New task failure projection changes user-visible status. No historical
status backfill or database migration is included.
- This is a broad lifecycle fix across server and UI. Live proof covers
graceful local Claude restart during startup and idle Claude Daytona
session recovery across controller shutdown. Live abrupt SIGKILL during
local Claude startup also recovered the same run. Unknown ownership or
missing action evidence still blocks reuse. Cold custom provider-pack
uploads still add startup latency; this change avoids unnecessary repeat
uploads.

## Model Used

OpenAI GPT-6 (Codex), with reasoning, code execution, browser
automation, and tool use. The exact hosted model ID and context window
are not exposed in this task.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-12 19:41:15 -05:00
Devin FoleyandPaperclip 1c4bcff2b1 fix(test): await issue lock before retry race assertions (#13273)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Retry decisions must respect changes to an issue owner.
> - Database tests verify this with two concurrent transactions.
> - The test started the competing operation before its fixture held the
lock.
> - That race can fail a correct source verification run and block
deployment.
> - This PR waits for lock acquisition before starting the competing
operation.

## Linked Issues or Issue Description

Refs #13257 for the deployment verification work that exposed this test
race. No duplicate fix was found.

**What happened?**

[Cloud verification job
103424158921](https://github.com/paperclipai/paperclip/actions/runs/34648268409/job/103424158921)
failed because the test did not observe a concurrent issue-row lock
waiter. The fixture and retry operation both started without an ordering
guarantee.

**Expected behavior**

The fixture must hold the issue lock before the competing retry
operation starts. The test must still prove that the retry waits for the
lock and observes the reassignment.

**Steps to reproduce**

1. Add a temporary 50 ms delay before the fixture acquires the issue
lock.
2. Run the promoteOrCancelDueRetry issue-lock test.
3. The original fixture fails with the same missing-waiter error as CI.
4. The synchronized fixture passes with that delay. The delay is not
part of this PR.

## What Changed

- Separate fixture readiness from its transaction completion promise.
- Wait for readiness at both callers before starting the retry decision.
- Propagate transaction failure during setup through Promise.race.

## Verification

- The original fixture fails under the temporary delayed-lock probe. The
fixed fixture passes the same probe.
- All 31 tests in
server/src/modules/run-dispatch/adapters/postgres.test.ts pass after
removing the probe.
- The real concurrent waiter, lock-order, and reassignment assertions
remain intact. No timeout was increased.
- Full local typecheck and build pass (221s and 50s). The full local
test command stopped in its server phase after 10,610 passes, 65 skips,
and 14 failures: 13 existing macOS skill-cache rename/permission
failures and one unchanged Telegram test assertion. The Telegram test
passes in a focused rerun. Later local phases did not run after that
failure. Current-head Greptile is 5/5 with no findings. All 32
current-head checks pass, including full Linux typecheck, build, native
verification, server suites, browser suites, and canary packaging. The
unchanged GitHub browser mock assertion passed its single failed-shard
retry.
- git diff --check passes.

## Risks

- Only test synchronization changes. Production database behavior is
unchanged.
- Awaiting the transaction itself during setup would deadlock the test.
Returning the completion promise inside an object avoids that problem.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — all 31 database adapter
tests pass; full local verification limits are disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 15:29:16 -07:00
DottaandPaperclip b1efd65edc fix: continue interrupted task conversations with bounded retries (#13237)
## Thinking Path

> - Paperclip manages AI agents and their tasks.
> - A task can outlive a provider process or a server restart.
> - Legacy recovery treated unknown tool outcomes as a permanent
execution hold.
> - That hold could also reject a later user message.
> - A conversation turn can use prior history without replaying prior
tool calls.
> - This pull request lets supported conversation adapters continue
within the existing retry budget.
> - Users can send a new message after automatic attempts stop.

## Linked Issues or Issue Description

**What happened?**

A server restart could interrupt a local ACP run and leave its task
behind a permanent recovery hold. A later user message could be
cancelled before the provider answered. The immediate recovery path
could also create a successor outside the durable failure counter.

**Expected behavior**

Continue with a bounded new conversation turn. Preserve a compatible
provider session or use full task context when it is unavailable. Do not
replay recorded tools. When automatic attempts stop, allow a new user
request through the normal execution gates.

**Steps to reproduce**

1. Start a task with a local conversation adapter.
2. Restart the server while the provider is working.
3. Let the previous run become interrupted.
4. Send a follow-up message and observe the recovery hold on the old
behavior.

Related work: Refs #13075 for durable task recovery. Refs #12946 for
retry-limit and checkout-lock handling. This change routes conversation
recovery through the existing bounded scheduler.

## What Changed

- Mark supported local conversation failures for continuation. Keep
native-runner and non-conversation recovery rules.
- Carry an interruption notice into the next turn. Retain stopped ACP
session history even when a write outcome is unknown.
- Clear unavailable ACP sessions so the next bounded attempt can use
full task context.
- Route immediate failure recovery through the same durable scheduler as
process-loss recovery. Release only the predecessor checkout when its
retry takes ownership.
- Retire obsolete conversation holds using immutable run evidence, in
bounded batches with an activity record. Preserve outcome evidence and
do not wake historical tasks.
- Block actual admission and Resume while a predecessor process or
environment lease is still active. Keep the original interruption notice
after a rejected wake. Preserve the upstream blocked-wake waiting
contract: bounded retry planning can happen during cleanup, while
deferred messages and execution remain gated.
- Add subprocess and database regression tests. Update the execution
contract.
- Add the current thread-status field to the native recovery provider
fixture so its damaged-journal test reaches the intended boundary.
Tolerate an already-exited fixture process during test cleanup while
still asserting both processes terminate.

## Verification

- Workspace typecheck passed: `pnpm -r typecheck`.
- Build passed: `pnpm build`.
- Module boundaries passed: `pnpm check:module-boundaries`.
- Focused tests passed: 293 recovery/session/dispatch tests, 66 retry
and response-gate tests, and 37 native-session tests. Some suites
overlap.
- Tests cover interrupted writes, missing sessions, concurrent retries,
restart persistence, pending questions and approvals, execution gates,
and historical holds.
- Built the Rust test executables with `pnpm --filter
@paperclipai/paperclip-runner build:rust` for native-runner
verification.
- Full Vitest coverage verified locally using the repository’s general
and serialized shards, with focused reruns for failures and files not
reached after a shard stopped. The ownership-gate regression is fixed
and the complete affected server shard passes (1,390 tests). Local
parallel runs also hit temporary-directory, resource, and timing
failures; those suites pass with canonical temporary paths and
sequential reruns. No test timeouts were increased.
- Final merged-branch regression run: 577 tests pass across process
recovery, retry scheduling, liveness, durable chat, wake-queue
application/adapter, dispatch, continuation, native sessions, and task
chat. Earlier focused verification also passed 19 native control tests.
Token gates and whitespace validation pass.
- Browser verification passed all three ACP Stop/continue/pause
scenarios, including a rerun after merging the upstream waiting
behavior: `PAPERCLIP_E2E_PORT=3397 pnpm test:e2e
tests/e2e/acp-stop-continuation.spec.ts`. The interrupted-write case
verifies that follow-up completes without a repeated write.

- Final-head [CI run
34625037394](https://github.com/paperclipai/paperclip/actions/runs/34625037394)
passed on `06ac4bd9d150f8b209a96e5fd609c696958794a0`: all 31 reported
checks are green, including server/workspace suites, all browser shards,
native runner verification, build, typecheck, release dry run, and
aggregate gates. The two conditional Storybook checks were skipped.
Greptile reviewed this exact commit at 5/5; all review threads are
resolved.

## Risks

- A new model turn can choose to repeat an action. Paperclip does not
replay recorded tool calls and does not certify unknown action outcomes.
- Conversation adapters now stop after their retry budget instead of
requiring action reconciliation. Explicit Stop, pause, dependency,
approval, budget, and ownership gates remain in force.
- No schema migration or dependency changes. Historical holds are folded
without changing task status or waking work.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and test execution. The session does not expose a more
specific model build ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 12:16:04 -05:00
DottaandPaperclip 889947c238 feat: add experimental native chat connectors (#13038)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People also ask agents for work in their existing chat tools.
> - Each external conversation needs one task and a current authorized
source.
> - Retries, Stop, and provider failures must not duplicate work or
expose private data.
> - The first chat PR establishes the opt-in provider and data
contracts.
> - This PR adds experimental channel integration and its durable
control plane.
> - Users can request work from connected channels and inspect delivery
in Paperclip.

## Linked Issues or Issue Description

Refs #13100 and #13092. This is the second of exactly two chat PRs.
Foundation #13100 is merged and changed 143 files. Runner prerequisite
#13092 is also merged. This PR changes 400 files against master, below
the 500-file review limit. It contains no wireframe images or HTML
galleries.

## What Changed

- Add native Slack, GitHub, Microsoft Teams, Telegram, and Discord chat
connections. Keep chat disabled unless the operator enables experimental
chat connectors. Preserve the production GitHub tool connection and its
normal setup path.
- Bind each provider bot identity to one immutable Paperclip agent. Bind
each admitted external conversation to one task. Paperclip owns tasks,
runs, permissions, and audit records.
- Add durable admission, per-conversation queues, questions, task
controls, progress, final replies, images, files, and delivery receipts.
Board comments remain internal unless explicitly sent to the channel.
- Check current identity, provider reach, resource access, credentials,
runtime generation, and exact source before provider effects. Keep
private responses private. Never send raw reasoning, private logs,
credentials, or tool arguments.
- Hold uncertain sends for explicit audited resolution. Make Board
Send-to-channel atomic and idempotent. Keep reconnect and setup
credentials in Paperclip secret storage.
- Preserve current native-runner authority across retries, lost
acknowledgements, and recovery. Keep immutable input and completion
contracts separate from newer user input. Receipt reconciliation cannot
launch a provider.
- Reconcile chat close/new ordering and provider-effect lock order.
Audit resource access changes in the same transaction. Submit only the
selected resource from each UI toggle so stale pages cannot undo
unrelated access changes.
- Drain Codex stdout before certifying process exit. Bound the drain
with the existing shutdown grace. Preserve observed terminal authority
without treating an undrained process as successful or reusable.
- Incorporate master `018ca5da` with its ACP Stop, mobile task layout,
runner packaging, and official lock changes. Preserve dedicated
chat-answer continuations in both directions when ordinary queued
comments are adopted after Stop.
- Fence late adapter readiness behind an earlier Stop for the same run.
Preserve verified cleanup for registered adapters. Handle single Stop,
agent pause, duplicate Stops, and failure release without creating a
false cancellation receipt.
- Incorporate master's `6dd48cad4` wake-queue extraction. Preserve exact
failed-chat retry authorization and lineage, retired question-source
suppression, and the block on generic recovery that would discard the
admitted source. Fresh deferred input retains its separate promotion
path.
- Incorporate master `2a05b5ed3` and its queue-admission extraction,
simplified transaction ports, and separate runner CI job. Preserve exact
durable receipts, actor separation, and dedicated-answer isolation
through the new module. A failed receipt insert rolls back the
accompanying deferred-wake merge.

## Verification

Current head: `afe19299d06253cb628eb398e91d1200ea9f412a`, incorporating
master `2a05b5ed3457ea33efd6895520447d1d97fe98d8`. The conflicts are
resolved. This successor fixes two test-harness boundaries exposed by
CI: per-case route-module preparation and actual durable-save completion
before intentional runner termination. Production code and all existing
test/turn deadlines are unchanged. [Exact-head Greptile
review](https://github.com/paperclipai/paperclip/pull/13038#issuecomment-5587250594)
is **5/5**, completed September 10 at 13:20:55 UTC, with no actionable
findings or open review threads. [Fresh exact-head
CI](https://github.com/paperclipai/paperclip/actions/runs/34481724341)
passes **all 24 jobs**, including Build and both required aggregates.
Normal exact-head guarded merge was attempted and rejected by the
remaining branch approval policy: CODEOWNER review is required and no
human approval is present. Normal **squash auto-merge is enabled** as of
September 10 at 13:36:26 UTC. Requested CODEOWNERS have been notified;
no approval bypass or self-approval was used. Earlier-head results below
remain historical evidence, not qualification of this successor.

- Final exact-head Linux evidence: 995/995 chat integration cases; 36/36
agent-skills routes; 35/35 runner live-session cases, including real
process kill/resume; 1948 runner Vitest cases with three existing
benchmark/platform guards; 870/870 API-authority cases; and 104 browser
cases with four existing optional skips. Rust, conformance/replay, full
repository build, typecheck, canary, all server/workspace shards, and
both required aggregates pass with normal CI concurrency. Earlier failed
attempts remain recorded below.

- Latest test-only qualification: 141/141
route/permissions/authentication cases pass in separate cold forks, with
plain server types and independent review clear. The real-runner suite
passes 35/35, with plain runner types and independent review clear. A
controlled premature-save acknowledgement fails as expected; matching
ownership/effect/process evidence, rejected saves, real turn outcome,
test abort, and pre-kill liveness are covered. No local reproduction of
the original CI scheduling failure is claimed. The preceding [CI
run](https://github.com/paperclipai/paperclip/actions/runs/34479680858)
passes 21/24 jobs, including all 995 Linux chat cases and browser
aggregate (104 passed, four existing optional skips); only Build, the
skills serialized shard, and the required verification aggregate fail.
Its exact-head Greptile review was 5/5. Both failed job logs are
retained.

- Final fixture qualification: all eight focused Discord cases and all
995 chat integration cases pass. The exact modal statement/PID is
observed before taking the real connection lock; the test then proves
its actual blocking relationship before mutation. Original SQL
execution, provider behavior, negative assertions, and 1s/15s timeouts
remain unchanged. Independent review is clear and test/production hashes
remain frozen. The preceding [CI
attempt](https://github.com/paperclipai/paperclip/actions/runs/34477184777)
passed 22 jobs, including Build/runner, typecheck, canary, all other
test shards, and browser aggregate (104 passed, four existing optional
skips); the two fixture failures and failed verification aggregate
remain recorded, not relabeled as a pass.

- Current queue-module composition: 308/308 recovery/batching/queue/Stop
tests; 995/995 full chat integration; 89/89 module tests, including real
PostgreSQL receipt-insert rollback; 24/24 workflow/module-boundary
tests; plain server and UI types. All four actual local process/ACP
browser paths pass in 1.4 minutes. Fresh databases, no skips or retries,
stable reviewed source hashes. The initial boundary failure is retained;
its no-op service wrapper was removed without changing recovery context
or weakening the check. An exploratory standalone test-directory
typecheck fails because its new upstream transformation config is not a
standalone typechecking project; standard CI/build does not invoke it,
and no configuration was weakened to suppress those diagnostics.

- The preceding head `e02a63d462ce5d47433b0aeb632bb6fd20aab1ba` passed
[all 24 CI
jobs](https://github.com/paperclipai/paperclip/actions/runs/34436462958)
and exact-head Greptile review at 5/5. Required CODEOWNER review
prevented its normal merge before master advanced again.

- Final extracted-module composition: 307/307 recovery, batching, queue
and Stop-control tests; 995/995 full chat integration; 49/49 module
tests including eight PostgreSQL adapter cases; and 19/19 issue-update
tests. Plain server types pass. All four actual local process/ACP
browser paths pass in 1.3 minutes. Fresh databases, no skips or retries
in these cohorts, frozen source hashes, and independent review clear.

- The preceding head `3e4e1c1c` passes [all PR CI
jobs](https://github.com/paperclipai/paperclip/actions/runs/34415826820),
including Build and required `ci / verify` and `ci / e2e`. Both the
original Rust failure and the previously load-sensitive lineage fixture
pass with unchanged Linux concurrency. Master advanced afterward and
required this reconciliation.
- Final master composition: 448/448 focused UI tests, 186/186 adapter
tests, 24/24 queue/control tests, and 11/11 packaging tests. Plain UI,
server, shared, and adapter types pass. Token gates and diff checks
pass. Independent server and UI reviews are clear.
- Stop-registration regression: both real-service cases fail against
exact `a95` source and pass with the fix. The full corrected
recovery/control suite passes 265/265. Duplicate-owner and failed-Stop
controls also pass. Plain server types pass. The readiness barrier
prevents provider startup without adding an acknowledgment to an already
terminal run.
- Final qualification strengthens terminal-field equality and repeats
both affected cases successfully on a fresh database. All four actual
local process/ACP browser paths pass again in 1.3 minutes, without skips
or retries. The final screenshot shows Cancelled, a paused subtree,
retained input, and no error toast.
- Two new actual-service regressions fail before the merge fix. They
prove that queued-comment adoption could consume a dedicated chat answer
or add unrelated input to that answer. The fixed four-case cohort
passes, including ordinary upstream continuation and adapter Stop
controls. Full recovery passes 257/257. All four actual local
process/ACP Stop browser flows pass in 1.4 minutes, without skips or
retries, on a fresh database.
- The unchanged runner artifact was qualified with 171/171 transport
tests, 870/870 API-authority tests, conformance 1/1, and replay 11/11.
Six controlled reader tests prove the exit/drain repair. Its local
serial Rust workspace passed 546 top-level cases plus two invoked
helpers; the later passing Linux CI supplies default-concurrency
evidence.
- Prior exact-source full chat integration passes 995/995. Settings
regressions cover concurrent stale pages, 501 destinations, pending
state, rejected updates, and explicit retry. These deterministic tests
do not prove live provider behavior.
- Retained failed attempts and their causes are in the [qualification
log](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-chat-queue-and-webhook-repair.md).
The first merge adapter run timed out while macOS slept for 290 seconds.
Its unchanged repeat passed with a temporary sleep guard. No assertion,
deadline, or CI gate was weakened.

Review commands include `pnpm --filter @paperclipai/server exec vitest
run src/__tests__/heartbeat-process-recovery.test.ts
src/__tests__/issue-queued-comments-routes.test.ts` and `pnpm exec
playwright test --config tests/e2e/playwright.config.ts
tests/e2e/acp-stop-continuation.spec.ts`. Database suites require fresh
disposable databases. See the [browser
runbook](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-04-chat-adapters-browser-e2e-runbook.md)
for provider setup and separate live acceptance steps.

## Risks

- This remains experimental. Deterministic tests and bounded live
evidence do not establish every provider feature, tenant, permission
layout, or media shape. Teams work-tenant qualification is still open.
- Failed and uncertain provider effects remain visible and can require
operator action. A transport receipt does not prove recipient
visibility.
- Native controller and runner artifacts must remain compatible.
Preserve lease ownership, terminal authority, source binding, and
quarantine during future changes.
- Access and audit rows commit together, but activity notifications
remain best-effort. This is not a new durable event outbox.
- The PR operation does not deploy a live server, replace its runner, or
change provider permissions. Remaining live qualification is documented
in the [temporary
handoff](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-open-qualification-followups.md).

## Model Used

OpenAI Codex assisted with implementation, tool execution, testing, and
review. The work records `gpt-6-astra` assistance. The environment does
not report a context-window size. No private reasoning traces are
included.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-10 10:06:45 -05:00
DottaandPaperclip 018ca5daaf fix: verify ACP Stop and preserve safe continuation (#13119)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task controls coordinate provider execution and queued user
messages.
> - Stop could finish before an embedded ACP provider stopped its tools.
> - A later request could be held for reconciliation without a clear
task response.
> - A restored provider could also retain the stopped run's API
credential.
> - This pull request verifies provider termination and preserves safe
session continuation.
> - Operators can continue known-safe work and see why uncertain work
cannot start.

## Linked Issues or Issue Description

**What happened?**

Stop could leave an embedded ACP provider running. A queued follow-up
followed by “go” could fail before it reached the provider. Task chat
could show a generic missing-response message. Even a restored session
could use the previous run's credential and fail its task update.

**Expected behavior**

Stop waits for confirmed provider termination. A later explicit wake
continues the same compatible session only when recorded actions have
known outcomes. It carries pending comments and the current run's
environment. Uncertain actions retain a visible reconciliation hold.
Composer Stop preserves the existing pause rule: conversation can
continue while paused, but task work requires Resume.

**Steps to reproduce**

1. Start an embedded ACP task.
2. Send a second request while the provider is running.
3. Interrupt the run, then send “go”. Also test composer Stop followed
by Resume work.
4. Check that the request is delivered once and that the provider can
complete the task through the current run's API credential.
5. Repeat with an unfinished write. Confirm that the write stops and
that further execution stays blocked with a visible reason.

**Paperclip version or commit**

Built from source on master at `3bc60dd8b` plus this branch.

**Deployment mode**

Local source build with an isolated embedded PostgreSQL instance.

Refs #11183. Refs #12552. Those changes address recovery after operator
cancellation. This change also covers embedded ACP termination, session
proof, pending-comment delivery, and task feedback.

## What Changed

- Propagate Stop into embedded ACP and wait for bounded adapter cleanup
and provider exit. Retain the actual ChildProcess object for forced
termination on all platforms; never signal a recycled numeric PID.
- Preserve interrupted checkpoints only for acknowledged, local,
persistent sessions with settled reads or no tools. Keep writes,
incomplete actions, and forced termination blocked.
- Restore the same compatible provider session with the current run's
environment. Reject fresh-session fallback for an interrupted
checkpoint.
- Adopt pending comments on the next explicit wake. Stop alone does not
dispatch them.
- Share the execution-blocker rule across dispatch, Resume, and task
detail. Show Stopped or Couldn't start with the recorded reason. Resolve
the stopped agent for the run link, including reviewer runs.
- Keep execution reconciliation holds intact when generic recovery sees
queued comments or healthy child tasks.
- Add process, service, component, and browser regression coverage. Fix
disposable database cleanup and React test settling exposed by the full
suite.

## Verification

- Passed `pnpm -r typecheck`, `pnpm build`, and `pnpm
check:token-gates`.
- Passed all three `acp-stop-continuation.spec.ts` browser journeys.
They use an actual ACP child process and require task completion through
the agent API.
- Passed 165 adapter execution, operator-stop, and child-process control
tests, 17 queued-comment route tests, and 65 tests in the two adjusted
UI suites. Earlier focused recovery, heartbeat, and task-control tests
also passed.
- Manually used the browser to queue a request, Stop, send “go” while
paused, and Resume. The same session answered once and moved the task to
Done with the current run's credential.
- Manually interrupted an unfinished write. Its file size stayed fixed
for five seconds. “Go” showed the reconciliation reason and did not
start another provider prompt.
- Separate live Claude ACP smoke checks confirmed that Stop ended a
disposable local write and that a no-tool interruption could resume the
exact provider session. The browser fixture does not call Drive or
another external app.
- Passed all 5,615 UI tests and 3,090 other workspace tests. The CLI and
general server groups pass with targeted retries: two transient server
failures passed together on retry, and two embedded-database startup
failures passed after removing abandoned shared-memory segments from
this task's completed browser fixtures. All 144 serialized server suites
completed, with 2,189 tests passing after two transient HTTP socket
failures passed on retry.
- Passed all 135 heartbeat process/recovery tests, including a
deterministic regression that failed before the recovery-sweep fix.
- Passed 18 dispatch integration tests, including stopped-reviewer
links, company boundaries, and malformed run IDs.
- Greptile is 5/5 on `7dd170d83`, with zero unresolved review threads.
The security scan and all required CI gates pass for the same commit.

## Risks

- Safe continuation depends on complete tool reporting and a restorable
local provider session. Unknown outcomes remain blocked and require
reconciliation.
- Provider cleanup can take time. A timeout does not grant replay
permission.
- The change adds optional adapter context fields and an optional issue
projection. It does not change the database schema or require a
migration.
- Test cleanup truncates company data only in a disposable test
database.

## Model Used

OpenAI GPT-6, running as Codex with repository tools, code execution,
and browser interaction. The runtime does not expose a more specific
model deployment ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-09 22:06:06 -05:00
DottaandPaperclip 35fdc0c66b fix: make task recovery durable and preserve current requests (#13075)
Make task recovery durable and preserve the latest user request across native and legacy continuations. Keep routine recovery quiet and prevent replay when action outcomes are uncertain.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-09 09:14:25 -05:00
Nicky LeachandPaperclip f65991a5f1 refactor(server): move scheduled-retry and queued-run dispatch into a run-dispatch module (#12920)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server heartbeat service dispatches scheduled retries and queued
runs.
> - The service kept policy decisions and database writes in one large
file.
> - This layout made policy branches harder to test and transaction
boundaries harder to inspect.
> - This pull request moves the policy rules and database transactions
into a run-dispatch module.
> - The benefit is a smaller service, pure policy tests, and clear
transaction ownership.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The server heartbeat service promotes scheduled retries and cancels
stale queued runs.

**Subsystem affected**

server/ — REST API and orchestration services.

**Current behavior**

The heartbeat service contains the policy rules and the database writes
for these dispatch paths.

**Proposed behavior**

A run-dispatch module owns pure policy functions and semantic database
transactions. The public service contracts stay unchanged.

**Reason and benefit**

The new layout separates branch rules from database effects. It makes
each policy branch easier to test and keeps each operation’s row writes
in one transaction.

**Breaking changes**

None. The public service contracts stay unchanged.

## What Changed

- Move scheduled-retry promotion and queued-run staleness rules into
pure functions.
- Add table-driven unit tests for each policy branch.
- Move promotion and cancellation writes into semantic transactions.
- Keep row locking, company isolation, and post-commit effects
unchanged.

## Verification

- `node scripts/check-module-boundaries.mjs` passes.
- `tsc --noEmit` from `server/` reports no errors.
- The focused server test command passes 228 tests in six files.
- Full pull request CI passes.
- Greptile reports 5/5, and all review threads are resolved.

## Risks

The main risk concerns changed transaction boundaries in scheduled-retry
promotion and queued-run cancellation. The focused tests retain coverage
for locking, transactionality, company isolation, and transport
contracts. The public service contracts do not change.

## Model Used

Codex, GPT-5, with code execution and tool use. The context window is
not provided by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I have addressed all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-08 11:00:55 -07:00