Commit Graph
4 Commits
Author SHA1 Message Date
Nicky LeachandPaperclip a924d2e2c7 Persist portable Runner ownership through controller recovery (#21066)
## Thinking Path

> - Paperclip delegates remote Runner execution to environment plugins.
> - A plugin can move a session to another task allocation.
> - The Core controller can lose an ownership reply or restart before
the Runner starts.
> - The source and target need a durable record of the same request.
> - This change binds restart and attachment to that request and the
original unlaunched task lease.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Recovery of plugin Runner sessions on replacement compute.

**Current behavior**
Core cannot replay a session ownership request from saved lease state
during startup.

**Proposed behavior**
Record the operation on both leases and check the matched ownership
receipt before attaching. Preserve the original pre-admission lease
across controller restart.

Managed environment configuration is in
[#21065](https://github.com/paperclipai/paperclip/pull/21065). Related
warm sandbox cleanup is in
[#20065](https://github.com/paperclipai/paperclip/pull/20065). This
layer preserves the same-sandbox fallback from
[#21001](https://github.com/paperclipai/paperclip/pull/21001). Workspace
placement follows in
[#20743](https://github.com/paperclipai/paperclip/pull/20743).

## What Changed

- Add the optional `sessionHandoffVersion` capability and the ownership
task operation.
- Save matching ownership claims and fence source checkpoint access.
Replay a pending request on the original target.
- Restore a portable session using the durable claim and the current
credentials for the same account. Start a fresh conversation when the
provider reports an unavailable checkpoint, or when Core's persisted
compatibility decision selects a different session after transfer. The
latter requires an unlaunched target; subsequent recovery must match the
fresh task's saved identity. Both paths retain the workspace transfer
and source fencing.
- Reuse the original unlaunched task lease after hard or graceful
controller restart. After a move has attached, validate the surviving
Runner through the saved claim rather than requiring its logical lease
ID to equal the new physical lease.
- Keep older retained leases without a saved account identity on the
existing resume path.
- Apply the existing supported MCP-only compatibility rule to warm
recovery and predecessor accounting. Persist the capability discovered
at admission and reuse it after restart without repeating discovery.
- Keep predecessor-retirement fallback scoped to the same physical
sandbox. A refusal after a portable move preserves the session instead
of requesting a replacement that the plugin driver cannot enforce.

## Verification

- On `1eb527be27`, server typecheck (including Runner build and SDK
dependencies) passed.
- Independent final verification passed 168 tests across portable source
recovery, session resume, controller recovery, and task authority; two
existing tests were skipped. The full portable source integration suite
passed 64 tests.
- The new integration follows confirmed ownership through an
incompatible instruction change, persisted fresh admission, target-only
prepare/auth/submit without the source checkpoint, and reattachment
after controller replacement. It also rejects unpersisted session
selection, ambiguous target leases, and incorrect saved identities.
- Before this final change, independent verification passed 1,667 tests
across executor, session resume, recovery usage, remote warm recovery,
runtime selection, and runtime execution; the environment lifecycle
suite passed 208 tests.
- Existing regressions cover previous-release account metadata,
supported and unsupported MCP changes, provider/instruction
incompatibility, and admission capability reuse without another
discovery call.
- On `51d7eec373`, all four S3 multipart throttle tests passed
independently. CI exposed an existing timing-sensitive test twice; its
20 ms overlap assumption now uses synchronized retries and an event-loop
yield. Production upload code is unchanged. Temporary concurrency caps
of one and eight both fail the retained exact-four assertion.
- Exact SDK source-pin checks passed for both stack heads. The upper
#20743 code patch remains unchanged; its documentation preserves both
layers' behavior.

## Risks

- The handoff protocol is opt-in. Existing plugin leases without
portable ownership metadata keep the previous release's resume path.
Live attachment still requires end-to-end qualification. An unverified
claim blocks startup. Git workspaces prepared on a portable allocation
also need the workspace placement layer.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 10:43:48 -08:00
Dotta a3e7e63330 Guard Native recovery with process-space proof and explicit containment (#19813)
## Thinking Path

> - Paperclip manages agents and their work.
> - Recovery must preserve execution ownership and accepted results.
> - A saved PID proves nothing outside its recorded process namespace.
> - Uncertain unfinished work must stay held until its original owners
stop.
> - Operators need a source-bound containment action and a separate
ordinary Retry.
> - Accepted results must use the existing finalizer without another
provider turn.
> - This PR keeps those paths safe, recoverable, and quiet between real
obligations.

## Linked Issues or Issue Description

Refs: #19713

Refs: #19591

Recovery could use a local PID to release or replace a controller from
another process namespace. Connected-chat Retry also rejected successful
containment on a task in `todo`. The new containment action used a
broader permission than the existing owner-stop action. A later review
found that result-less guards also rejected the existing accepted-result
confirmation action.

## What Changed

- Record and verify the controller's kernel/PID namespace. Keep unknown
and foreign unfinished ownership held. Preserve independent durable stop
evidence and the existing remote lease fences.
- Use the shared accepted-result ownership reconciler. Bind settlement
to the observed lease token, generation, boot, PID, namespace and
process birth. Retain finalizer, cleanup, current-controller, live-owner
and remote exclusions.
- Share the remote-owner reader across recovery consumers. Scope runner
leases to the same company, task and run. Do not make local PID probes
for remote accepted-result ownership.
- Gate original-host containment on `runtime:confirm_owner_stop`. Bind
it to the current task, source, assignee, original expiry, lease and
Stop state. Audit separate controller/provider containment and reviewed
action outcomes.
- Keep containment separate from ordinary Retry. Use one canonical
committed receipt and the normal admission, budget, workspace and
lineage checks. Preserve the recorded external-action outcomes.
- Keep the controller-only confirmation actionable for both result-less
work and accepted results. Result-less work gets its next source-bound
attention in the same transaction. Accepted results keep their original
result and use normal shared settlement/finalization.
- Keep identity inspection quiet. Preserve the original foreign-owner
expiry and use the existing scoped finalizer in accepted-result recovery
controls.
- Update the recovery card, validators, execution projection, docs and
access maps. Reuse existing canonical columns; add no migration or
ownership backfill.

## Verification

Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`.

**The current fix has a causal red/green check.** On the test-only
revision, both foreign-local stopped/fenced controller cases failed
because recovery probed saved runner, process-group and provider PIDs in
this host's namespace. This measured the wrong probes; it did not run
duplicate providers. After the fix, the same two cases passed every
no-probe, no-claim, no-controller-stamp, no-stop-receipt and
unchanged-authority assertion. Three existing remote preparation, grace
and fenced-controller progress cases also passed. Total: **5 passed, 0
failed;170 filtered rows excluded**. All three normal affected App
TypeScript programs passed on this head (660,681 and428 roots).

The fix separates a controller wait from execution stop proof.
Independent live/grace evidence still schedules recovery. A
stopped/fenced foreign controller can pass this namespace guard only for
remote execution, which keeps its later checkpoint and never-launched
checks. Foreign local execution stays held before runner or provider PID
inspection.

Earlier source-scoped results remain evidence, not fresh executions on
the current head:

- On `c86dd741`, the complete failed-owner file passed **25/25**, with
no skips or filters. Empty scans wrote no controller stamps. Foreign
ownership kept its original expiry. Normal returned-claim settlement and
attention at the exact deadline passed.
- On `c86dd741`, **15 selected accepted-result/restart controls
passed**. The81 filtered rows are excluded. Both completion cases used
the existing scoped finalizer and kept their accepted result without
another provider turn.
- Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334`
plus `c86dd741`) passed the complete **17-case maintenance-server
file**. This includes the original90-real-second quiet observation and
two qualified hourly windows. The quiet case took118,918ms. The four
jobs that failed on the earlier publication all passed on that synthetic
merge.
- On `6243b32c`, the complete ownership-report file passed **29/29**.
Native NULL namespace stayed unprovable and generic compatibility
remained. Three selected recovery controls passed;170 filtered rows are
excluded.
- On `f81f2c3c`,356 business cases passed: whole ownership-route221,
recoveryUI70, owner-proof38 and controller-confirmation8; selected
accepted-result authority18; and one real connected-chat containment,
separate authorized Retry, useful Native result and downstream reply
case. The2,024 filtered rows are excluded. A separate earlier
accepted-result causal case kept the original result and completed after
restart without replay.

Normal Shared exports, App programs, UI typecheck, token gates and map
checks passed on their recorded revisions. Unchanged DB, adapter, Eval
and Runner inputs/outputs have explicit carry proofs. The clean normal
merge with fetched master `8fe524f7` preserves all31 exclusive owned
paths and14,827 unrelated master entries. The nine overlapping paths
were reviewed, including the upstream unidentified-run stop action.
Fresh CI must validate that actual merge composition.

PG cases used existing external PG17, fresh owned databases, pools at
most4, one worker and no SQL observer during cases. Registered clients
and recorded process groups closed. Independent final SQL audits found
zero campaign sessions before normal database cleanup. Original failing
fixtures remain retained.

Original failures are preserved. One new test first failed compilation
due to a missing getter import; an import-only correction passed its
full owning programs before the causal red/green check. A local quiet
attempt stopped in the normal prerequisite hook before any business case
and its collector rejected two normal build markers. The later17-case CI
pass establishes quiet recovery; it does not turn that failed local
collector into a pass. An earlier useful case passed before its private
cleanup collector failed on an owned read-only directory; normal cleanup
completed after all owners closed. Earlier runner-shutdown cancellation
and a separate private whole-suite failure remain unexplained.

Current-head checks are complete:57 success,1 neutral and3 skipped,
including the successful aggregate CI gate. The legacy status passed.
Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this
head. All seven review threads are resolved. Full recursive typecheck,
the full test suite and release build were not repeated locally. No
result here proves deployed suspension, a live provider or termination
on another host.

## Risks

- Unknown physical ownership can require an owner/admin action. A false
containment statement can be unsafe.
- New work, reassignment, Stop, renewed authority, accepted results or
pending cleanup can invalidate an action. The operator must refresh.
- Controller confirmation does not prove provider/tool containment.
Completed external actions must not be replayed.
- Generic legacy and child-process compatibility remains separate from
the strict Native controller policy.
- Existing saved state and canonical schema are preserved. Permanent
safety refusals remain held with their normal named recovery path.

## Model Used

OpenAI `gpt-6.1-sol` agents implemented and tested the changes with
reasoning, repository tools and code execution. A GPT-6-based Codex
managing agent reviewed and integrated them. The host does not expose
the manager's exact build or the context-window sizes.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-09 12:06:11 -06:00
DottaandPaperclip d8c1b9d4b5 fix(runtime): reclaim failed remote checkpoint retries (#18877)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native recovery must preserve the original execution and provider
state.
> - A failed remote run can retain its provider PID after heartbeat
releases the task execution lock.
> - Normal resumption excludes that PID, while restart recovery rejects
the released lock.
> - Recovery must reclaim that exact execution only after current
admission and durable ownership checks pass.
> - Failed retries and admission holds need separate finite budgets.
Neither can prove a provider stopped.

## Linked Issues or Issue Description

**What happened?**

A failed remote native run remains in retryable failure without a task
execution owner. Normal resumption and restart recovery both reject it
before inspecting the preserved remote provider. The current remote
workspace reference format also misses the due-retry routing path.

**Expected behavior**

A due exact-checkpoint retry reclaims its original task execution only
when current ownership and safety gates permit it. Newer task input
prevents an old claim. Lost or ambiguous provider state still prevents
replay or replacement. Ineligible recovery eventually becomes a quiet
manual reconciliation obligation.

**Steps to reproduce**

1. Start a remote native run and persist its provider checkpoint and
encrypted Runner authority.
2. Fail with exact-checkpoint recovery and a retained remote PID.
3. Release the task execution lock at heartbeat completion.
4. Run ordinary resumption and restart recovery after the retry
deadline.
5. Observe that both reject the same eligible run before remote
inspection.

**Paperclip version or commit**

Reproduced from source at `d59791828482`; integrated with upstream
`db9be4c5a6`.

**Deployment mode**

Remote workspaces with PostgreSQL. Local regressions use disposable
PostgreSQL and encrypted checkpoint persistence.

Related public repairs #18846, #18854, #18860 and #18841 cover finished
per-turn owners, pre-provider failures, unstarted follow-ups and storage
migration. They do not reclaim this failed run's released execution
lock. Merged #18879 also adds scheduled-resume lock retention and
reclaim. This PR keeps that compatibility and requires encrypted Runner
authority, canonical source ordering and a charged failed retry before
reclaim. Its scheduled-resume regression now uses a real encrypted
authority and retains a snapshot-only refusal control.

## What Changed

- Share normal admission checks and one guarded execution-binding
restore/event helper between normal and remote resumption.
- Keep agent-name normalization in a module with no imports. Preserve
heartbeat's existing export. Native admission no longer loads heartbeat
orchestration for a string key.
- Recheck company, assignment, agent eligibility, Stop, pause, approval,
dependencies, execution and checkout ownership before reclaim.
- Reject newer task input using PostgreSQL creation-time and ID
ordering, preserving timestamp precision.
- Require the recorded remote target, encrypted Runner authority, exact
checkpoint identity and existing controller fence.
- Commit task binding, controller lease, run status and recovery event
atomically. Recognize both remote workspace reference formats.
- Charge failed reattachments against the existing
three-provider-attempt budget, including retained-lock crash windows.
Active controller adoption preserves its charged attempt.
- Use the existing five-retry/30-minute system-recovery policy for
admission holds. Ended tasks and newer sources exhaust it immediately.
Preserve PID, checkpoint and task bindings, suppress resource
settlement, and assign manual reconciliation with executor stop/state
explicitly unverified.
- Register recovery intent before deadline writes; preserve live/future
deadlines quietly and roll back zero-row writes. Record changed hold
reasons once and terminal dispositions separately.
- Keep historical provider-budget exhaustion behind current
admission/source/controller/authority checks. Make an existing
orphan-cleanup fixture explicitly due.
- Include exact plugin task leases that have no workspace reference.
Reuse every reclaim and retry-budget guard, with a bounded
company/task/run lease projection.
- Share strict manual ownership proof across stale-lock cleanup,
adoption, queue release and wake admission. Require termination receipts
for every exact remote lease before normal cleanup can release either
binding.
- Preserve governing Stop through the existing audited cancellation
path. Its acknowledgement does not release an unverified provider.
Ordinary Retry can wait for exact Stop and file preservation, then queue
one continuation.

## Verification

- Current head is `5d7baf74aa89d00d7ac6e1493d38a6ce2ae7b7b5`. Its
[current CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/41782198858)
is terminal and clean: 54 successful, three skipped and two neutral
contexts. Required `ci / verify` and `ci / e2e` pass. Greptile is 5/5 on
this exact head with no open findings. No fresh Paperclip 5/5 assessment
is claimed.
- The [prior complete CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/41754365388)
passed on `3e641466d3edcaa6dd5021d979627082d388eb05`, with Greptile and
Paperclip both 5/5 and no open threads.
- All 233 tests across the six affected heartbeat, restart-isolation,
remote-admission, scheduled-resume and session-binding families pass on
production-identical `a5104fa89f9cb493091e000bf21ed3858871ae46` under
Node 24.22.2 and pnpm 9.15.4. They cover both workspace formats and
plugin task leases, competing controllers, exact rollback, Stop,
assignment/company boundaries, attempts, work-intent ordering, finite
backoff and quiet held state.
- The merge preserves #18931's per-run audit-failure isolation and this
PR's unchanged-transition hold. Two initial composition cases failed
because their setup or expected result described the older contract. The
rollback test now expects the exact per-run deferral while retaining
full coordinator equality and no work intent. The routine fixture writes
a real encrypted checkpoint through the normal owner before reaching the
actual routine unique constraint. No production admission guard or
rollback assertion was weakened. A separate agent reviewed this exact
merge and fixture delta without finding a blocker.
- Full recursive `pnpm -r typecheck` and `pnpm build` pass on production
merge `89c616fe45ceff75db450f67fd34224b4df7441d`. Subsequent commits
change only three test files. Both the two-file `a5104fa` and final
`5d7baf` server test-layer typechecks pass.
- Four production-identical `a5104fa` probes pass with the rebuilt
actual Rust Runner, a scripted ACPX subprocess and encrypted PostgreSQL
checkpoints under Node 24.22.2. Workspace-reference and plugin-lease
admission each cover surviving and lost providers. Replacement preserves
one provider launch, one original side effect, the charged attempt and
restored task execution binding. Lost providers reject snapshot recovery
without replacement. This is protocol-boundary evidence, not a live
Claude or hosted provisioning claim.
- Initial `a5104fa` CI found one additional routine integration fixture
missing the persisted `exact_checkpoint_resume` mode. Its encrypted
checkpoint was already present. The exact case fails locally before
repair and passes after recording that mode and expecting the charged
provider attempt. Every exact routine conflict, rollback, checkpoint,
task binding, successor-count and sibling-startup assertion remains.
Final head changes only that fixture. The correct native-server group
passes all nine files: 175 executed cases and 14 container-only skips,
matching the now-successful native-server CI gate.
- The earlier canonical HTTP Retry acceptance passed on the prior
reviewed head with real PostgreSQL, the recovery worker and real files.
Pending Stop protects both bindings and starts no successor. Exact
termination and copyback queue one preserved-workspace continuation. No
browser or live-model outcome is inferred from that test.
- The predecessor's full local `pnpm test:run` failed in an unchanged
Discord Gateway fixture with a 200 ms lease and a 422 initial ownership
error. Focused predecessor and untouched-base controls pass; that
full-run cause remains unproved. The required command was also started
on production-identical `a5104fa`. All six local chat shards passed,
covering 1,850 cases. The duplicate local invocation was interrupted
with SIGINT while the general-server group was still running after this
head's complete CI and the combined repair qualification had passed. The
wrapper exited 1 on cancellation, no process retained its test log, and
no full local pass is claimed. An earlier attempt to select the native
lane through the CI wrapper instead selected local package sharding:
2,631 cases passed and 13 failed. Those separate failures are retained
without causal attribution or unrelated source changes; that run is not
counted as native-server qualification. The direct native group and
current complete CI matrix pass.
- Live Claude Opus 5.5 continuation, hosted plugin provisioning,
deployment and recovery of historical tasks remain unverified. No
original task was retried or mutated for this verification.

## Risks

- Restoring a task binding grants remote inspection; it does not prove
that the provider survived.
- Unknown effects, lost providers and missing semantic results retain
their existing reconciliation requirements.
- Exhausted admission holds preserve execution bindings and resource
evidence for manual reconciliation. They do not authorize a fresh
executor or provider/resource release.
- The source guard is conservative: newer task input prevents recovery
of an older failed source.
- No database migration or provider profile change is required.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, code editing and
shell execution. The runtime does not expose the exact model snapshot or
context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 07:18:07 -06:00
DottaandPaperclip 889947c238 feat: add experimental native chat connectors (#13038)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People also ask agents for work in their existing chat tools.
> - Each external conversation needs one task and a current authorized
source.
> - Retries, Stop, and provider failures must not duplicate work or
expose private data.
> - The first chat PR establishes the opt-in provider and data
contracts.
> - This PR adds experimental channel integration and its durable
control plane.
> - Users can request work from connected channels and inspect delivery
in Paperclip.

## Linked Issues or Issue Description

Refs #13100 and #13092. This is the second of exactly two chat PRs.
Foundation #13100 is merged and changed 143 files. Runner prerequisite
#13092 is also merged. This PR changes 400 files against master, below
the 500-file review limit. It contains no wireframe images or HTML
galleries.

## What Changed

- Add native Slack, GitHub, Microsoft Teams, Telegram, and Discord chat
connections. Keep chat disabled unless the operator enables experimental
chat connectors. Preserve the production GitHub tool connection and its
normal setup path.
- Bind each provider bot identity to one immutable Paperclip agent. Bind
each admitted external conversation to one task. Paperclip owns tasks,
runs, permissions, and audit records.
- Add durable admission, per-conversation queues, questions, task
controls, progress, final replies, images, files, and delivery receipts.
Board comments remain internal unless explicitly sent to the channel.
- Check current identity, provider reach, resource access, credentials,
runtime generation, and exact source before provider effects. Keep
private responses private. Never send raw reasoning, private logs,
credentials, or tool arguments.
- Hold uncertain sends for explicit audited resolution. Make Board
Send-to-channel atomic and idempotent. Keep reconnect and setup
credentials in Paperclip secret storage.
- Preserve current native-runner authority across retries, lost
acknowledgements, and recovery. Keep immutable input and completion
contracts separate from newer user input. Receipt reconciliation cannot
launch a provider.
- Reconcile chat close/new ordering and provider-effect lock order.
Audit resource access changes in the same transaction. Submit only the
selected resource from each UI toggle so stale pages cannot undo
unrelated access changes.
- Drain Codex stdout before certifying process exit. Bound the drain
with the existing shutdown grace. Preserve observed terminal authority
without treating an undrained process as successful or reusable.
- Incorporate master `018ca5da` with its ACP Stop, mobile task layout,
runner packaging, and official lock changes. Preserve dedicated
chat-answer continuations in both directions when ordinary queued
comments are adopted after Stop.
- Fence late adapter readiness behind an earlier Stop for the same run.
Preserve verified cleanup for registered adapters. Handle single Stop,
agent pause, duplicate Stops, and failure release without creating a
false cancellation receipt.
- Incorporate master's `6dd48cad4` wake-queue extraction. Preserve exact
failed-chat retry authorization and lineage, retired question-source
suppression, and the block on generic recovery that would discard the
admitted source. Fresh deferred input retains its separate promotion
path.
- Incorporate master `2a05b5ed3` and its queue-admission extraction,
simplified transaction ports, and separate runner CI job. Preserve exact
durable receipts, actor separation, and dedicated-answer isolation
through the new module. A failed receipt insert rolls back the
accompanying deferred-wake merge.

## Verification

Current head: `afe19299d06253cb628eb398e91d1200ea9f412a`, incorporating
master `2a05b5ed3457ea33efd6895520447d1d97fe98d8`. The conflicts are
resolved. This successor fixes two test-harness boundaries exposed by
CI: per-case route-module preparation and actual durable-save completion
before intentional runner termination. Production code and all existing
test/turn deadlines are unchanged. [Exact-head Greptile
review](https://github.com/paperclipai/paperclip/pull/13038#issuecomment-5587250594)
is **5/5**, completed September 10 at 13:20:55 UTC, with no actionable
findings or open review threads. [Fresh exact-head
CI](https://github.com/paperclipai/paperclip/actions/runs/34481724341)
passes **all 24 jobs**, including Build and both required aggregates.
Normal exact-head guarded merge was attempted and rejected by the
remaining branch approval policy: CODEOWNER review is required and no
human approval is present. Normal **squash auto-merge is enabled** as of
September 10 at 13:36:26 UTC. Requested CODEOWNERS have been notified;
no approval bypass or self-approval was used. Earlier-head results below
remain historical evidence, not qualification of this successor.

- Final exact-head Linux evidence: 995/995 chat integration cases; 36/36
agent-skills routes; 35/35 runner live-session cases, including real
process kill/resume; 1948 runner Vitest cases with three existing
benchmark/platform guards; 870/870 API-authority cases; and 104 browser
cases with four existing optional skips. Rust, conformance/replay, full
repository build, typecheck, canary, all server/workspace shards, and
both required aggregates pass with normal CI concurrency. Earlier failed
attempts remain recorded below.

- Latest test-only qualification: 141/141
route/permissions/authentication cases pass in separate cold forks, with
plain server types and independent review clear. The real-runner suite
passes 35/35, with plain runner types and independent review clear. A
controlled premature-save acknowledgement fails as expected; matching
ownership/effect/process evidence, rejected saves, real turn outcome,
test abort, and pre-kill liveness are covered. No local reproduction of
the original CI scheduling failure is claimed. The preceding [CI
run](https://github.com/paperclipai/paperclip/actions/runs/34479680858)
passes 21/24 jobs, including all 995 Linux chat cases and browser
aggregate (104 passed, four existing optional skips); only Build, the
skills serialized shard, and the required verification aggregate fail.
Its exact-head Greptile review was 5/5. Both failed job logs are
retained.

- Final fixture qualification: all eight focused Discord cases and all
995 chat integration cases pass. The exact modal statement/PID is
observed before taking the real connection lock; the test then proves
its actual blocking relationship before mutation. Original SQL
execution, provider behavior, negative assertions, and 1s/15s timeouts
remain unchanged. Independent review is clear and test/production hashes
remain frozen. The preceding [CI
attempt](https://github.com/paperclipai/paperclip/actions/runs/34477184777)
passed 22 jobs, including Build/runner, typecheck, canary, all other
test shards, and browser aggregate (104 passed, four existing optional
skips); the two fixture failures and failed verification aggregate
remain recorded, not relabeled as a pass.

- Current queue-module composition: 308/308 recovery/batching/queue/Stop
tests; 995/995 full chat integration; 89/89 module tests, including real
PostgreSQL receipt-insert rollback; 24/24 workflow/module-boundary
tests; plain server and UI types. All four actual local process/ACP
browser paths pass in 1.4 minutes. Fresh databases, no skips or retries,
stable reviewed source hashes. The initial boundary failure is retained;
its no-op service wrapper was removed without changing recovery context
or weakening the check. An exploratory standalone test-directory
typecheck fails because its new upstream transformation config is not a
standalone typechecking project; standard CI/build does not invoke it,
and no configuration was weakened to suppress those diagnostics.

- The preceding head `e02a63d462ce5d47433b0aeb632bb6fd20aab1ba` passed
[all 24 CI
jobs](https://github.com/paperclipai/paperclip/actions/runs/34436462958)
and exact-head Greptile review at 5/5. Required CODEOWNER review
prevented its normal merge before master advanced again.

- Final extracted-module composition: 307/307 recovery, batching, queue
and Stop-control tests; 995/995 full chat integration; 49/49 module
tests including eight PostgreSQL adapter cases; and 19/19 issue-update
tests. Plain server types pass. All four actual local process/ACP
browser paths pass in 1.3 minutes. Fresh databases, no skips or retries
in these cohorts, frozen source hashes, and independent review clear.

- The preceding head `3e4e1c1c` passes [all PR CI
jobs](https://github.com/paperclipai/paperclip/actions/runs/34415826820),
including Build and required `ci / verify` and `ci / e2e`. Both the
original Rust failure and the previously load-sensitive lineage fixture
pass with unchanged Linux concurrency. Master advanced afterward and
required this reconciliation.
- Final master composition: 448/448 focused UI tests, 186/186 adapter
tests, 24/24 queue/control tests, and 11/11 packaging tests. Plain UI,
server, shared, and adapter types pass. Token gates and diff checks
pass. Independent server and UI reviews are clear.
- Stop-registration regression: both real-service cases fail against
exact `a95` source and pass with the fix. The full corrected
recovery/control suite passes 265/265. Duplicate-owner and failed-Stop
controls also pass. Plain server types pass. The readiness barrier
prevents provider startup without adding an acknowledgment to an already
terminal run.
- Final qualification strengthens terminal-field equality and repeats
both affected cases successfully on a fresh database. All four actual
local process/ACP browser paths pass again in 1.3 minutes, without skips
or retries. The final screenshot shows Cancelled, a paused subtree,
retained input, and no error toast.
- Two new actual-service regressions fail before the merge fix. They
prove that queued-comment adoption could consume a dedicated chat answer
or add unrelated input to that answer. The fixed four-case cohort
passes, including ordinary upstream continuation and adapter Stop
controls. Full recovery passes 257/257. All four actual local
process/ACP Stop browser flows pass in 1.4 minutes, without skips or
retries, on a fresh database.
- The unchanged runner artifact was qualified with 171/171 transport
tests, 870/870 API-authority tests, conformance 1/1, and replay 11/11.
Six controlled reader tests prove the exit/drain repair. Its local
serial Rust workspace passed 546 top-level cases plus two invoked
helpers; the later passing Linux CI supplies default-concurrency
evidence.
- Prior exact-source full chat integration passes 995/995. Settings
regressions cover concurrent stale pages, 501 destinations, pending
state, rejected updates, and explicit retry. These deterministic tests
do not prove live provider behavior.
- Retained failed attempts and their causes are in the [qualification
log](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-chat-queue-and-webhook-repair.md).
The first merge adapter run timed out while macOS slept for 290 seconds.
Its unchanged repeat passed with a temporary sleep guard. No assertion,
deadline, or CI gate was weakened.

Review commands include `pnpm --filter @paperclipai/server exec vitest
run src/__tests__/heartbeat-process-recovery.test.ts
src/__tests__/issue-queued-comments-routes.test.ts` and `pnpm exec
playwright test --config tests/e2e/playwright.config.ts
tests/e2e/acp-stop-continuation.spec.ts`. Database suites require fresh
disposable databases. See the [browser
runbook](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-04-chat-adapters-browser-e2e-runbook.md)
for provider setup and separate live acceptance steps.

## Risks

- This remains experimental. Deterministic tests and bounded live
evidence do not establish every provider feature, tenant, permission
layout, or media shape. Teams work-tenant qualification is still open.
- Failed and uncertain provider effects remain visible and can require
operator action. A transport receipt does not prove recipient
visibility.
- Native controller and runner artifacts must remain compatible.
Preserve lease ownership, terminal authority, source binding, and
quarantine during future changes.
- Access and audit rows commit together, but activity notifications
remain best-effort. This is not a new durable event outbox.
- The PR operation does not deploy a live server, replace its runner, or
change provider permissions. Remaining live qualification is documented
in the [temporary
handoff](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-open-qualification-followups.md).

## Model Used

OpenAI Codex assisted with implementation, tool execution, testing, and
review. The work records `gpt-6-astra` assistance. The environment does
not report a context-window size. No private reasoning traces are
included.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-10 10:06:45 -05:00