Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-01 23:43:16 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,280 Commits 1,350 Branches 2,210 Tags
62eddee7dbda4b17440a6d24d34bcd4aa5bb0f13
Commit Graph
425 Commits
Author SHA1 Message Date
DottaandPaperclip 4c3bb2946d Carry qualified card search repair through 18382 
Normal parent merge; only the five qualified card/search paths change. Preserve every other owned and upstream blob.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 10:33:38 -05:00
DottaandPaperclip bd78176b40 Carry qualified card search repair through 18399 
Normal parent merge; only the five qualified card/search paths change. Preserve every other owned and upstream blob.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 10:33:38 -05:00
DottaandPaperclip d2ef6c81d0 Carry qualified card search repair through 18388 
Normal parent merge; only the five qualified card/search paths change. Preserve every other owned and upstream blob.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 10:33:37 -05:00
DottaandPaperclip e0986ac344 Paginate watched status-card issues before unrelated search results 
Add an internal result-kind constraint owned by company search, preserving all/comments/documents task matching and default mixed search. Apply it to card snapshots and preview without unrelated entity producer fanout.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 10:30:16 -05:00
DottaandPaperclip 74a3b3c129 Carry final qualified cards repair through 18382 
Normal parent merge; only the three qualified status-card paths change. Preserve every other owned and upstream blob.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 10:19:44 -05:00
DottaandPaperclip 3dc77ca942 Carry final qualified cards repair through 18399 
Normal parent merge; only the three qualified status-card paths change. Preserve every other owned and upstream blob.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 10:19:43 -05:00
DottaandPaperclip 42ece47d98 Carry final qualified cards repair through 18388 
Normal parent merge; only the three qualified status-card paths change. Preserve every other owned and upstream blob.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 10:19:43 -05:00
DottaandPaperclip 9a9668425b Preserve completed shared status-card refresh generations 
Guard stale cleanup with the durable generation ledger and a transaction-bound canonical cancellation. Coalesce the original completed manual request, retain future manual work, and preserve one-connection and outer-commit effects.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 10:18:18 -05:00
DottaandPaperclip f0abcbb18e Carry qualified cards repair through merge confirmations 
Keep the complete confirmation owner and latest routine parent intact.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 09:31:16 -05:00
DottaandPaperclip 508115ef70 Carry qualified cards repair through routine status work 
Preserve the complete routine status slice and additive retention ownership.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 09:30:58 -05:00
DottaandPaperclip 8e414f54f4 Merge final cards revision ownership and parent acceptance 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* fix/sleep-merge-18401:
  Scope status-card refresh tasks to their claim revision
  test(chat): retain completion identity before link metadata loads
  test(chat): locate completion through its rendered task link
  fix(chat): recover committed native Agent Chat replies (#18465)
  perf(heartbeat): drive retry promotion from durable deadlines (#18387)
  fix(ui): make the Coordinator composer footer opaque (#18455)
  fix(testops): verify unexpanded result job runs (#18462)
  fix(runner): preserve OpenCode written-answer defaults (#18456)
  fix(native): fence retries when replaying an audited Stop (#18457)
  test: await async outcomes and drive saved deadlines deterministically (#18416)
 2026-10-31 09:29:14 -05:00
DottaandPaperclip 1d800abd49 Scope status-card refresh tasks to their claim revision 
Keep concurrent refreshes of one revision deduplicated while preventing a superseded refresh from cancelling a newer revision task before its claim.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 09:26:46 -05:00
DottaandPaperclip cc25f28525 Merge qualified routine parent into merge confirmation work 
Preserve confirmation candidates and notification ownership alongside the current routine, native, security, and retention changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 09:10:30 -05:00
DottaandPaperclip 63c77102a8 Merge qualified retention parent into routine status work 
Preserve routine status producers alongside current native, security, retention, and deferred execution owners.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 09:08:08 -05:00
DottaandPaperclip aab54d5bec Merge current master and generated card schema into attention retention 
Preserve master attention, issue, native cleanup and tool authority changes alongside the exact reviewed retention patch. Keep bounded blocker facts, finalization precedence, source locks, archive eligibility and producer intent unchanged. Publication remains dependent on the parent qualification.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 08:51:00 -05:00
DottaandPaperclip 56f8ff7bd5 Merge current recovery and regenerate status-card migration after master 0419 
Preserve the published concurrent comment uniqueness and read-state backfill migration. Regenerate the combined schema as migration0420 with the repository workflow and retain five snapshots. Add both upgrade-path controls without changing status-card execution policy.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 08:39:44 -05:00
Dotta 3549fad1dc Merge commit '0b4e48be9b881484dc5c48225a3b501595329ce2' into codex/sleep-merge-18409 
* commit '0b4e48be9b881484dc5c48225a3b501595329ce2':
  fix(ui): retain startup autosave through session recovery (#18412)
  perf(server): recover independent native replies concurrently (#18421)
  perf(chat): scope task questions and batch inbox queue checks (#18422)
  fix(native): recover controllers across webserver replacement (#18343)
  fix(ui): restore task document authoring and add persistence journey (#18356)
  fix(tools): stabilize tied profile entry order (#18427)
  fix(adapter-utils): tolerate entries removed during workspace snapshots (#18426)
  test(runner): qualify Kimi Code lifecycle and recovery (#18029)
  feat(runner): deliver native Kimi Code (#18391)
  Preserve selected Google services when setup resumes (#18048)
  feat(runner): add pinned Kimi Code native runtime (#18028)
  fix(ui): show agent-created subtasks inline in the task feed (#18320)
  fix(evals): preserve opaque TestOps parameters (#18394)
  feat(chat): deliver proactive task messages to agent chats (#18374)
  fix(ci): track repository inputs across test projects (#18398)
  chore(lockfile): refresh pnpm-lock.yaml (#18414)
  fix(runner): build on Windows and add a Windows CI lane (#18250)
 2026-10-31 08:27:30 -05:00
DottaandPaperclip 0b4e48be9b Merge current controller recovery with scoped issue deadlines 
Preserve keyed concurrent checks and their default serial behavior alongside issue-recovery reconstruction spacing and before-pass preparation. Qualify keyed ordering, joined preparation, exact deadline preemption and retained late-write reconstruction debt across drain.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 08:20:41 -05:00
DottaandPaperclip d1f9534c54 Merge current master qualification into event-driven run dispatch 
Preserve retained-controller recovery deadlines and rearming together with company-scoped dispatch forwarding. Keep both source owners and all incoming master changes; no execution or retry policy changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 08:03:05 -05:00
DottaandPaperclip 492d537340 fix(native): recover controllers across webserver replacement (#18343) 
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native execution runs in a remote Runner while the webserver owns
its controller.
> - Encrypted PRP checkpoints and session snapshots already survive a
webserver restart in Postgres.
> - Ordinary warm and per-turn admission still needs local markers and
supervisor metadata to find that state.
> - Replacing the webserver with an empty native controller directory
can therefore lose a usable retained session.
> - This change adds bounded recovery metadata to the existing snapshot
transaction and reconstructs local projections from the selected durable
source.
> - The benefit is native execution continuity across webserver
replacement, with the existing ownership and agent lifecycle policies.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Native remote controller recovery and retained-session continuation
during a webserver upgrade.

**Subsystem affected**

server/ — native runtime, checkpoint persistence, recovery scheduling
and internal upgrade reporting. Product E2E qualification also changes.

**Current behavior**

Without both the local PRP marker and supervisor file, normal
continuation can return before discovering the saved Postgres
checkpoint. The snapshot already exists in Postgres, but its local reuse
compatibility envelope cannot be reconstructed.

**Proposed behavior**

Select the existing durable source, validate its identity and ownership,
restore its PRP state and supervisor projection, then use normal
authenticated admission. Missing or inconsistent evidence holds
recovery. It does not authorize fresh provider work.

**Reason and benefit**

A replacement webserver must recover active execution and later continue
its retained session without the old server's native files. This is one
component of deployment continuity. HTTP traffic switching, workspace
storage migration and Railway rollout remain separate.

**Breaking changes**

No new public endpoint, schema migration or database pool. The private
run-profile envelope is additive. Disk-independent rollback also
requires a reader for the staged controller format and compact
checkpoint part; the first production release containing these readers
is the floor. Older file-backed sessions need an owner checkpoint or the
bounded internal importer.

**Additional context**

Related foundations: #17401, #17916, #17254, #14621, #14622. The
implementation contract and open acceptance gates are in [the dated
plan](doc/plans/2026-10-30-native-controller-checkpoints.md).

## What Changed

- Persist a versioned recovery envelope, at most 4 KiB, beside the
redacted session snapshot in its existing transaction. Keep it out of
public run projections.
- Compare frozen admission separately from prepared attachment
instructions during staged recovery, preserving exact
template/identity/authority checks.
- Bound the existing lazy checkpoint pool at three connections after
measured two-slot burst backlog; keep request/renewal capacities and SQL
operation counts unchanged. The tuning is a separate commit.
- Discover and restore the exact selected retained source without local
native markers. Preserve the 32-candidate window, progress barriers and
current authority checks.
- Fence snapshot and result writes. Acquire coordinator, issue and run
locks in order and check expiry after lock waits. Combine snapshot
ownership/read work into one statement.
- Reuse existing recovery deadlines and drain/resume scheduling.
Retained-owner waits preserve the original unstarted run and actual
lease deadline. Stop, ownership and provider-progress fences precede
deferral. Add bounded native evidence reporting, retained-source import
and retention protection.
- Admit a settled warm source with an authenticated idle goal-read
suffix while retaining active-work and mixed-authority holds.
- Permit lifecycle `turn.stop` checkpoints after accepted results under
existing ownership fences. Claim the existing successor coordinator
before retiring the old warm session, and stage shared attachments only
after verified retirement. Preserve omitted isolated-home defaults and
existing fresh skill-selection order during recovery without hiding real
configuration changes.
- Carry the exact durable six-field PRP binding during active recovery
after a healthy warm handoff. Keep the Runner mismatch guard and
preserve opaque attachment IDs.
- Preserve an already verified active or idle warm process during remote
preparation; revalidate its proof/liveness and retain later
authenticated PRP attachment. Ordinary resumed leases retain suspension
checks.
- Drain retained source events under their original authority, reconcile
immutable source accounting and block unknown source tool dispatch
before target work.
- Persist compact staged handoff intent through authenticated adoption
and recover direct/listener command-loss windows without another Runner
or duplicate attachment.
- Rebuild suspended lifecycle receipts for resumed prepared tasks, and
retry missing lifecycle reads after partial PRP materialization without
overwriting contradictory evidence.
- Add normal warm/per-turn diskless continuation coverage, real Postgres
races and SQL-cost checks. Add a reproducible paired load workload.
- Give only the explicit outage fixture a 15-minute provider idle
window, with a red/green catalog regression preserving all production
timing and lifecycle policies.
- Publish the goal-less task-session pointer in the existing
post-acceptance terminal checkpoint transaction, with current ownership,
accepted-result, reset and newer-progress fences. Keep a private bound
receipt so late completion cannot duplicate or resurrect publication.
- Use a retained warm sandbox in the dedicated continuity fixture, and
accept an exact output token followed by an explanation. Keep the strict
identity oracle.
- Extend the explicit Daytona cell to a five-minute active outage and
another replacement after completion. Require retained provider
identity, a new tool receipt and a useful follow-up result. Add an owned
loopback Postgres E2E fixture for hosts that cannot start embedded
Postgres.

## Verification

**Merge recommendation — `fc68584ce6231a02e47ecd743f55b036ccea034f`:
ready for the qualified native-controller component.** Full CI, fresh
review, local application paths, controlled performance qualification
and the strict live Linux/Daytona case pass. Independent resource
cleanup is complete. Broader rollout, browser reconnection and the
unexecuted transport/platform combinations remain integration gates.
Integrated 484 passed full local recursive typecheck/build and ordinary
active→completion→cache-loss→C component qualification. Fresh review
then found a valid attached-wake failure: preparation appends attachment
instructions before the verifier compares execution to the frozen
admission. 9f4 fixes that by separately checking the pre-preparation
admission and exact prepared execution. Both hashes are captured
privately per admission; the full template, identity, configuration and
authority checks remain. Its regression fails before the fix; 36 focused
cases, 42 staged source/target cases and server typecheck pass
afterward. The review thread is resolved. Exact-head full CI passes
([run
41168228822](https://github.com/paperclipai/paperclip/actions/runs/41168228822):
48 successful jobs, one intended skip); the complete `pnpm test:run`
partitions are forced, with the five native suites in the Runner lane.
This is not a claim that an unsharded local command passed. The fresh
exact-head Greptile check succeeds with 5/5 and all six threads
resolved.

The original two-connection database comparison closes **steady PASS and
burst FAIL**. Every relative CPU/WAL/request-p99 gate passes and all
operations/ownership remain correct, but candidate arm 3 drains in
1,015.99 ms and baseline arm 4 in 1,059.67 ms against the unchanged
1,000 ms absolute gate. Those results remain failed. A separate 2→3→4→2
diagnostic reproduces drain times 1,051.57 / 956.79 / 882.02 / 1,065.72
ms: three is the smallest tested passing cap, and the repeated two-cap
failure survives run-order/cache effects.

Separately reviewable `fc68584ce6` changes the existing lazy checkpoint
pool default to three connections. Request capacity stays ten and
liveness two; idle release, shutdown fencing, transaction counts and
pool count are unchanged. Thirteen DB lifecycle/cap tests and DB
typecheck pass. Full recursive typecheck/build pass at exact fc68584,
with the matching server build stamp. The final paired native
feature-cost comparison explicitly uses the **same cap-three policy and
common current DB factory in both variants**; its baseline is
controlled, not the unmodified historical cap-two configuration.
Original gates and workload counts remain unchanged. The one strict
Linux/Daytona case started at 04:40:56 UTC only after both new profiles
passed; it passed on the first attempt with all 49 workflow checks.
Independent cleanup confirms all three owned sandboxes absent, owned
Postgres/controllers/observer gone, VM terminated, root volume absent
and ephemeral SSH key/agent removed.

**Final controlled cap-three qualification: PASS.** Baseline native
implementation `59da15066a` and candidate `fc68584ce6` use the same
current DB factory and checkpoint policy (request/liveness/checkpoint =
10/2/3). Eight 65-second ABBA arms cover 300 controllers and 10,000
historical tasks. All correctness, absolute and 10% relative gates pass;
original cap-two failures remain unchanged.

| Mode | DB CPU µs/op, baseline → candidate | WAL B/op, baseline →
candidate | DB-request p99 ms, baseline → candidate | Maximum drain |
| --- | --- | --- | --- | --- |
| Steady | 542.40 → 554.83 (+2.29%) | 1509.01 → 1558.27 (+3.26%) |
107.11 → 111.90 (+4.48%) | 26.35 ms |
| Burst | 1036.36 → 1063.11 (+2.58%) | 4153.12 → 4317.64 (+3.96%) |
103.736 → 103.841 (+0.10%) | 930.14 ms |

Every matched arm preserves operation/statement/transaction counts,
including 3,900 PRP and 3,900 snapshot transactions. All three actual
pools are included; there are zero lost owners, missed lease deadlines,
dropped operations or errors. Peak Postgres CPU is 18.31%/9.47% of two
cores and memory 614.29/684.42 MiB (steady/burst). The benchmark uses
one Node process simulating 300 controllers; it does not qualify
two-webserver overlap, HTTP latency or hosted capacity. Its observer SQL
contributes to measured database CPU. The original unmodified cap-two
baseline is separately retained as failed; this cap-three comparison is
explicitly controlled.

The exact final-source 8/512/8,192-chunk test also passes: zero recovery
SQL, transactions or changed rows in each text-only interval, zero idle
SQL, seven lifecycle transactions and twelve changed rows per sample.
Existing lease traffic remains separate. The final ordinary managed
application replay passes active recovery, completion, native-cache
removal and C continuation; its UNKNOWN local debug-daemon provenance
and partial instruction-observer limits remain recorded. The live case
uses the attested Linux producer instead.


**Live qualification: PASS, one attempt, no retries.** Campaign
`native-controller-retained-fix-fc68584ce6-ec2-20261031T044055Z` started
04:40:56 UTC on the exact final Linux webserver source; suite definition
hash `12f9bb97823c1537e43f0398e7cddf39890f77bce7b413ca4dfd944f4a30e5d5`.
The existing immutable image and attested Bookworm Runner artifact are
unchanged.

- A was SIGKILLed and its native cache discarded. B started after
300,149 ms, naturally acquired generation 2, authenticated at 316,624 ms
and resumed at 317,682 ms within the original 420,000 ms grace. The same
active run, Runner, provider PID/start fingerprint and one
submitted/started/accepted turn were preserved; one final
result/terminal and workspace finalization completed, with no
retry/recovery action.
- After active and queued completion, B was SIGKILLed and its native
cache discarded again. C received exactly one ordinary task-comment API
follow-up, completed a new tool and produced a useful reply while
preserving native/provider/Runner identities. This closes the earlier
failed post-C gate. A separate C OS-process PID field was not exported;
the physical-PID proof is specifically A→B.
- Four distinct results were accepted and published; all four
finalizations committed at attempt 1, all four accounting-v2 projections
settled, and the task-session pointer advanced to C. The coordinator
independently checked these facts and all 49 canonical checks against
the sanitized receipts.
- Canonical cleanup passed and independent checks confirmed three exact
sandbox 404s plus complete inventory/zero owned remnants, no owned PG or
controller/observer processes, EC2 terminated, root volume absent, and
the ephemeral SSH key/agent removed by 04:54:25 UTC.

Live scope is classic remote Codex warm continuity with non-native
storage dependencies available. Prepared-task completed B→C, full
per-turn application disk loss, browser reconnection, two-server load
and overall Railway/HTTP continuity were not live-qualified here.
Earlier failed campaigns and the original cap-two burst failure remain
failed historical evidence. A report-renderer cache setup failure was
preserved; the normal sanitizer subsequently succeeded with an explicit
cache path and a new output directory, without changing validators.


The repair keeps retained events under their original six-field
authority until persistence and accounting accept them. Unknown semantic
effects stay unresolved. A compact staged target survives source
bootstrap, cached-command delivery and authenticated handoff, including
an uncertain commit response. Direct and same-guest-listener recovery
require the verified original Runner and the exact endpoint. Target
snapshot publication marks readiness in memory, with retirement
coalesced into an existing command write. Headers contain only the
staged descriptor/digest; a prequeue template is written once as a
changed checkpoint part and then recovered from the existing
command/receipt. No second session snapshot transaction, streaming SQL
path, new pool or recurring idle scan is added.

| Current repair gate | Evidence and scope |
| --- | --- |
| Source-tail regression | Real Runner protocol RED reproduced
attachment rejection. Frozen core/transport pass five source-tail cases:
source usage attribution, held persistence/no premature ACK, lost commit
response plus cache loss, unknown semantic hold and accounting rejection
hold. |
| Handoff loss windows | Direct and listener modes pass
queued/prepared/activated/completed boundaries (eight cells), preserving
the actual Runner/provider process. Missing adoption proof and changed
endpoints reject recovery. |
| Compact storage | A 31 KiB attachment grant writes its separate part
once, leaves it unchanged across source snapshots and removes it at
queueing. Headers do not duplicate the grant. |
| Accounting | 21 real-Postgres usage tests and 19 source-binding tests
pass. Recovery hydrates immutable source receipts without resetting
their producer or billing classification; conflicting/incomplete usage
holds. |
| Admission and malformed evidence | Driver 16, staged evidence 42,
restart classifier 27 and source usage 19 focused cases pass. Four
malformed-file regressions pass at d6a7, whose executor is
byte-identical at 484, and server typecheck passes. Full executor file:
990 pass; 16 embedded-Postgres `initdb` setup failures before
assertions, retained separately. |
| Required checks | Exact fc68584 recursive typecheck and build pass
locally, with matching server stamp. Forced full CI succeeds (48 jobs
pass, one intended skip); all PR checks are clean. Maps/token gates pass
for their unchanged source surfaces. Fresh exact-head Greptile is 5/5,
all six threads resolved. |
| Ordinary application path | Exact fc68584 managed component passes
active recovery after natural expiry, completion, native-cache loss and
ordinary C continuation. Production bytes match before/after; temporary
fixtures restored and owned DB/services cleaned. UNKNOWN local
debug-daemon/partial instruction-observer limits remain; the separate
live case uses the attested Linux producer. |
| Producer | Locked Bookworm build from exact 5b7 Rust tree
d03e494217, equal to final fc68584 Rust
source. SHA-256
65b8f8cfdd6099e9ad24f1ca154e8a260777269cc82f6e23c6e6a180c531eaaf;
38,515,328 bytes. Actual Debian 12 execution and live admission pass.
Controller/admission/DB changes do not relabel the producer source. |
| Historical cap-two comparison | Exact 59da baseline vs 5b7 candidate,
300 controllers/10,000 history, ABBA 4×65 s per mode, all three actual
pools and real snapshots. Steady CPU/op 541.36→554.64 µs (+2.45%);
WAL/op 1512.13→1560.99 B (+3.23%); DB-request p99 102.78→105.99 ms
(+3.12%): PASS. Burst CPU/op +4.71%, WAL/op +5.29%, p99 −0.10% pass
relative gates, but absolute drains above fail. Identical operation/SQL
counts, zero lost leases/drops/errors. One Node process simulates 300
controllers; this does not qualify two-webserver overlap or HTTP
capacity. |
| Live acceptance | Strict final-source Linux classic Codex case passes
1/1 on attempt 1, all 49 checks. Active recovery within grace and useful
retained C continuation succeed; four
accepted/published/committed/settled outcomes. Independent provider,
PG/process, VM/root and SSH cleanup complete. Prepared/task and broader
rollout limits are explicit above. |

**Historical qualification snapshot through `9eb9ac6`.** The head at
this snapshot was `9eb9ac6f9e312f3ebe8040f5a7565e88ae8824f3`. Server and
Runner package sources are byte-identical to `283e05ffaa`; the two later
commits change the explicitly scoped qualification and its documentation
only. It includes master through the storage-owner fix
`45a3a92e6dc52d20923b9aa34a22fee2ec5443f9` (#18357). At that snapshot,
CI/review passed but the strict post-C native recovery gate failed; the
PR stayed draft and required another native repair. Prior `39a4343` CI
and review passed, but its full live workflow failed; that grade remains
unchanged. The first repaired `435ccd2` live attempt failed before any
injected outage. It exposed a legitimate terminal snapshot before result
acceptance; no accepted result ID or publication receipt was observed.
Its canonical report remains 0/1 `candidate_failure`, with cleanup
passed and independent provider absence confirmed. `0862062` defers
publication until result acceptance and fixes the fresh review's
coordinator/issue lock-order finding. Both failures have deterministic
real-Postgres regressions. The `0862062` live attempt then proved
accepted-result publication but failed active recovery: a resumed-lease
preparation guard demanded suspension after the exact live Runner was
already verified for adoption. `64c0b746` preserves that verified
active/idle-warm process, revalidates its proof/liveness, and keeps
ordinary suspension checks and later PRP authentication. Its live run
passed preparation but exposed a second active-warm boundary: all
durable binding fields matched except opaque turn/item attachment IDs,
which the server had regenerated. `7a203274` carries the complete
verified six-field binding and keeps the strict Runner mismatch guard.
Its cloud case then passed active adoption, completion, publication and
accounting. The queued continuation exposed accepted-result retirement
writes and an implicit-home fingerprint drift; both are fixed with
deterministic regressions in `39a4343` before this fresh full
qualification.

The prior live case exposed a real publication gap. A goal-less native
turn could persist its accepted result and terminal snapshot before
provider cleanup, while the task-session pointer was still empty. Normal
later admission then selected a fresh session. The fix publishes that
pointer and a private bound receipt in the **existing post-acceptance
terminal checkpoint transaction**, before cleanup. Earlier terminal
observations persist without a handled receipt or continuation pointer.
No result-acceptance or snapshot transaction was added. The late
completion path recognizes the receipt. This also prevents a late
callback from restoring a pointer after a user reset. Goal/control
publication keeps its existing owner.

The dedicated Daytona fixture now uses the existing warm/reusable
environment. Its sandbox, Runner and provider storage must survive. The
former generic per-turn fixture disposed the sandbox after completion
and kept provider-history backup bytes under the discarded directory.
Whole per-turn disk-loss acceptance therefore has a separate
provider-backup storage dependency. It remains distinct from the passing
native per-turn component checks with backup storage available. The
60-second lease, 300-second outage and 420-second grace are unchanged.

| Gate | Current repair evidence |
| --- | --- |
| Assembled regression | Baseline fails with an accepted result and
terminal snapshot but a NULL task pointer while real provider cleanup is
held. The fix passes normal continuation and reset cases, with no
pointer seeding. Exact merged `92b7447` passed 2/2 twice, including
after normal package TypeScript output refresh. Production files are
byte-identical at frozen `435ccd2`. |
| Persistence and concurrency | 19 helper/transaction cases and all 59
existing control-port cases pass on `0862062`. They cover bound result
IDs, reset/newer epochs, Stop, lock waits and expiry, receipt replay,
missing stubs, foreign bindings and atomic 4 KiB rollback. Existing
snapshot transaction count stays one. The new tests reproduce the exact
live missing-result-ID error and the inverse replay/status lock order
against `435ccd2`, then pass after the correction. Replayed results
still reject stale generations. |
| Completion | All 32 completion-boundary tests pass on merged
`92b7447`; the source is unchanged at `435ccd2`. Native result-shaped
session metadata is preserved. |
| Qualification fixture | A valid review report reproduced three
rejected correct replies. Exact token-boundary matching fixes them. 140
focused E2E/catalog cases and Product E2E typecheck pass. Wrong tokens,
marker prefixes/suffixes and wrong authors remain rejected;
retained-identity checks are unchanged. |
| Required checks | Full recursive typecheck and standard build passed
at `435ccd2`. Both exact commands also passed on `0862062`, with its
exact server build stamp; both also pass on `7a203274`, with its exact
server build stamp. Fresh `0862062` Greptile passed 5/5, with zero
unresolved threads. The `64c0b746` normal CI then passed all 49 jobs (48
success, one expected skip) and Greptile passed 5/5 with zero unresolved
findings. The `7a203274` normal CI also passed 48 jobs with one expected
skip, with fresh Greptile 5/5 and zero unresolved findings. Current
`39a4343` recursive typecheck and standard build both pass with
persisted exit codes and its exact build stamp; fresh Greptile is 5/5
with zero unresolved findings. Its normal CI completed successfully: 48
jobs passed and one expected job skipped. The fresh live result remains
separately failed. The `0862062` normal CI had 46 passing jobs, one
expected skip, one worker communication failure and its aggregate
failure; GitHub states the self-hosted runner lost communication. No
failed assertion or specific OOM/spot cause was established, and no
blind retry was used. |
| PostgreSQL supplement | One corrected all-pool pair at the preceding
repair passed 300-controller/10,000-history correctness and absolute
gates. SQL 76,590→75,990; transactions 9,013→8,713; UPDATE rows 11,613
each; 300 correct pointer writes each. PG CPU 31.99→30.46 s, WAL
70.36→67.87 MB, request p99 100.98→103.12 ms. This is one pair, not a
repeated improvement claim or an exact `0862062` CPU measurement. Its
two later changes defer premature publication and reorder existing
locks; they add no transaction, stream/idle writer, pool or timer. The
first collapsed-pool attempt remains invalid and retained. This is not
an exact final-head CPU/WAL measurement; later fixes change ordering,
identity selection and in-memory skill replacement without adding
snapshot transactions, pools, stream writes or idle SQL. |
| Live publication proof (`0862062`) | Baseline run
`06fcc806-f186-4190-9b57-087b61d75533` accepted result
`58f912dc-94d9-4b4d-bd3d-d5f689c83fd3`; the publication receipt names
the same result, the task pointer has parameters and `lastRunId`
matches, and the normalized native identity matches the following active
run. Accounting v2 settled. |
| Active recovery (`0862062`) | **FAIL.** SIGKILL/cache discard at
23:07:28.666 UTC; hold 300,000.859 ms; B ready at 308,896.486 ms;
natural generation-2 ownership at 312,906 ms. At about 319 s,
preparation failed `runner_harness_state_mismatch:
resumed_not_suspended`. No authenticated after-receipt passed within 420
s. Exact original Runner PID 668/start/fingerprint was independently
still alive; provider auto-stop was 15 minutes. No retained C follow-up
qualified. Canonical 0/1 `cleanup_failure`/cleanup-failed report remains
failed; subsequent independent cleanup deleted all three owned resources
with exact 404s and confirmed full intended-account inventory with zero
owned resources. |
| Verified adoption preparation (`64c0b746`) | 91 focused cases pass,
including actual preparation for four active providers and a distinct
prior/current warm proof. Unproved resumed state, dead process and stale
warm owner remain rejected. The earlier options-only fixture missed this
gate. No spawn, backup restore or PRP-authentication bypass was added.
Server typecheck passes. |
| Active warm identity (`64c0b746`) | **FAIL.** B claimed generation 2
at 312,995 ms. Preparation passed, then strict PRP recovery rejected the
generated attachment IDs at about 319 s. The saved and desired Runner,
lease, normalized session and run IDs matched; only
`turn_attachment_…`/`item_attachment_…` differed from generated
`turn-<run>`/`item-<run>`. Canonical 0/1 cleanup-failed report remains
failed; independent cleanup confirmed deletion/404s and zero owned
provider resources. |
| Six-field identity repair (`7a203274`) | Actual server-factory
regression RED4→GREEN38. A real Rust Runner test passes healthy first
run → random-ID warm handoff → held active second run → restored
controller. It verifies a new live authenticated connection, exact
identity, same Runner/provider OS identities, no duplicate launch/turn,
and no redispatch or invented receipt for an uncertain tool. The
synthetic wrong pair still fails the unchanged guard. This protocol case
does not claim business completion. Server and Runner typechecks pass. |
| Active recovery/completion (`7a203274`) | **PASS for the active
component.** Full five-minute SIGKILL/cache-loss hold. B ready at
308,936.651 ms; natural generation 2 observed 312,979 ms; authenticated
at 317,720 ms and resumed processing at 318,527 ms. Same run
`1e1ccb00-0728-4935-97da-57562d81cb42`, Runner
`246f73f7-a689-4177-a9c2-7ce711d63aed`, remote PID 669/start
23:48:27.470, provider lease/session and one submitted turn. Accepted
result `49097529-5310-4ac3-af95-5bf9b91d19f4`, publication, finalization
and accounting v2 all settled. |
| Full workflow (`7a203274`) | **FAIL, preserved.** Queued
late-requirement run `5e61f38d-c7d1-4fc1-bcbf-bce9d46ce4eb` failed
`runner did not durably suspend before checkpoint` before outage 2.
Canonical 0/1 candidate failure, cleanup passed, evidence valid. C
retention was not exercised. Independent provider absence,
controllers/PG cleanup, EC2 termination and root-volume absence all
confirmed; closed receipt SHA-256
`676a3b15eb9937b6bf995c36fb8666c88964d509e5ceb9109fb02da0aa2bc909`. |
| Retirement and stable configuration (`39a4343`) | Real Runner +
Postgres RED reproduced the exact suspension failure and nine owner
rejections. `turn.stop` is now lifecycle checkpoint maintenance under
unchanged owner/result/generation fences. GREEN25/25 proves actual
persisted suspension, exact identity and completed stop/drain/suspend;
unaccepted, foreign, stale and new-work negatives remain rejected. Two
default-home fingerprint REDs become GREEN9/9; explicit mode changes
still invalidate. No global matching exemption or historical fingerprint
rewrite. |
| Fresh full Daytona warm cell (`39a4343`) | **FAIL, preserved.**
Launched 00:15:29 UTC with automatic retries disabled. Active adoption
passed: actual offline hold 300,000.848 ms; B ready 309,922.155 ms;
natural generation 2 at 311,821 ms; authenticated 317,989 ms; resumed
330,440 ms. Same run `3e126819-e786-4b3b-b923-c975cf4f5b79`, Runner
`4079138a-8d7c-44a4-8558-07a5411d1f55`, PID 668/start and
provider/session identities. Accepted result
`08e504af-4dec-48a8-bf05-bf0a32280559` and publication passed. Queued
run `31d168c9-7ab9-4700-abb6-fafb90912237` failed the same strict
suspension error before outage 2. Its row selected a new native
identity, but no new third provider process or invocation was observed.
Recovery-time configuration still reset only `adapterConfig`; native
provider, context and workspace subtrees were unchanged. Canonical 0/1
candidate failure, evidence valid, normal cleanup passed; all campaign
provider/PG/process resources independently absent. Closed campaign
receipt SHA-256
`5e7d6281c13c812f3a946667f97f3c7d339b5ad86a3a819bd185b2fa73bc4a19`;
EC2/SSH remain temporarily held for the concrete follow-up repair.
Original 60/300/420-second limits and strict queued/second-replacement
checks remain unchanged. Direct-process path; no new live prepared-task
claim. |
| Native retirement authority (`283e05ffaa`) | Exact `39a4343` facade
regression RED3→GREEN9. The successor now uses its existing production
coordinator claim and prompt renewal before strict retirement. A real
Runner/Postgres case starts in the actual native-observed/unclaimed
phase, claims through production code, then closes and durably suspends
the predecessor in 1.657 s. All 25 real-Postgres cases pass;
unclaimed/foreign writers remain rejected. Shared attachment staging
stays after retirement; Stop and denied claims neither close nor stage.
No new claim or SQL path was added. |
| Configuration continuity (`283e05ffaa`) | Real shared/project-primary
environment selection → configuration with four skill versions
reproduces `reset: true`, changed category exactly `adapterConfig`,
after a Postgres JSONB read. Pinned restoration had reordered the
generated preference while actual runtime skills stayed identical.
Replacing existing pinned rows in place preserves the original fresh
name/key order and fingerprint. The assembled regression passes, as do
three existing pinned-version/rename/alias cases and server typecheck.
Genuine model/skill-selection changes still invalidate. No
fingerprint-format change or broad compatibility exemption. |
| Final production repair (`283e05ffaa`) | Full recursive
typecheck/build and [normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41151145106)
PASS: 48 jobs plus one expected skip. Fresh review 5/5, zero unresolved
threads. Live baseline, recovered run and queued continuation all
succeeded, each accepted/published/committed once with accounting v2
settled. Their native/provider identities and configuration fingerprint
match; both recovery and continuation report no reset, no changed
categories and task-session reuse. Active outage/authentication remains
within original timing. |
| Browser-dependent continuation (`283e05ffaa`, definition 6) | **Whole
cell FAIL, preserved.** After B was killed at 00:46:38.928 and its cache
discarded, C became ready in 9,151.410 ms. `openParent()` then hit a
30-second `page.goto`/DOMContentLoaded timeout before posting the final
request. No fourth native run or C continuation was exercised. Cause of
navigation timeout is unknown. C task had no execution lock, queued
comments, handoff or recovery blocker; all three prior runs were
committed/accounted. Normal cleanup passed; independent provider
absence, controller/PG cleanup, EC2 termination and exact root-volume
absence confirmed. Sanitized report bundle SHA-256
`2a3b841d62efb2ae30ef14f96a431ac4fa7fe3a782f1144831512f78edc65022`. |
| Native API continuation qualification (`9eb9ac6f9e`, definition 7) |
**FAIL, preserved.** The real post-C API request dispatched run
`c71fbb18-f979-4fe4-93ce-9c4be46dbf21`, which failed
`native_session_interrupted` / `native_session_recovery_failed` at
01:10:00.640 UTC. It selected the same normalized native identity,
Runner and checkpoint-provider identity as the three succeeded turns. No
C result was accepted; pending accounting remains explicit. Active
recovery passed: actual offline hold 300,001.098 ms, B ready 308,335.019
ms, natural generation 2 observed 311,818 ms, authenticated 315,540 ms,
resumed 316,612 ms. Queued continuation succeeded/accepted/published. B
was then killed/cache discarded at 01:09:34.500 and C ready in 8,783.803
ms. Original classifier `transient_infrastructure`, whole-case failure
and cleanup passed are unchanged; that classifier does not establish the
root cause. All owned provider/PG/process resources, EC2, root volume
and SSH agent independently absent. Closed receipt SHA-256
`7f21ac4a22efc07b849fa7e17866083b6127b3321af12671b4fc054ce57039a9`;
sanitized bundle
`71ffef14a30a553e008320a31840a6d6c0de85c1bfdb60f783fdb53a6aae812a`.
Exact backend detail is unavailable in approved structured evidence. |
| Current required checks (`9eb9ac6f9e`) | Recursive typecheck, standard
build and exact build stamp PASS. [Normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41152908417)
PASS: 48 jobs plus one expected skip. Fresh Greptile 5/5, latest edited
summary checked, zero unresolved threads. Exact8/512/8,192 stream probes
again report zero recovery SQL/transactions/rows and settled-idle SQL;
seven control transactions/twelve changed rows each, with 36 separate
legitimate renewals in the longest case. CI success does not supersede
the failed live C gate. |

The completion supplement observed secondary request/liveness/checkpoint
transaction p99 increases of 23.0%/45.7%/15.6% in its single pair.
Renewal p99 stayed below 0.865 s, with 900 successful renewals per arm
and no lost leases. The named pool caps stayed 10+2+2. An aggregate
activity sample peaked at 15 non-observer entries without backend-type
classification, so no claim is made that all were client connections. No
sampled lock episodes does not establish zero lock waits.

Twelve pure reader/selector cases also pass with the new adjacent
receipt. This is not another executed process rollback. Development
reader `5cf7aead` can parse the recovery state, but its old run-detail
redactor can expose receipt IDs/outcome to an authorized run reader. The
first production release containing recovery readers, complete
attachment-identity handling and both private-field redactions is the
rollback floor. The implicit-home fix prevents new recovery-time
fingerprint drift; it does not rewrite previously stored fingerprints
from failed candidates. The development probe is not a published
supported rollback target.

Latest actual `0862062` Runner tests independently passed 8/512/8,192
text chunks with zero recovery SQL, transactions and changed rows, plus
zero settled-idle SQL. All three observed seven setup/control
transactions and twelve changed rows. The 8,192-chunk case separately
recorded 31 legitimate lease-renewal statements, with no lost lease.
These are component SQL observations, not a production CPU/WAL claim.
The new adoption and identity branches affect recovery preparation only.
The `64c0b746`, `7a203274` and exact `39a4343` CI independently passed
all three stream sizes with the same zero-SQL/idle result; each latest
8,192-chunk case recorded 36 separate renewal statements. Exact
`39a4343` Runner 2/2 passed 133 files / 2,317 tests plus two skips; its
separate native group passed 150 tests plus eight skips.

The first API fixture revision omitted the mandatory final screenshot;
this was caught before paid launch and allocated no provider/model work.
The correction captures the actual retained Chromium page with an
explicit unqualified-browser label, without changing the standard report
validator.

Automatic approval review rejected exporting raw private API/workflow
snapshots because of sensitive payloads and insufficient
destination-specific authorization. Sanitized reports and
bound/hash-only receipts are retained; raw snapshots were not copied
through another channel. This does not change the failed grade.

Repair characterization also preserves its failed attempts. An
unconditional staging relocation broke three existing same-scope
scripted cases; the final implementation defers staging only for an
actual superseded warm scope, and all nine order/race cases pass. One
baseline transform overlapped an edit and is explicitly invalid; the
actual immutable `39a4343` loader then produced the three genuine REDs.
The independent task-directory hypothesis produced a RED but was
excluded by the live shared-workspace evidence; no directory policy
change was landed. A shared-workspace fixture authorization omission was
corrected separately from the four-skill semantic RED.

Setup failures are retained separately from semantic outcomes. These
include an old host port after the owned Postgres fixture restarted and
a sandbox-denied TSX IPC pipe. The supported external-Postgres run
passed, and the final exact recursive typecheck passes with normal
process permissions. Seven stale declaration errors after the master
merge cleared after compiling the actual merged SDK/Runner TypeScript
sources. No declarations were edited by hand.

<details>
<summary>Earlier qualification measurements and failed attempts
(historical, before this repair)</summary>

**Historical blocked gate (before the repair below).** Final native
implementation is `abed921e12d543b62f57a56435419c0dafb0bc33`; current
branch head is `af9e2d157dcc349e55374ad186c9be2d6cb96bf9`. The two later
commits change only qualification fixtures, their matcher and
documentation; `server/` and `packages/` are unchanged. It includes
master through `fd008979f5794fda3d5f54151d36acd31d0de140`.

The live Daytona measurement proves five-minute active recovery,
once-only completion and accounting, but **fails retained-session
identity after the second replacement**. The saved checkpoint and
recovery metadata exist. The task-session pointer remains empty, and
ordinary admission creates new native/provider sessions. The exact
skipped publication boundary remains unproven. A useful saved-file
result alone does not satisfy retained-session continuation. This
application-path gap blocks merge independently of CI.

A separate new-master CI failure is owned by the storage workstream (fix
in #18357). No storage guard, session-identity oracle, lease/grace limit
or provider result was relaxed.

| Gate | Observed result and exact scope |
| --- | --- |
| Distinct source recovery and compatible rollback | PASS at A
`5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → B
`5ea45f8d17e33c9a62bd79982dcf0375b07733d0` → fresh B → A. Four
controller processes, natural 60-second lease expiry, absent native
projections before replacements, three succeeded/committed unique
results, one provider thread, unchanged Runner identity, and strict idle
cleanup. 177.782 seconds. |
| Immutable installed native component | PASS at installed A
`5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → installed B
`eaef0d56552c087aaa4fa68a8655282ffbd12667` → fresh B → A. Each
controller imports its contained compiled server and DB. Three unique
committed results, stable Runner/thread, no extra provider restart on
same-contract rollback, strict cleanup. 106.672 seconds. This is
compiled component proof, separate from full CLI-native application
acceptance. |
| Normal public CLI startup | PASS for B
`eaef0d56552c087aaa4fa68a8655282ffbd12667`, Node 24.22.1, darwin-arm64:
fresh external DB, 415 migrations, health ready, root HTML 200, company
POST 201 plus GET readback. The installed tree remained unchanged after
execution. |
| Five-minute component outage | Actual controller absence 300,168 ms.
Natural generation-2 claim at 407.196 seconds after crash, same provider
attempt and Runner, original run succeeded/committed. The larger
540-second case timed out before its retained follow-ups, so it is an
active-adoption pass, not a complete four-controller pass. The complete
short source/installed cases above independently cover follow-up and
rollback. |
| Drain/resume review regression | Fixed in
`226f03e736d1ee99fea1762ab32ea3a8a9241c20`. Five facade-level cases
pass: overdue normal/restart waits, future timer cancellation, another
facade resuming the original owner, and transient retry. Against exact
prior scheduler `5ea45f8d17e33c9a62bd79982dcf0375b07733d0`, four fail
and one passes. The later head changes documentation only. |
| Final corrected-fixture Daytona | **PARTIAL: active recovery PASS;
retained identity FAIL**, exact live source `0b6afc499f`. A was
SIGKILLed and native cache discarded; actual offline hold 300,000.375
ms. B was ready at 308,326.961 ms, naturally claimed generation 2 at
312,059 ms, authenticated at 315,996 ms and resumed processing at
316,519 ms, within the original 420-second boundary. The same run,
Runner PID/start, session, lease and event prefix survived; one turn
submitted. Original execution succeeded and settled accounting. After
completion, B was killed and its native cache discarded; C was ready in
9,077.779 ms. C executed the saved project and returned correct
`queue_r`, with all four runs succeeded/accounted. However, native and
true provider-session identities changed: the unchanged retained oracle
fails. No whole-workflow pass is claimed. |
| Prior matched-producer Daytona | FAILED at exact `abed921e12`. The
matching current-source Runner passed normal admission, completed the
first task and settled its accounting. During the active revision, A was
SIGKILLed and its native cache discarded. Actual offline hold was
300,000.727 ms; B was ready at 308,304 ms and naturally claimed
generation 2 at 309,651 ms. At about 318 seconds, the remote process
check reported `process_not_running` for the original admitted Runner.
No authenticated after-receipt or retained follow-up passed. The current
failure is preserved; no automatic retry, lease/grace relaxation, debt
edit or provider replay is used. Bounded forensic read found preserved
exact-bound state with no recoverable failure, a transport-reset
diagnostic and no terminal delivery; this supports external process loss
without proving the exact cause. The sandbox had a five-minute auto-stop
policy, conflicting with the required surviving-sandbox assumption.
Owned cleanup precedes the separate corrected-fixture case. |

**Live retained-session boundary.** Prior completed run `3806…` and C
follow-up `8adf1df3-6744-40bf-b35d-100451f29d9f` both have database
session snapshots and `nativeControllerRecovery`. Their tool
fingerprint, provider configuration, workspace descriptor,
runtime-context digest, driver and work mode match. There is no recorded
force-fresh/workspace-selection change, and this is an ordinary task
rather than a conversation-generation reset. The same issue/agent
task-session row has `session_params_json: null` and `last_run_id: null`
from baseline time. Both native normalized UUID and actual provider
session/thread hashes differ after C. Full application admission
therefore did not select the retained source. No fallback to file
execution or relaxed identity comparison is accepted as proof of
continuity.

**Independent cleanup completed.** Full current-account provider
inventory confirms no resources owned by either final fixture, with
exact-ID absence checks. Owned controllers and campaign Postgres are
stopped/removed. EC2 `i-01551c09af098dde5` is terminated, its
delete-on-termination root volume returned `InvalidVolume.NotFound` at
22:14:08 UTC, and the temporary SSH key/agent were removed. The
canonical failed report and immutable oracle input are retained. Closed
qualification receipt SHA-256:
`b4b5fefce287ebfdbf2cc2238efaad62d8ad485aed0e9c0628ae95dbc18a8575`.

**Original measurement and grader repair remain separate.** The C reply
contains the requested marker followed by a newline and the correct
output. The old workflow waited for one literal space, so its normal
receipt collection stalled; that original attempt was closed and its
canonical `cleanup_failure`/interrupted result is preserved. Commit
`af9e2d157d` repairs only this reply predicate, with 144 focused cases,
E2E typecheck, discovery and diff checks passing. Ordinary string
matching stays literal. The corrected predicate accepts the saved reply,
but the unchanged retained-identity oracle still fails. Pure
active-recovery grading of the captured measurement passes 7/7; this is
not a rewritten passing campaign.

The sandbox idle-window conflict was corrected only for this explicit
fixture (15 minutes). Actual baseline and later sandbox metadata confirm
that policy; the original active box was disposed before a post-gate
policy read, so its 15-minute setting is established by the normal
create/environment configuration rather than an after-read of that box.
Production native lifecycle and the 60/300/420-second constants remain
unchanged.

The development compatibility reader A is a genuine backport commit
based on `59ff89a18fe1312b5f0bb908439bbdb1237b0252`, with the required
idle-goal reader fixes. It is not a previously published release. Its
older executor differs from B. The rollback floor must include this
reader capability. The source component cases admitted the Linux ARM64
Runner packaged at `59ff89a18fe1312b5f0bb908439bbdb1237b0252`; the
installed component case admitted the `5cf7aead` package. Their Runner
bytes are identical: SHA-256
`b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8`. Each
admitted artifact remained unchanged throughout its case.

[Final public package
build](https://github.com/paperclipai/paperclip/actions/runs/41121460635)
and [reader-floor
build](https://github.com/paperclipai/paperclip/actions/runs/41122170506)
both passed all four daemon builds, package assembly and both Darwin
clean consumers. Both Linux ARM consumers failed before provider work
with the same existing local Codex permission-profile error: bubblewrap
attempts `activity.fifo/.git` and reports “Not a directory.” This
matched baseline failure remains open; it is not a native remote
continuity pass. Full post-run tree and build-stamp revalidation also
passed after the compiled native case. The installed component trees are
A `c29114c1bf71c5af00f761aeb9e3d84ed3d3075b9f5600dee3ef2543eb2b6ec5` and
B `eda575505373a3bdc630fa4919862b359791fe3a7ef637961a25b73b38efe4b0`.

**Controlled PostgreSQL qualification passed.**
Baseline/candidate/candidate/baseline workloads used 300 controllers,
10,000 historical tasks, real snapshot/encrypted PRP writes and
unchanged pools. PostgreSQL and client were each limited to two CPUs and
1536 MiB on an isolated eight-vCPU Linux host.

| Workload | PG CPU per operation | WAL per operation | Database request
p99 |
| --- | ---: | ---: | ---: |
| Steady | +1.18% | +1.81% | +1.91% |
| Aligned burst | +3.40% | +2.56% | +3.81% |
| Two logical servers | +3.11% | +2.56% | +9.68% |

All arms passed absolute/correctness gates with unchanged control
transaction counts, no dropped operations and no owner loss. Overlap p99
has little margin below the relative threshold. It uses two clients in
one Node process, not two HTTP servers. [Closed measurements and
limitations](doc/plans/2026-10-30-native-controller-qualification-evidence.json)
and exact-source reproduction patches are retained. [Earlier
contended-host
failures](doc/plans/2026-10-30-native-controller-load-evidence.json)
remain unchanged. The production DB-claim probe claimed 300 owners in
2.84 seconds and restored/committed controls and snapshots in 8.74
seconds, with zero owner loss or settled-idle SQL; its provider
references were synthetic. The measured checkpoint/snapshot/renewal
paths retain the same persistence behavior after integration; later
accounting and business writes have separate owners.

Actual Runner stream tests at `da8c7f6e1837ddd896dfe83a7524812fee813487`
passed 8/512/8,192 chunks: zero recovery SQL, transactions and changed
rows during text intervals; zero settled-idle SQL. Each retained seven
control transactions and twelve changed records. Existing renewal SQL
was measured separately. The guard/scheduler fixes add no stream path,
pool or recurring idle scan.

The live fixture used current-source producer SHA-256
`f0000c90cb83880191479da639201e659aa6ce0fc7d67c859faa172ec977849e`, a
normal locked exact-head build with pinned Rust 1.97.1 in Bookworm.
Actual network-isolated, read-only ABI execution passed (GLIBC
requirement 2.34 against runtime 2.36). The immutable base image
remained
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:36069788f53c51f228764fa36180575443dfb5d1f84a342d7c0daceb28dc61cc`,
source `a0ef3ce7`. This was an unpublished development producer,
explicitly distinct from the base image and admitted before the fresh
fixture began. The earlier host build failed ABI preflight before
admission. The preceding old-producer fixture completed its initial task
but lacked the usage-basis marker required by current master; it
remained pending/unpriced and correctly paused. It was independently
cleaned without debt or budget changes. Two preceding local Linux setup
attempts OOM-killed before provider allocation, and their cleanup
passed. These attempts remain separate from the final recovery failure.

**Failures retained and resolved separately.** The earlier `98f05fc`
probe rejected a valid settled PRP v2 goal suffix; `eaef0d5` fixed that
reader and the fresh follow-up passed. Later component cleanup failures
were traced by a scoped exception observer to the fixture adapter
rejecting directory downloads. A bounded owned-directory transfer with
exact-byte/mode preflight corrected the fixture; source and installed
strict-cleanup cases then passed. No production guard was relaxed. The
initial local integrated control-port run hit its 30-second embedded
Postgres startup hook before executing 59 cases; 18 other focused cases
passed. A bounded rerun used a fresh owned external Postgres fixture,
applied normal migrations before Vitest, and passed all 59 cases with
the original timeouts unchanged. Its container and data were removed;
the failed attempt remains retained.

The additional opt-in full Product E2E Vitest run is not green: 5,313
passed, seven failed and three skipped. An actual `abed921e12` archive
with the same dependencies, Node and IPC permissions reproduces all
seven failures with the same reasons. Five are unchanged
workflow-policy/profile-pin assertions, including Pi profile version 37
versus the test pin 35. Two database suites failed during embedded
Postgres startup before their assertions; semaphore exhaustion is
confirmed in the observer log. The observer passes its supported
external-Postgres rerun with original timeouts and source unchanged. The
controller-database client-kill/postmaster assertions remain unexecuted
after `initdb` bootstrap failure; no external override is supported. The
unit launcher stopped before its Node-test phase, which was then run
independently and passed 181/181. The original failed full run is
preserved. These results remain distinct from the passing focused checks
and normal repository CI. The first local Product E2E typecheck also
encountered stale compiled Runner declarations; a genuine package-local
TypeScript compile refreshed them and the unchanged check passed,
without a dependency install or producer rebuild.

Earlier [Daytona campaign
41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900)
failed at sandbox creation before any agent, with the underlying
provider error unavailable. Its [sanitized
report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-41108162900-2/index.html)
remains a `cleanup_failure`. Both old creation names are absent in the
current authenticated account, but the old CI account identity was not
retained, so account equivalence is unproven. The subsequent exact
`eaef0d5` source Product E2E attempt passed image/agent setup and the
initial project, then held the controller offline for 300,003.7 ms. Its
Mac supervisor failed owned-process inspection before B readiness; no
adoption passed. Full current-company lease/label checks, provider
deletion, exact-ID 404s and complete inventory confirmed cleanup of that
attempt. Its local server and owned Postgres stopped. The prior AWS
campaigns and database qualification VM were terminated with root
volumes removed.

The old admitted test Runner lacks the newer usage-basis marker
introduced on master. The integrated component B turns retained
unpriced, incomplete accounting obligations with `costAccountingPending:
true` and no settled charge. The compatibility-reader A turn used its
legacy receipt interpretation but likewise retained pending accounting
and no settlement timestamp. This proves obligation preservation, not a
whole-run accounting migration. Runner/receiver release ordering remains
governed by the usage owner's contract.

At native implementation `abed921e12d543b62f57a56435419c0dafb0bc33`,
Greptile was 5/5 with zero unresolved threads and CI completed with 54
successful checks and five expected skips. At `0b6afc499f`, Greptile
again passed 5/5 with zero unresolved threads, but CI ended with 51
successful checks, five skips and three failures. Server 9/12 fails a
deterministic managed-skills maintenance assertion newly inherited from
master `198218bf` in merge `c80e00ad`; that test is absent from the PR
head, and the storage owner has the explicit-StorageService fix in
#18357. Runner 2/2 passed 2,283 unit cases but its runner received a
shutdown signal during the long stream probe; its cancelled step and
aggregate failure are preserved. No blind retry was used. Current
`af9e2d157d` checks are pending; the matcher-only change passes 144
focused cases and Product E2E typecheck. Earlier scheduling-sensitive CI
failures and the interrupted unsharded local test run remain recorded in
the dated plan. CI runs the repository's normal test partitions. Final
verification does not claim Railway rollout, uninterrupted HTTP service
or whole-webserver filesystem loss.

</details>

## Risks

- The task-pointer repair passes its assembled regression. The first
live repair exposed pre-acceptance ordering and the fresh review exposed
a replay lock-order issue; both are fixed and pass real Postgres tests
at `0862062`. The following live attempt exposed the
verified-running-versus-suspended preparation gate. Its narrow fix
passes 91 focused cases at `64c0b746`. The next live case exposed
synthetic attachment IDs; `7a203274` fixes that field selection and
passes real Runner authentication. The subsequent queued continuation
exposed strict retirement maintenance and a false default-home change;
`39a4343` fixes both with real Runner/Postgres and fingerprint
regressions. The `39a4343` live continuation failed again despite green
CI. `283e05ffaa` repairs the actual native-observed admission ordering
and reproduces/fixes the JSONB skill-order fingerprint drift. Both pass
targeted real-Postgres regressions and the live active plus queued
paths. The definition 6 post-C browser navigation timed out before
admission. Explicit definition 7 now verifies native continuation
through the ordinary comment API; its strict post-C native gate failed
after a real dispatch and remains historical failed evidence. The final
fc68584 repair now passes its fresh strict 49-check Linux/Daytona
campaign; the prior failed grades remain unchanged. Browser reconnection
is handed to its own workstream with the original failed evidence. No
prior failed campaign is reclassified as passing.
- Incorrect checkpoint/source binding or stale ownership could duplicate
external work. Missing evidence remains a hold; existing receipt and
finalization owners keep uncertain effects unresolved.
- Recovery must retain maintenance writes after accepted results without
allowing a stale business turn to write. Real lock-wait and generation
tests cover that boundary.
- The current two-connection paired burst profile fails its absolute
drain gate in both baseline and candidate, despite passing relative load
gates and all correctness checks. A causal capacity diagnostic selected
three slots; the final controlled cap-three profiles pass unchanged
thresholds. Full application replacement and two-server overlap capacity
remain separate. There is no new streaming writer, pool or recurring
idle scan.
- The remote sandbox and Runner must survive the recovery window.
Provider auto-stop is a separate lifetime bound from reconnect grace;
the explicit qualification fixture uses a longer idle window.
- Workspace and provider backup availability remain external
dependencies. This PR does not certify whole-webserver filesystem loss,
uninterrupted HTTP service or a Railway rollout.

## Model Used

Implementation and initial verification used OpenAI GPT-6 through Codex;
the initial hosted snapshot was not exposed. The follow-up qualification
used three OpenAI `gpt-6.1-sol` workers coordinated through Codex, with
repository tools, code execution, CI and scoped test infrastructure.
Context-window size is not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused and exact-source
application/cost checks pass; full forced CI passes; earlier local
embedded-PG setup failures remain recorded)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (exact fc68584 full CI and
auxiliary checks pass; forced complete test partitions)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(exact fc68584 check and edited summary pass; all six threads resolved)
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-31 07:33:52 -05:00
DottaandPaperclip 2d59dd1ba3 Merge qualified sleep review fixes and prerequisites 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '74970cc21e2e5e1881516228bc7c7dbbba881a63':
  fix(retention): wake blocker retention after no-run workspace export
  Bound blocker workspace history reads by candidate workspace
  Validate eligibility before first attention retention materialization
  fix(retention): exclude finalized blockers before graph bounds
  test: hold lifecycle verification before retry identity assertions
  Exercise saved legacy input through its eligibility producer
 2026-10-31 02:01:55 -05:00
DottaandPaperclip 74970cc21e Merge qualified sleep review fixes and prerequisites 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '5726ecf7a39339c2d68b094d9bbe1b1d0c436845':
  fix(retention): wake blocker retention after no-run workspace export
  Bound blocker workspace history reads by candidate workspace
  Validate eligibility before first attention retention materialization
  fix(retention): exclude finalized blockers before graph bounds
  test: hold lifecycle verification before retry identity assertions
  Exercise saved legacy input through its eligibility producer
 2026-10-31 02:01:53 -05:00
DottaandPaperclip 5726ecf7a3 Merge qualified sleep review fixes and prerequisites 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit 'f07e5cf343785503266d5a3a06a7c3ad31d163bb':
  test: hold lifecycle verification before retry identity assertions
  Exercise saved legacy input through its eligibility producer
 2026-10-31 02:01:52 -05:00
DottaandPaperclip a0e2a3515d fix(retention): wake blocker retention after no-run workspace export 
Register finalization intent before state writes, preserving caller transactions. Shared workspace export invalidates only the company blocker source; known issue-only export stays scoped and unattached operations do not activate retention. Add actual recorder controls and the shared-root integration journey.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 02:00:39 -05:00
DottaandPaperclip a53f22861f fix(retention): exclude finalized blockers before graph bounds 
Share the dependency owner finalization predicate with bounded blocker selection, resolve relation identities before correlated checks, and refuse automatic archives from incomplete fresh graphs. Cover completed history, live evidence and primary-key lookup work.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 00:40:30 -05:00
DottaandPaperclip f07e5cf343 Merge qualified recovery browser and lifecycle fixtures 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '26275cae7f301e8206ce9b3b226f64f7b985123c':
  test: hold lifecycle verification before retry identity assertions
  Exercise saved legacy input through its eligibility producer
 2026-10-31 00:33:47 -05:00
DottaandPaperclip bf9eebe5bd Exercise saved legacy input through its eligibility producer 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 00:29:28 -05:00
DottaandPaperclip 4d3e046e88 fix: cancel merge confirmation acceptance under lock 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 00:26:12 -05:00
DottaandPaperclip 420206ce0a Merge qualified recovery grouping and prerequisites 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '0b081d9ffebaacc51190a6217dcbbfdffe032acc':
  Preserve native receipt grouping within each bounded worker pass
 2026-10-31 00:06:48 -05:00
DottaandPaperclip 0b081d9ffe Merge qualified recovery grouping and prerequisites 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '2aa0179c35564fae529eb8053f0ed098f60dcc64':
  Preserve native receipt grouping within each bounded worker pass
 2026-10-31 00:06:47 -05:00
DottaandPaperclip 2aa0179c35 Merge qualified recovery grouping and prerequisites 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '0cb21925bf9901d818d456b8621ebf58e82ec119':
  Preserve native receipt grouping within each bounded worker pass
 2026-10-31 00:06:47 -05:00
DottaandPaperclip 0cb21925bf Merge qualified recovery grouping and prerequisites 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '6e8119550dc3dccafa165d5aebed21bd43bb4ee3':
  Preserve native receipt grouping within each bounded worker pass
 2026-10-31 00:06:47 -05:00
DottaandPaperclip 6e8119550d Preserve native receipt grouping within each bounded worker pass 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 00:06:19 -05:00
Dotta cb2e694c7c Merge commit 'b4b18c9a37188c912507e18feae9677ba1b21ffa' into codex/sleep-final-qualified-18382 
* commit 'b4b18c9a37188c912507e18feae9677ba1b21ffa':
  test: align deferred recovery clocks with real execution
  Keep recovery acceptance timers on their owning clock
  fix(server): recheck reactive sources after status summary commit
  test: measure retention page fairness on a controlled clock
  fix(heartbeat): resume saved chat work on feature enablement
  test: await resolved interaction execution before outcome assertions
  Wake saved conversation assignments when Agent Chat becomes enabled
 2026-10-30 23:27:04 -05:00
Dotta b4b18c9a37 Merge commit '96b516dcb638cede15dae7d6488c716c5a4df18f' into codex/sleep-final-qualified-18399 
* commit '96b516dcb638cede15dae7d6488c716c5a4df18f':
  test: align deferred recovery clocks with real execution
  Keep recovery acceptance timers on their owning clock
  fix(server): recheck reactive sources after status summary commit
  test: measure retention page fairness on a controlled clock
  fix(heartbeat): resume saved chat work on feature enablement
  test: await resolved interaction execution before outcome assertions
  Wake saved conversation assignments when Agent Chat becomes enabled
 2026-10-30 23:27:03 -05:00
Dotta 96b516dcb6 Merge commit '499afce285c429f814f5cd22fd437bafabf530e7' into codex/sleep-final-qualified-18388 
* commit '499afce285c429f814f5cd22fd437bafabf530e7':
  test: align deferred recovery clocks with real execution
  Keep recovery acceptance timers on their owning clock
  fix(server): recheck reactive sources after status summary commit
  test: measure retention page fairness on a controlled clock
  fix(heartbeat): resume saved chat work on feature enablement
  test: await resolved interaction execution before outcome assertions
  Wake saved conversation assignments when Agent Chat becomes enabled
 2026-10-30 23:27:02 -05:00
Dotta 499afce285 Merge commit 'a0f5caa00241e9f2653992f47106bbe38fec130f' into codex/sleep-final-qualified-18401 
* commit 'a0f5caa00241e9f2653992f47106bbe38fec130f':
  test: align deferred recovery clocks with real execution
  Keep recovery acceptance timers on their owning clock
  test: measure retention page fairness on a controlled clock
  fix(heartbeat): resume saved chat work on feature enablement
  test: await resolved interaction execution before outcome assertions
  Wake saved conversation assignments when Agent Chat becomes enabled

# Conflicts:
#	server/src/services/instance-settings.ts
 2026-10-30 23:26:43 -05:00
DottaandPaperclip a0f5caa002 Merge qualified issue recovery and test prerequisites 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '5c1ef3b5f4a49e2e45d915e66bd280ad34e0b863':
  Keep recovery acceptance timers on their owning clock
  test: measure retention page fairness on a controlled clock
  test: await resolved interaction execution before outcome assertions
  Wake saved conversation assignments when Agent Chat becomes enabled
 2026-10-30 23:24:40 -05:00
DottaandPaperclip 005310cd0d Keep recovery acceptance timers on their owning clock 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 23:20:31 -05:00
DottaandPaperclip 09a70c3bef fix(server): recheck reactive sources after status summary commit 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 22:55:58 -05:00
DottaandPaperclip e43163bfaf Wake saved conversation assignments when Agent Chat becomes enabled 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 22:36:47 -05:00
DottaandPaperclip 59da15066a feat(runner): deliver native Kimi Code (#18391) 
Deliver native Kimi Code setup, exact models, scoped credentials, verified packages and runtime packs. Preserve current upstream release attestations and restore verified same-run remote retry dispatch without weakening process ownership.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 22:26:15 -05:00
DottaandPaperclip 468285e0b6 Merge verified sleep prerequisites into 05 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '5bc8bef9f4c37637e90ea51a2aa7e6fa2c486dbe':
  Remove unused aggregate watchdog diagnostics
  test: settle delegated chat completion before plan assertions
  test: settle saved retry cleanup before idle assertion
  fix: wake retry scheduling only when retry state changes
 2026-10-30 22:12:53 -05:00
DottaandPaperclip 5bc8bef9f4 Merge verified sleep prerequisites into 03 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '98a68b492f933890da59e2ec2f9ddd31f997d5b5':
  Remove unused aggregate watchdog diagnostics
  test: settle delegated chat completion before plan assertions
  test: settle saved retry cleanup before idle assertion
  fix: wake retry scheduling only when retry state changes
 2026-10-30 22:12:53 -05:00
Dotta 98a68b492f Merge verified sleep prerequisites into 01 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit 'd287074fba0ab0365d61416dadc392cc3078eee5':
  Remove unused aggregate watchdog diagnostics
  test: settle delegated chat completion before plan assertions
  test: settle saved retry cleanup before idle assertion
  fix: wake retry scheduling only when retry state changes

# Conflicts:
#	server/src/services/heartbeat/retries.ts
#	server/src/services/native-runtime/native-provider-capacity-retry.ts
 2026-10-30 22:12:53 -05:00
Dotta d287074fba Merge verified sleep prerequisites into 07 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '78bd3560a886c7fd3eea03baccb6cf49f6e8499a':
  Remove unused aggregate watchdog diagnostics
  test: settle delegated chat completion before plan assertions
  test: settle saved retry cleanup before idle assertion
  fix: wake retry scheduling only when retry state changes

# Conflicts:
#	server/src/services/heartbeat/retries.ts
 2026-10-30 22:12:31 -05:00
DottaandPaperclip 100b364264 Preserve selected Google services when setup resumes (#18048) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connections control which external services an agent can use.
> - Google Workspace setup lets a person choose services and read or
write access.
> - A draft lost that choice after a declined or expired OAuth handoff.
> - Finish setup could then show all services with write access instead
of the saved narrow choice.
> - This pull request saves the exact draft choice without treating it
as granted access.
> - The benefit is safe setup recovery that preserves the person's
choice and existing account limits.

## Linked Issues or Issue Description

Refs #15545. The duplicate search found no open PR for Workspace draft
consent recovery. Related #18000 concerns satisfied gateway access
requests, not saved OAuth setup intent.

**What happened?**

A cold Workspace draft resumed with the new-account service defaults.
The selected request lived only in short-lived OAuth state. Denial
consumed that state. The picker ignored draft connections.

**Expected behavior**

Finish setup restores the same person's saved service choice. It does
not use another person's choice, a broad profile hint, or a credential
grant as draft intent. An old draft with no reliable choice asks the
person to choose services.

**Steps to reproduce**

1. Start a new Google Workspace connection. Select Gmail read-only. Turn
off Drive and Calendar.
2. Continue to Google, then decline authorization.
3. Close the setup page. Return through Finish setup.
4. Open the service picker. Before this fix, all three services can be
enabled with write access.

**Paperclip version or commit**

Reproduced in tests on master
`820c458465fa40137c81fe510f3e6dc573b4f895`.

**Deployment mode**

Authenticated and local source builds. The source regression uses real
PostgreSQL fixtures and the production React setup components. Live
provider acceptance of this repair is still pending deployment.

## What Changed

- Save validated, non-secret draft intent before managed or
customer-owned OAuth. Bind it to company, connection, actor, grant kind,
and subject.
- Restore that intent after denial, state expiry, and cold Finish setup.
Keep active reconnect bound to its saved grant. Do not change
new-account defaults.
- Preserve agent-to-human setup handoffs. Consume only the addressed
person's reviewed proposal. Keep other people's proposals.
- Lock and merge the current draft before saving intent. Return 409 if
its credential identity or method changed during setup.
- Preserve the latest saved choice across overlapping customer-owned
starts. Merge both current configuration records under short locks.
Reject stale client-binding changes. Keep each OAuth state's own
requested scopes.
- Add shared, service, handoff, picker, and cold-page regression tests.
Update the connector contract and existing feature plan.

- Route the default ungrouped test/release paths through six sequential
exact-cover chat partitions. Preserve remaining server/native coverage,
explicit grouped CI, selection-first execution and shared-database
paths.
- Correct the retention test driver so pending owned real I/O does not
advance logical deadlines. Pump zero-duration connection timers and add
a held-I/O regression; keep production scheduling and original timing
assertions unchanged.

The shared helper is needed by the server and picker to validate the
same narrow binding. It does not resolve credentials or grant access. No
production schema migration, new route, provider scope or deployment
configuration is added. The test-harness changes do not change
production timers.

## Verification

### Current candidate: source qualification passed; live acceptance
pending

- Current published candidate:
`82d13d4d698a5579378b5d506cb7a1f9cb64b5c8`, tree
`78d6a821ce53f481fe3e9991edce56f0acffb22f`. It combines the Workspace
repair with launcher commit `bf29c9` and retention-test commit `e25987`,
then integrates master `493e229d93ba7798e80152f4549e2e9177ce50ae`. The
integration brings 40 upstream commits across 713 files. One launcher
conflict keeps upstream validated CI selection and its selection-first
path, then applies six default chat partitions only without a selection.
The authored PR diff has 16 files, 833 additions and 100 deletions; it
adds no authored lockfile or `.github` changes. The public head is
confirmed at this revision. Fresh CI and review passed as recorded
below. Source qualification is complete using exact-tree full CI and the
local results below. The separate local whole command was interrupted by
host disk protection and is not called a pass. Live repaired-draft
acceptance remains pending deployment.
- Exact clean `82d13d4` passed one Node 24.19.0 builtin-only run: 41/41
cases (37 launcher and four CI-workflow cases), zero failures/skips.
HEAD, tree and relevant file hashes stayed unchanged and clean before,
at start and after. This proves launcher fixtures, not real chat
execution, current typecheck/build, the dependency-backed selection
graph, or full feature acceptance.
- The new default ungrouped whole command uses six sequential
exact-cover chat partitions, then the remaining server suites,
workspaces and serialized suites: 28 expected invocations. It preserves
all native suites, including upstream usage coverage. The plain
ungrouped release launch path also changes. Explicit grouped CI,
validated selection and shared-database execution stay unchanged.
Launcher hard RED had three expected body failures and two unchanged
passes; the repaired focused five and final 37 passed. A later 41-case
staged-merge pass is retained separately; the clean 82 run above is the
frozen proof.
- Fresh [CI run
41153113944](https://github.com/paperclipai/paperclip/actions/runs/41153113944)
completed successfully on October 31 at 01:14:53 UTC. Current head has
52 successful check runs, four conditional skips and no pending/failing
checks; legacy Snyk also passed. The skips are test-file selection, the
optional paired benchmark and two Storybook checks. The full
server/chat/workspace/serialized matrix, all eight e2e shards, normal
build, build-gap typecheck, Runner checks and final aggregate gates
passed. Build and all three chat job logs identify checkout
`e5e831fd3a8534d0d0f1e892118c0f8fbe12dd92`, merging this head into
`493e229d93`. Its Git tree is exactly
`78d6a821ce53f481fe3e9991edce56f0acffb22f`, identical to the clean
candidate. The six real chat partitions passed 314, 358, 273, 239, 321
and 311 cases: 1,816 total. Other cases shown as skipped in each
partition were selection exclusions, not extra passed cases. This is
current tree-equivalent CI proof, not a pass of the separate default
ungrouped local whole command.
- Fresh [Greptile
summary](https://github.com/paperclipai/paperclip/pull/18048#issuecomment-6473856992),
edited October 31 at 01:05:04 UTC, explicitly reviews exact head
`82d13d4` at 5/5 with no actionable code issue. Its exact-head check
passed. The prior overlapping-start discussion is resolved. This review
is source-only, not provider or live acceptance.
- Exact clean `82d13d4` passed full recursive typecheck and normal `pnpm
build` on the independent darwin/arm64 snapshot. Typecheck finished on
October 31 at 01:27:33 UTC; build finished at 01:28:33 UTC. Both exited
0. Normal native preparation and code signing passed. Source HEAD, tree
and tracked/untracked state stayed unchanged and clean. `pnpm
check:token-gates` also passed all token, palette and ClipLab checks on
this candidate. The earlier forced-clone `ENOSYS` preflight remains
recorded as setup failure, not an assertion result.
- The unchanged default `pnpm test:run` started at 01:51:33.796 UTC on
clean exact `82d13d4`. All six real chat partitions passed: 358, 314,
273, 239, 311 and 321 cases, or 1,816 actual cases. The original FIFO
and ambiguous-retry cases both passed under their unchanged 15-second
limits. General-server then reported 37 completed file rows with 5,438
passed cases and five default remote-continuity skips, including all 498
tool-access service cases and all 83 connection-route cases. At 02:39:44
UTC, the unchanged 5-GiB host-disk guard terminated the owned run. The
child ended with SIGTERM and the container/harness exited 143, with no
OOM or recorded assertion failure. Only seven of 28 RUN invocations
started. Retention had no completed local row. Later invocations did not
run. This whole local command is interrupted/incomplete, not passed.
- A coverage check found no default suite missing from current full PR
CI. The maintained launcher tests prove exact coverage for all 12 PR
server shards, chat and five native suites; all configured workspace
projects; and all nine serialized shards. The final Runner CI shard
executes the five native server suites. The 41 local launcher/workflow
cases cover six sequential chat children, per-child isolation, native
retention, fail-fast propagation and unchanged explicit
selection/shared-database paths. The actual six-chat run above exercises
that changed default path. The remaining unqualified local difference is
uninterrupted sequential whole-command composition and resource
accumulation, not omitted test selection. Current exact-tree full CI
supplies broad source-test qualification; no local whole-pass claim or
provider acceptance is inferred.
- A task-owned remote runner was considered to complete that separate
composition check. Its provider rejected the requested allocation with
per-sandbox limits of 8 GiB RAM and 10 GiB disk. No sandbox was admitted
and no remote tests ran. No tests, timeouts, assertions, database
synchronization, source or provider settings were changed to avoid the
capacity limit. Historical a0 failures remain below. No deployment or
live repaired-draft acceptance has run.

### Bounded follow-up on the prior source

- On a0, one finite phase-traced chat diagnostic executed exactly the
two failed cases. Both passed (1,374 and 1,767 ms); 1,814 other cases
were filtered, not passed. This did not reproduce a blocked await or
establish the original monolithic timeout cause. No assertion or
15-second limit changed.
- The retention diagnosis preserved an initial zero-body selector
failure, then corrected only the selector: one original case passed in
783 ms, with 42 filtered. Trace showed the old settling helper advancing
logical time while real I/O was active. A new held-I/O regression then
failed hard because the old helper advanced 1,000 ms. This proves a
test-driver defect, not the precise cause of the old 5,340 ms full-run
gap.
- The first manual-clock repair failed the complete 44-case file: 43
passed, one body hit 15 seconds and cleanup hit 30 seconds. Its planned
mutant did not run. Source then identified a fresh PostgreSQL
connection's zero-duration timer. The corrected driver pumps
zero-duration timers while initial readiness or tracked I/O is pending,
without advancing logical Date. That pre-integration file passed 44/44
with no skips. A separate scratch-only 5,000 ms normal-continuation
mutant failed the original hard `5000 < 3000` assertion; its expected
failure is not a test pass. The original page-count, fairness, expiry
and quiet assertions stayed intact. The authored repair does not change
the production worker. The merged master changes an older retention case
and dependencies; these pre-integration results are not a complete 82
retention or whole-suite pass.

### Historical qualification: SHA-bound, not 82 acceptance

- Historical a0 candidate: `a0f9089fc1ff466bf4e1eeddc5b1a910d09e380b`,
tree `b9d73713319dcd52b9eaa6053008b5190e43fdf1`. It adds the
overlapping-start repair to parent `6c18`, which already merged pinned
master `1511c92965325249ed6b52dd5be52eb6d736fbfa`. That candidate
changed 12 files. Its CI and review passed; its local full qualification
remained failed/incomplete.
- Greptile's previous `6c18` review found P1 [Latest service choice gets
lost](https://github.com/paperclipai/paperclip/pull/18048#discussion_r4425965828),
confidence 4/5. Two hard-assert regressions reproduced it on untouched
`6c18`: the already-bound path overwrote the later choice, and real
first-use client stamping also restored the old choice. Existing OAuth
states/grants were not widened, but a later setup could restore an older
request. The new repair covers both write boundaries without holding
locks over external work or copying another attempt's scopes.
- The complete affected service file passed 498 tests with no failures
or skips. All 31 required controls ran, including both unchanged RED
cases, 14 new client-drift cases, seven earlier identity guards,
reconnect/cold-resume cases and upstream transaction tests. The tested
three-file patch is byte-identical to this commit; the connector
contract update is separate. Independent scoped review found no
additional actionable issue. Fresh Greptile reviewed exact `a0f` at 5/5
with no actionable findings. The previous P1 thread is resolved.
- On exact `a0f`, the five shared/UI files passed 312 tests. An initial
complete connection-management run then failed: 74 passed, eight timed
out at the unchanged 15-second limit, and one AI-account verification
returned 422 instead of 201. The Google addressed-user handoff passed.
The container exited 1 without OOM and stopped before chat/token/whole
tests. This material failure remains retained and its cause
unclassified; the later controlled recovery below is not a causal
explanation.
- One diagnostic executed the first failed gateway case and the two AI
cases with temporary phase-only tracing. All three passed at 8.516,
5.511 and 3.523 seconds; 80 other cases were filtered. Original hard
assertions and timeouts stayed unchanged. No timeout or suspected late
provider cleanup reproduced, so the complete-file failure remains
unresolved. Vitest exited 0; the harness exited 1 because its trace
parser missed ANSI prefixes. All 203 events were recovered from retained
logs without rerunning tests. Instrumentation was removed. This is not
acceptance.
- Historical a0 CI
[41097596364](https://github.com/paperclipai/paperclip/actions/runs/41097596364)
completed successfully: 52 successful checks, two expected Storybook
skips and no failed/pending checks. Legacy Snyk passed. All 48 CI jobs
passed, including all 1,816 chat cases and explicit passing lines for
the original 20 local failures. [Server
11/12](https://github.com/paperclipai/paperclip/actions/runs/41097596364/job/124207469276)
explicitly passed the complete 83-case connection file in 129.49
seconds, including all nine local failure titles. The actual checkout
was synthetic merge `06851da6eb969d63e1c7dc9f7c95f08d09fa4ff1`,
combining `a0f` with `aa58fa107389e8b9ae5949f7ad815943dfd94292`. This is
not standalone local `a0f` qualification and does not erase the nine
local route failures.
- Linux admission uses Node 24.22.2, pnpm 9.15.4 and Rust 1.97.1. Frozen
offline install, exact native hashes and real migrated PostgreSQL 18.1
passed. Native/protocol/build/lock inputs are identical between `6c18`
and `a0f`; validated cache reuse retains the actual `6c18` native-build
provenance. The selected packaged release binary is not called a fresh
`a0f` build. Database synchronization settings stayed on. This admission
alone does not explain the route failures.
- A fresh Mac snapshot of exact `a0f` passed frozen install, workspace
prerequisites, full sequential recursive typecheck and normal `pnpm
build`, both on their first attempt. Typecheck completed at 17:43:05
UTC, build at 17:44:43 UTC; both exited 0 with clean tracked/untracked
source. Pinned Node 24.19, pnpm 9.15.4, Rust 1.97.1, package concurrency
one, Cargo jobs one and a 12 GiB V8 heap ceiling were retained.
Largest-process maximum RSS was 7,668,154,368 bytes for typecheck and
10,038,837,248 bytes for build, not aggregate tree peaks. The first
build correctly withheld its freshness stamp because 125 ignored, pinned
Copilot assets appeared during preparation; exact path/hash analysis
explained this. The normal build then recorded a genuine stamp and
reused Cargo outputs in 1.18 seconds. No manual stamp, source workaround
or gate retry. This is darwin/arm64 static/build proof, not Linux
packaging/runtime or full tests.
- The historical a0 candidate's token checker passed all four gates
across 1,410 files, run directly with pinned Node 24 against the script
behind `pnpm check:token-gates`.
- The single controlled complete-file comparison passed: baseline `6c18`
83/83 in 287.07 seconds, then candidate `a0f` 83/83 in 293.03 seconds,
with zero skips. All nine earlier failure titles passed on both. Both
child/container exits were 0, no OOM/interruption; source and
native/resolver/PostgreSQL guards matched. This ran only after all owned
Mac compilation ended, with fresh processes/databases and identical
four-CPU-quota/6-GiB/overlay inputs. No filtering, instrumentation,
assertion/timeout/source changes or retry loop. This establishes
standalone complete-file recovery, not the cause of the earlier eight
timeouts/AI 422.
- The first unchanged full `pnpm test:run` on exact clean `a0f` ran from
October 30 at 18:01:43 UTC to 21:06:19 UTC. It failed naturally with
test and container exit 1, without OOM. General-server reported 1,257
passed, one failed and seven skipped files; 28,633 passed, one failed
and 103 skipped tests. Only the first of 22 invocations ran. The later
four groups and 21 invocations did not run. The sole failure was the
unchanged chat-shard CLI meta-test: its nested projectless pnpm lookup
hit the offline network guard. This was a failed full gate, not Google
provider acceptance.
- A finite environment pair reproduced that one meta-test failure, then
passed the same unchanged case after supported `corepack prepare
pnpm@9.15.4 --activate` as the existing non-root node user. The cached
pnpm payload was identical. Only the missing node-user last-known-good
record changed. Core source, dependencies, lockfile, fixtures,
assertions, timers and offline guards stayed unchanged. This isolates
the nested collection defect; it does not explain the earlier route
failures or complete the full gate.
- The corrected unchanged full command ran from October 30 at
21:29:03.500 UTC to October 31 at 00:02:44.207 UTC on the same exact
clean `a0f`. It failed naturally with exit 1, signal null and no OOM.
Source stayed clean before, at start and after. It retained two visible
CPUs/quota, 6 GiB, private IPC, normal overlay, matching native inputs,
synchronized real PostgreSQL 18.1, and the default five groups/22
invocations. Only the first general-server invocation ran; the other 21
did not run. General-server reported 1,265 files: 1,256 passed, two
failed and seven skipped. It reported 28,737 tests: 28,631 passed, three
failed and 103 skipped. The two chat cases emitted `Test timed out in
15000ms` at `chat-channels.integration.test.ts:77310` and `:81790`, with
durations of 15,045 and 15,058 ms. These were the durable pre-run FIFO
notice and ambiguous GitHub text restart/retry cases. The retention case
emitted `expected 5340 to be less than 3000` at
`skill-version-retention.test.ts:593`. Its real duration was 1,727 ms;
5,340 ms was the recorded fake-time directory-page gap, not a deadline
timeout. The timeout stacks do not identify the blocked internal await.
The cause of these failures remains unclassified. Inside this same
failed run, the complete 83-case connection file and 498-case service
file passed. The former Corepack meta-case also passed in 6,396 ms.
These component passes do not waive the failed/incomplete full gate. No
rerun, cancellation, timeout/assertion change or full-pass claim. A
later fresh two-case phase trace passed both chat cases; it did not
reproduce or establish the original timeout cause. At that checkpoint,
the PR stayed draft.
- Prior `6c18` targeted checks passed 930 tests across eight files and
token gates. Its selected queue group passed all 53 cases, including the
original 20 failures and 12 new named controls. Backlog cases took 10.49
and 8.45 seconds under the unchanged 15-second limit. Another 1,763 chat
cases were intentionally filtered. This is historical targeted recovery,
not a full pass or proof of the new repair. No timeout or assertion was
relaxed.
- Prior `6c18` full local qualification failed during serial typecheck:
server TypeScript was killed with code 137. Docker confirmed OOM in the
6 GiB container; full tests and build never started. The failure is
retained. The new Mac resource plan avoids repeating that undersized run
and does not change the shared Docker VM.
- Prior CI
[41089793558](https://github.com/paperclipai/paperclip/actions/runs/41089793558)
completed successfully for synthetic merge
`a73bffcfa3ede051961bcabcfa576c703d10c932` (`6c18` plus
`abc270ee4c03e13b4d6eaad156a41b4ebed4bb0a`). All 48 CI jobs passed; all
1,816 chat cases passed, including explicit log lines for the original
20. Check runs were 51 success, two expected skips and one Greptile
failure; legacy Snyk passed. These results do not qualify `a0f` or erase
its local route failures.
- Red proof on the unmodified master baseline: cold picker and Finish
setup restore broad defaults; managed and customer-owned drafts lack
saved intent. The stale setup tests also fail before the lock guard.
- Live staging baseline `b25f3e2af8e77d882cd22d36ca55640e6468e46f` also
reproduces the ordinary failure: select Gmail read-only only, cancel at
Google, reload the connector catalog, then Finish setup. All three
groups return enabled at read/write. Authorization was declined; no new
Google access was granted. This is baseline failure evidence, not a
deployed-fix pass.
- Historical affected suites: 800 tests passed before the final lock
guard. The final focused service and handoff cases passed 126 tests
after that guard. These are not results for the integrated candidate.
- Historical `pnpm -r typecheck` and final lock-guard server typecheck
passed. The historical a0 candidate passed the full recursive gate
separately, as recorded above.
- `pnpm build`: passed again on the final lock-guard source in commit
`1c31cde0ed199fbf4e41b7649590554f7886dfc2`.
- Historical `pnpm check:token-gates` passed; the historical a0
candidate's direct checker pass is recorded above.
- `pnpm test:run` on macOS failed on the frozen commit. It exited 1
after 9,432.41 seconds. General server results: 1,195 files passed, 10
failed, eight skipped; 26,706 tests passed, one timed out, 157 skipped.
Nine setup hooks timed out while starting embedded PostgreSQL. The
retained log records host-wide semaphore exhaustion (`semget`), not disk
exhaustion. One unchanged publication-text performance case also timed
out. Its cause is not established. Later workspace and serialized groups
did not run. This is a failed and incomplete local gate.
- The independent full-source Linux run also failed and was incomplete
on the same frozen commit. Native and real-database admission passed.
Chat integration reported 20 failures in 1,593 tests. Tool-access passed
all 479 tests. Native-session-executor passed all 965 tests. Docker
interrupted the run after about 48 minutes on October 29. The container
exited 143 and was not OOM-killed. The run produced no final group
totals or detailed assertion summary. Shell trap metadata reports zero,
but this is not a valid test-pass result. Source and evidence remain
unchanged. The focused diagnosis below retains those failures; their
cause is not assumed from the interruption. Neither historical local
full run passed.
- A fresh focused replay on the frozen source then executed exactly
those 20 case bodies. All 20 failed; this was not a setup-hook failure.
The two backlog cases reached the unchanged 15-second timeout. The 18
FIFO cases failed on missing notices or receipt rows. All 18 FIFO cases
passed when run alone in a fresh process and database. This supports an
order interaction, not a complete cause or baseline conclusion. A
separate two-backlog control used RAM-backed temporary storage and real
PostgreSQL with synchronization settings on. One backlog case passed in
10.65 seconds; the other still timed out at 15 seconds. No source,
assertion or timeout was changed. The full local gate remains open.
- Earlier mixed-revision run: it began before the final guard and
imported older source before collecting later regression cases. It
failed those new cases with that older cached module. This run is
retained, not counted as candidate acceptance. The fresh focused run
passed all seven stale-setup cases. No case was skipped to hide a
failure.
- Historical remote gates for `1c31` were terminal and clean: 53
successful contexts and two expected Storybook skips. Main CI
[40720536514](https://github.com/paperclipai/paperclip/actions/runs/40720536514)
passed. Greptile reviewed exact head
`1c31cde0ed199fbf4e41b7649590554f7886dfc2` at 5/5 with no actionable
findings; the separate review job also passed, and there were no inline
review comments. These results do not qualify `6c18`.
- CI source qualification: the chat jobs checked out synthetic merge
`ef36746b3d2ec52ba5727dcbe4da265a03421fe4`, which merges the PR into
`7d75ae0569e41786640cf09647ae8849411d2f22`. All 20 locally failing case
names have explicit passing CI log lines. These passes do not establish
standalone local acceptance of `1c31`. The merge adds two upstream Teams
tests, so it collects 1,595 chat cases instead of 1,593. Neither
backlog/FIFO assertions nor their timeouts changed.
- Independent source review found an earlier credential-identity race.
The guard and seven deterministic cases resolve that finding. A later
integration review missed the separate overlapping-start race. Both RED
failures and that review miss are retained. The narrow follow-up repair
passed 498 service cases, independent source review and fresh remote
CI/5-of-5 review; full local qualification remains incomplete.
- Source qualification is ready for merge. Live repaired-draft recovery,
delivery and Google approval are not established by these results. The
local whole command remains explicitly incomplete.

## Risks

- Historical drafts without a matching intent now need an explicit
service choice. This is deliberate and prevents silent broadening.
- A concurrent credential or method edit now returns 409. Reloading lets
the person review the changed setup.
- Overlapping starts keep separate requested scope sets. Updating draft
intent does not relabel an earlier OAuth state's consent request.
- Saved intent is not consent. Provider-reported scopes and current
action policies still control discovery and dispatch.
- No migration is needed. No existing token or grant is rewritten by
saving draft intent.

- Default ungrouped launch paths now execute six chat children in
sequence and 28 whole-command invocations. Exact coverage, local
launcher contracts and all six real default chat partitions passed. A
complete uninterrupted local whole-command run remains unqualified
because of host capacity.
- The retention test-clock repair depends on owned-I/O and zero-duration
connection-timer progress. The controlled regression, sensitivity mutant
and current exact-tree CI protect the original assertion. It did not
reach a completed row in the interrupted local whole run.

## Model Used

OpenAI Codex, described by the runtime as GPT-5 based. The exact runtime
model ID and context-window size are not exposed to this task. Used
reasoning, code execution, browser verification, and independent
delegated review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used, including the runtime metadata
limit
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the bug report template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket or instance identifier
- [x] I have run tests locally and they pass — exact clean 82 passed 41
launcher/workflow cases and all 1,816 real default chat cases, full
recursive typecheck, normal build and token gates. Full exact-tree PR CI
passed. The separate local whole `pnpm test:run` was interrupted by disk
protection and remains incomplete; it is not claimed as a local whole
pass.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green — current exact-head checks
passed; the verified CI checkout tree equals clean 82. The separate
interrupted local whole command is disclosed above.
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
fresh exact-82 summary and check passed; the earlier finding is
resolved. This is source review, not acceptance.
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 22:02:58 -05:00
DottaandPaperclip 822b173281 test: preserve retry producer boundaries in receipt recovery 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* fix/sleep-forward-02b-security:
  fix: wake retry scheduling only when retry state changes
 2026-10-30 21:51:11 -05:00
Dotta 63c922a193 Merge commit 'c2ff3eb92113fb16713b270e34932636956ff16b' into fix/sleep-forward-02b-security 
* commit 'c2ff3eb92113fb16713b270e34932636956ff16b':
  fix: wake retry scheduling only when retry state changes
 2026-10-30 21:50:49 -05:00
DottaandPaperclip c2ff3eb921 fix: wake retry scheduling only when retry state changes 
Separate retry-schedule notifications from ordinary queued-run eligibility. Preserve exact deadlines and prove native producer commit/rollback behavior without global retry scans on ordinary agent or issue edits.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 21:49:38 -05:00