Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-03 01:27:09 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,215 Commits 1,362 Branches 2,218 Tags
42ece47d98864cd00a9ba1021652ae364158ca18
Commit Graph
236 Commits
Author SHA1 Message Date
DottaandPaperclip aab54d5bec Merge current master and generated card schema into attention retention 
Preserve master attention, issue, native cleanup and tool authority changes alongside the exact reviewed retention patch. Keep bounded blocker facts, finalization precedence, source locks, archive eligibility and producer intent unchanged. Publication remains dependent on the parent qualification.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 08:51:00 -05:00
DottaandPaperclip 56f8ff7bd5 Merge current recovery and regenerate status-card migration after master 0419 
Preserve the published concurrent comment uniqueness and read-state backfill migration. Regenerate the combined schema as migration0420 with the repository workflow and retain five snapshots. Add both upgrade-path controls without changing status-card execution policy.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 08:39:44 -05:00
Dotta 3549fad1dc Merge commit '0b4e48be9b881484dc5c48225a3b501595329ce2' into codex/sleep-merge-18409 
* commit '0b4e48be9b881484dc5c48225a3b501595329ce2':
  fix(ui): retain startup autosave through session recovery (#18412)
  perf(server): recover independent native replies concurrently (#18421)
  perf(chat): scope task questions and batch inbox queue checks (#18422)
  fix(native): recover controllers across webserver replacement (#18343)
  fix(ui): restore task document authoring and add persistence journey (#18356)
  fix(tools): stabilize tied profile entry order (#18427)
  fix(adapter-utils): tolerate entries removed during workspace snapshots (#18426)
  test(runner): qualify Kimi Code lifecycle and recovery (#18029)
  feat(runner): deliver native Kimi Code (#18391)
  Preserve selected Google services when setup resumes (#18048)
  feat(runner): add pinned Kimi Code native runtime (#18028)
  fix(ui): show agent-created subtasks inline in the task feed (#18320)
  fix(evals): preserve opaque TestOps parameters (#18394)
  feat(chat): deliver proactive task messages to agent chats (#18374)
  fix(ci): track repository inputs across test projects (#18398)
  chore(lockfile): refresh pnpm-lock.yaml (#18414)
  fix(runner): build on Windows and add a Windows CI lane (#18250)
 2026-10-31 08:27:30 -05:00
DottaandPaperclip d1f9534c54 Merge current master qualification into event-driven run dispatch 
Preserve retained-controller recovery deadlines and rearming together with company-scoped dispatch forwarding. Keep both source owners and all incoming master changes; no execution or retry policy changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 08:03:05 -05:00
DottaandPaperclip 492d537340 fix(native): recover controllers across webserver replacement (#18343) 
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native execution runs in a remote Runner while the webserver owns
its controller.
> - Encrypted PRP checkpoints and session snapshots already survive a
webserver restart in Postgres.
> - Ordinary warm and per-turn admission still needs local markers and
supervisor metadata to find that state.
> - Replacing the webserver with an empty native controller directory
can therefore lose a usable retained session.
> - This change adds bounded recovery metadata to the existing snapshot
transaction and reconstructs local projections from the selected durable
source.
> - The benefit is native execution continuity across webserver
replacement, with the existing ownership and agent lifecycle policies.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Native remote controller recovery and retained-session continuation
during a webserver upgrade.

**Subsystem affected**

server/ — native runtime, checkpoint persistence, recovery scheduling
and internal upgrade reporting. Product E2E qualification also changes.

**Current behavior**

Without both the local PRP marker and supervisor file, normal
continuation can return before discovering the saved Postgres
checkpoint. The snapshot already exists in Postgres, but its local reuse
compatibility envelope cannot be reconstructed.

**Proposed behavior**

Select the existing durable source, validate its identity and ownership,
restore its PRP state and supervisor projection, then use normal
authenticated admission. Missing or inconsistent evidence holds
recovery. It does not authorize fresh provider work.

**Reason and benefit**

A replacement webserver must recover active execution and later continue
its retained session without the old server's native files. This is one
component of deployment continuity. HTTP traffic switching, workspace
storage migration and Railway rollout remain separate.

**Breaking changes**

No new public endpoint, schema migration or database pool. The private
run-profile envelope is additive. Disk-independent rollback also
requires a reader for the staged controller format and compact
checkpoint part; the first production release containing these readers
is the floor. Older file-backed sessions need an owner checkpoint or the
bounded internal importer.

**Additional context**

Related foundations: #17401, #17916, #17254, #14621, #14622. The
implementation contract and open acceptance gates are in [the dated
plan](doc/plans/2026-10-30-native-controller-checkpoints.md).

## What Changed

- Persist a versioned recovery envelope, at most 4 KiB, beside the
redacted session snapshot in its existing transaction. Keep it out of
public run projections.
- Compare frozen admission separately from prepared attachment
instructions during staged recovery, preserving exact
template/identity/authority checks.
- Bound the existing lazy checkpoint pool at three connections after
measured two-slot burst backlog; keep request/renewal capacities and SQL
operation counts unchanged. The tuning is a separate commit.
- Discover and restore the exact selected retained source without local
native markers. Preserve the 32-candidate window, progress barriers and
current authority checks.
- Fence snapshot and result writes. Acquire coordinator, issue and run
locks in order and check expiry after lock waits. Combine snapshot
ownership/read work into one statement.
- Reuse existing recovery deadlines and drain/resume scheduling.
Retained-owner waits preserve the original unstarted run and actual
lease deadline. Stop, ownership and provider-progress fences precede
deferral. Add bounded native evidence reporting, retained-source import
and retention protection.
- Admit a settled warm source with an authenticated idle goal-read
suffix while retaining active-work and mixed-authority holds.
- Permit lifecycle `turn.stop` checkpoints after accepted results under
existing ownership fences. Claim the existing successor coordinator
before retiring the old warm session, and stage shared attachments only
after verified retirement. Preserve omitted isolated-home defaults and
existing fresh skill-selection order during recovery without hiding real
configuration changes.
- Carry the exact durable six-field PRP binding during active recovery
after a healthy warm handoff. Keep the Runner mismatch guard and
preserve opaque attachment IDs.
- Preserve an already verified active or idle warm process during remote
preparation; revalidate its proof/liveness and retain later
authenticated PRP attachment. Ordinary resumed leases retain suspension
checks.
- Drain retained source events under their original authority, reconcile
immutable source accounting and block unknown source tool dispatch
before target work.
- Persist compact staged handoff intent through authenticated adoption
and recover direct/listener command-loss windows without another Runner
or duplicate attachment.
- Rebuild suspended lifecycle receipts for resumed prepared tasks, and
retry missing lifecycle reads after partial PRP materialization without
overwriting contradictory evidence.
- Add normal warm/per-turn diskless continuation coverage, real Postgres
races and SQL-cost checks. Add a reproducible paired load workload.
- Give only the explicit outage fixture a 15-minute provider idle
window, with a red/green catalog regression preserving all production
timing and lifecycle policies.
- Publish the goal-less task-session pointer in the existing
post-acceptance terminal checkpoint transaction, with current ownership,
accepted-result, reset and newer-progress fences. Keep a private bound
receipt so late completion cannot duplicate or resurrect publication.
- Use a retained warm sandbox in the dedicated continuity fixture, and
accept an exact output token followed by an explanation. Keep the strict
identity oracle.
- Extend the explicit Daytona cell to a five-minute active outage and
another replacement after completion. Require retained provider
identity, a new tool receipt and a useful follow-up result. Add an owned
loopback Postgres E2E fixture for hosts that cannot start embedded
Postgres.

## Verification

**Merge recommendation — `fc68584ce6231a02e47ecd743f55b036ccea034f`:
ready for the qualified native-controller component.** Full CI, fresh
review, local application paths, controlled performance qualification
and the strict live Linux/Daytona case pass. Independent resource
cleanup is complete. Broader rollout, browser reconnection and the
unexecuted transport/platform combinations remain integration gates.
Integrated 484 passed full local recursive typecheck/build and ordinary
active→completion→cache-loss→C component qualification. Fresh review
then found a valid attached-wake failure: preparation appends attachment
instructions before the verifier compares execution to the frozen
admission. 9f4 fixes that by separately checking the pre-preparation
admission and exact prepared execution. Both hashes are captured
privately per admission; the full template, identity, configuration and
authority checks remain. Its regression fails before the fix; 36 focused
cases, 42 staged source/target cases and server typecheck pass
afterward. The review thread is resolved. Exact-head full CI passes
([run
41168228822](https://github.com/paperclipai/paperclip/actions/runs/41168228822):
48 successful jobs, one intended skip); the complete `pnpm test:run`
partitions are forced, with the five native suites in the Runner lane.
This is not a claim that an unsharded local command passed. The fresh
exact-head Greptile check succeeds with 5/5 and all six threads
resolved.

The original two-connection database comparison closes **steady PASS and
burst FAIL**. Every relative CPU/WAL/request-p99 gate passes and all
operations/ownership remain correct, but candidate arm 3 drains in
1,015.99 ms and baseline arm 4 in 1,059.67 ms against the unchanged
1,000 ms absolute gate. Those results remain failed. A separate 2→3→4→2
diagnostic reproduces drain times 1,051.57 / 956.79 / 882.02 / 1,065.72
ms: three is the smallest tested passing cap, and the repeated two-cap
failure survives run-order/cache effects.

Separately reviewable `fc68584ce6` changes the existing lazy checkpoint
pool default to three connections. Request capacity stays ten and
liveness two; idle release, shutdown fencing, transaction counts and
pool count are unchanged. Thirteen DB lifecycle/cap tests and DB
typecheck pass. Full recursive typecheck/build pass at exact fc68584,
with the matching server build stamp. The final paired native
feature-cost comparison explicitly uses the **same cap-three policy and
common current DB factory in both variants**; its baseline is
controlled, not the unmodified historical cap-two configuration.
Original gates and workload counts remain unchanged. The one strict
Linux/Daytona case started at 04:40:56 UTC only after both new profiles
passed; it passed on the first attempt with all 49 workflow checks.
Independent cleanup confirms all three owned sandboxes absent, owned
Postgres/controllers/observer gone, VM terminated, root volume absent
and ephemeral SSH key/agent removed.

**Final controlled cap-three qualification: PASS.** Baseline native
implementation `59da15066a` and candidate `fc68584ce6` use the same
current DB factory and checkpoint policy (request/liveness/checkpoint =
10/2/3). Eight 65-second ABBA arms cover 300 controllers and 10,000
historical tasks. All correctness, absolute and 10% relative gates pass;
original cap-two failures remain unchanged.

| Mode | DB CPU µs/op, baseline → candidate | WAL B/op, baseline →
candidate | DB-request p99 ms, baseline → candidate | Maximum drain |
| --- | --- | --- | --- | --- |
| Steady | 542.40 → 554.83 (+2.29%) | 1509.01 → 1558.27 (+3.26%) |
107.11 → 111.90 (+4.48%) | 26.35 ms |
| Burst | 1036.36 → 1063.11 (+2.58%) | 4153.12 → 4317.64 (+3.96%) |
103.736 → 103.841 (+0.10%) | 930.14 ms |

Every matched arm preserves operation/statement/transaction counts,
including 3,900 PRP and 3,900 snapshot transactions. All three actual
pools are included; there are zero lost owners, missed lease deadlines,
dropped operations or errors. Peak Postgres CPU is 18.31%/9.47% of two
cores and memory 614.29/684.42 MiB (steady/burst). The benchmark uses
one Node process simulating 300 controllers; it does not qualify
two-webserver overlap, HTTP latency or hosted capacity. Its observer SQL
contributes to measured database CPU. The original unmodified cap-two
baseline is separately retained as failed; this cap-three comparison is
explicitly controlled.

The exact final-source 8/512/8,192-chunk test also passes: zero recovery
SQL, transactions or changed rows in each text-only interval, zero idle
SQL, seven lifecycle transactions and twelve changed rows per sample.
Existing lease traffic remains separate. The final ordinary managed
application replay passes active recovery, completion, native-cache
removal and C continuation; its UNKNOWN local debug-daemon provenance
and partial instruction-observer limits remain recorded. The live case
uses the attested Linux producer instead.


**Live qualification: PASS, one attempt, no retries.** Campaign
`native-controller-retained-fix-fc68584ce6-ec2-20261031T044055Z` started
04:40:56 UTC on the exact final Linux webserver source; suite definition
hash `12f9bb97823c1537e43f0398e7cddf39890f77bce7b413ca4dfd944f4a30e5d5`.
The existing immutable image and attested Bookworm Runner artifact are
unchanged.

- A was SIGKILLed and its native cache discarded. B started after
300,149 ms, naturally acquired generation 2, authenticated at 316,624 ms
and resumed at 317,682 ms within the original 420,000 ms grace. The same
active run, Runner, provider PID/start fingerprint and one
submitted/started/accepted turn were preserved; one final
result/terminal and workspace finalization completed, with no
retry/recovery action.
- After active and queued completion, B was SIGKILLed and its native
cache discarded again. C received exactly one ordinary task-comment API
follow-up, completed a new tool and produced a useful reply while
preserving native/provider/Runner identities. This closes the earlier
failed post-C gate. A separate C OS-process PID field was not exported;
the physical-PID proof is specifically A→B.
- Four distinct results were accepted and published; all four
finalizations committed at attempt 1, all four accounting-v2 projections
settled, and the task-session pointer advanced to C. The coordinator
independently checked these facts and all 49 canonical checks against
the sanitized receipts.
- Canonical cleanup passed and independent checks confirmed three exact
sandbox 404s plus complete inventory/zero owned remnants, no owned PG or
controller/observer processes, EC2 terminated, root volume absent, and
the ephemeral SSH key/agent removed by 04:54:25 UTC.

Live scope is classic remote Codex warm continuity with non-native
storage dependencies available. Prepared-task completed B→C, full
per-turn application disk loss, browser reconnection, two-server load
and overall Railway/HTTP continuity were not live-qualified here.
Earlier failed campaigns and the original cap-two burst failure remain
failed historical evidence. A report-renderer cache setup failure was
preserved; the normal sanitizer subsequently succeeded with an explicit
cache path and a new output directory, without changing validators.


The repair keeps retained events under their original six-field
authority until persistence and accounting accept them. Unknown semantic
effects stay unresolved. A compact staged target survives source
bootstrap, cached-command delivery and authenticated handoff, including
an uncertain commit response. Direct and same-guest-listener recovery
require the verified original Runner and the exact endpoint. Target
snapshot publication marks readiness in memory, with retirement
coalesced into an existing command write. Headers contain only the
staged descriptor/digest; a prequeue template is written once as a
changed checkpoint part and then recovered from the existing
command/receipt. No second session snapshot transaction, streaming SQL
path, new pool or recurring idle scan is added.

| Current repair gate | Evidence and scope |
| --- | --- |
| Source-tail regression | Real Runner protocol RED reproduced
attachment rejection. Frozen core/transport pass five source-tail cases:
source usage attribution, held persistence/no premature ACK, lost commit
response plus cache loss, unknown semantic hold and accounting rejection
hold. |
| Handoff loss windows | Direct and listener modes pass
queued/prepared/activated/completed boundaries (eight cells), preserving
the actual Runner/provider process. Missing adoption proof and changed
endpoints reject recovery. |
| Compact storage | A 31 KiB attachment grant writes its separate part
once, leaves it unchanged across source snapshots and removes it at
queueing. Headers do not duplicate the grant. |
| Accounting | 21 real-Postgres usage tests and 19 source-binding tests
pass. Recovery hydrates immutable source receipts without resetting
their producer or billing classification; conflicting/incomplete usage
holds. |
| Admission and malformed evidence | Driver 16, staged evidence 42,
restart classifier 27 and source usage 19 focused cases pass. Four
malformed-file regressions pass at d6a7, whose executor is
byte-identical at 484, and server typecheck passes. Full executor file:
990 pass; 16 embedded-Postgres `initdb` setup failures before
assertions, retained separately. |
| Required checks | Exact fc68584 recursive typecheck and build pass
locally, with matching server stamp. Forced full CI succeeds (48 jobs
pass, one intended skip); all PR checks are clean. Maps/token gates pass
for their unchanged source surfaces. Fresh exact-head Greptile is 5/5,
all six threads resolved. |
| Ordinary application path | Exact fc68584 managed component passes
active recovery after natural expiry, completion, native-cache loss and
ordinary C continuation. Production bytes match before/after; temporary
fixtures restored and owned DB/services cleaned. UNKNOWN local
debug-daemon/partial instruction-observer limits remain; the separate
live case uses the attested Linux producer. |
| Producer | Locked Bookworm build from exact 5b7 Rust tree
d03e494217, equal to final fc68584 Rust
source. SHA-256
65b8f8cfdd6099e9ad24f1ca154e8a260777269cc82f6e23c6e6a180c531eaaf;
38,515,328 bytes. Actual Debian 12 execution and live admission pass.
Controller/admission/DB changes do not relabel the producer source. |
| Historical cap-two comparison | Exact 59da baseline vs 5b7 candidate,
300 controllers/10,000 history, ABBA 4×65 s per mode, all three actual
pools and real snapshots. Steady CPU/op 541.36→554.64 µs (+2.45%);
WAL/op 1512.13→1560.99 B (+3.23%); DB-request p99 102.78→105.99 ms
(+3.12%): PASS. Burst CPU/op +4.71%, WAL/op +5.29%, p99 −0.10% pass
relative gates, but absolute drains above fail. Identical operation/SQL
counts, zero lost leases/drops/errors. One Node process simulates 300
controllers; this does not qualify two-webserver overlap or HTTP
capacity. |
| Live acceptance | Strict final-source Linux classic Codex case passes
1/1 on attempt 1, all 49 checks. Active recovery within grace and useful
retained C continuation succeed; four
accepted/published/committed/settled outcomes. Independent provider,
PG/process, VM/root and SSH cleanup complete. Prepared/task and broader
rollout limits are explicit above. |

**Historical qualification snapshot through `9eb9ac6`.** The head at
this snapshot was `9eb9ac6f9e312f3ebe8040f5a7565e88ae8824f3`. Server and
Runner package sources are byte-identical to `283e05ffaa`; the two later
commits change the explicitly scoped qualification and its documentation
only. It includes master through the storage-owner fix
`45a3a92e6dc52d20923b9aa34a22fee2ec5443f9` (#18357). At that snapshot,
CI/review passed but the strict post-C native recovery gate failed; the
PR stayed draft and required another native repair. Prior `39a4343` CI
and review passed, but its full live workflow failed; that grade remains
unchanged. The first repaired `435ccd2` live attempt failed before any
injected outage. It exposed a legitimate terminal snapshot before result
acceptance; no accepted result ID or publication receipt was observed.
Its canonical report remains 0/1 `candidate_failure`, with cleanup
passed and independent provider absence confirmed. `0862062` defers
publication until result acceptance and fixes the fresh review's
coordinator/issue lock-order finding. Both failures have deterministic
real-Postgres regressions. The `0862062` live attempt then proved
accepted-result publication but failed active recovery: a resumed-lease
preparation guard demanded suspension after the exact live Runner was
already verified for adoption. `64c0b746` preserves that verified
active/idle-warm process, revalidates its proof/liveness, and keeps
ordinary suspension checks and later PRP authentication. Its live run
passed preparation but exposed a second active-warm boundary: all
durable binding fields matched except opaque turn/item attachment IDs,
which the server had regenerated. `7a203274` carries the complete
verified six-field binding and keeps the strict Runner mismatch guard.
Its cloud case then passed active adoption, completion, publication and
accounting. The queued continuation exposed accepted-result retirement
writes and an implicit-home fingerprint drift; both are fixed with
deterministic regressions in `39a4343` before this fresh full
qualification.

The prior live case exposed a real publication gap. A goal-less native
turn could persist its accepted result and terminal snapshot before
provider cleanup, while the task-session pointer was still empty. Normal
later admission then selected a fresh session. The fix publishes that
pointer and a private bound receipt in the **existing post-acceptance
terminal checkpoint transaction**, before cleanup. Earlier terminal
observations persist without a handled receipt or continuation pointer.
No result-acceptance or snapshot transaction was added. The late
completion path recognizes the receipt. This also prevents a late
callback from restoring a pointer after a user reset. Goal/control
publication keeps its existing owner.

The dedicated Daytona fixture now uses the existing warm/reusable
environment. Its sandbox, Runner and provider storage must survive. The
former generic per-turn fixture disposed the sandbox after completion
and kept provider-history backup bytes under the discarded directory.
Whole per-turn disk-loss acceptance therefore has a separate
provider-backup storage dependency. It remains distinct from the passing
native per-turn component checks with backup storage available. The
60-second lease, 300-second outage and 420-second grace are unchanged.

| Gate | Current repair evidence |
| --- | --- |
| Assembled regression | Baseline fails with an accepted result and
terminal snapshot but a NULL task pointer while real provider cleanup is
held. The fix passes normal continuation and reset cases, with no
pointer seeding. Exact merged `92b7447` passed 2/2 twice, including
after normal package TypeScript output refresh. Production files are
byte-identical at frozen `435ccd2`. |
| Persistence and concurrency | 19 helper/transaction cases and all 59
existing control-port cases pass on `0862062`. They cover bound result
IDs, reset/newer epochs, Stop, lock waits and expiry, receipt replay,
missing stubs, foreign bindings and atomic 4 KiB rollback. Existing
snapshot transaction count stays one. The new tests reproduce the exact
live missing-result-ID error and the inverse replay/status lock order
against `435ccd2`, then pass after the correction. Replayed results
still reject stale generations. |
| Completion | All 32 completion-boundary tests pass on merged
`92b7447`; the source is unchanged at `435ccd2`. Native result-shaped
session metadata is preserved. |
| Qualification fixture | A valid review report reproduced three
rejected correct replies. Exact token-boundary matching fixes them. 140
focused E2E/catalog cases and Product E2E typecheck pass. Wrong tokens,
marker prefixes/suffixes and wrong authors remain rejected;
retained-identity checks are unchanged. |
| Required checks | Full recursive typecheck and standard build passed
at `435ccd2`. Both exact commands also passed on `0862062`, with its
exact server build stamp; both also pass on `7a203274`, with its exact
server build stamp. Fresh `0862062` Greptile passed 5/5, with zero
unresolved threads. The `64c0b746` normal CI then passed all 49 jobs (48
success, one expected skip) and Greptile passed 5/5 with zero unresolved
findings. The `7a203274` normal CI also passed 48 jobs with one expected
skip, with fresh Greptile 5/5 and zero unresolved findings. Current
`39a4343` recursive typecheck and standard build both pass with
persisted exit codes and its exact build stamp; fresh Greptile is 5/5
with zero unresolved findings. Its normal CI completed successfully: 48
jobs passed and one expected job skipped. The fresh live result remains
separately failed. The `0862062` normal CI had 46 passing jobs, one
expected skip, one worker communication failure and its aggregate
failure; GitHub states the self-hosted runner lost communication. No
failed assertion or specific OOM/spot cause was established, and no
blind retry was used. |
| PostgreSQL supplement | One corrected all-pool pair at the preceding
repair passed 300-controller/10,000-history correctness and absolute
gates. SQL 76,590→75,990; transactions 9,013→8,713; UPDATE rows 11,613
each; 300 correct pointer writes each. PG CPU 31.99→30.46 s, WAL
70.36→67.87 MB, request p99 100.98→103.12 ms. This is one pair, not a
repeated improvement claim or an exact `0862062` CPU measurement. Its
two later changes defer premature publication and reorder existing
locks; they add no transaction, stream/idle writer, pool or timer. The
first collapsed-pool attempt remains invalid and retained. This is not
an exact final-head CPU/WAL measurement; later fixes change ordering,
identity selection and in-memory skill replacement without adding
snapshot transactions, pools, stream writes or idle SQL. |
| Live publication proof (`0862062`) | Baseline run
`06fcc806-f186-4190-9b57-087b61d75533` accepted result
`58f912dc-94d9-4b4d-bd3d-d5f689c83fd3`; the publication receipt names
the same result, the task pointer has parameters and `lastRunId`
matches, and the normalized native identity matches the following active
run. Accounting v2 settled. |
| Active recovery (`0862062`) | **FAIL.** SIGKILL/cache discard at
23:07:28.666 UTC; hold 300,000.859 ms; B ready at 308,896.486 ms;
natural generation-2 ownership at 312,906 ms. At about 319 s,
preparation failed `runner_harness_state_mismatch:
resumed_not_suspended`. No authenticated after-receipt passed within 420
s. Exact original Runner PID 668/start/fingerprint was independently
still alive; provider auto-stop was 15 minutes. No retained C follow-up
qualified. Canonical 0/1 `cleanup_failure`/cleanup-failed report remains
failed; subsequent independent cleanup deleted all three owned resources
with exact 404s and confirmed full intended-account inventory with zero
owned resources. |
| Verified adoption preparation (`64c0b746`) | 91 focused cases pass,
including actual preparation for four active providers and a distinct
prior/current warm proof. Unproved resumed state, dead process and stale
warm owner remain rejected. The earlier options-only fixture missed this
gate. No spawn, backup restore or PRP-authentication bypass was added.
Server typecheck passes. |
| Active warm identity (`64c0b746`) | **FAIL.** B claimed generation 2
at 312,995 ms. Preparation passed, then strict PRP recovery rejected the
generated attachment IDs at about 319 s. The saved and desired Runner,
lease, normalized session and run IDs matched; only
`turn_attachment_…`/`item_attachment_…` differed from generated
`turn-<run>`/`item-<run>`. Canonical 0/1 cleanup-failed report remains
failed; independent cleanup confirmed deletion/404s and zero owned
provider resources. |
| Six-field identity repair (`7a203274`) | Actual server-factory
regression RED4→GREEN38. A real Rust Runner test passes healthy first
run → random-ID warm handoff → held active second run → restored
controller. It verifies a new live authenticated connection, exact
identity, same Runner/provider OS identities, no duplicate launch/turn,
and no redispatch or invented receipt for an uncertain tool. The
synthetic wrong pair still fails the unchanged guard. This protocol case
does not claim business completion. Server and Runner typechecks pass. |
| Active recovery/completion (`7a203274`) | **PASS for the active
component.** Full five-minute SIGKILL/cache-loss hold. B ready at
308,936.651 ms; natural generation 2 observed 312,979 ms; authenticated
at 317,720 ms and resumed processing at 318,527 ms. Same run
`1e1ccb00-0728-4935-97da-57562d81cb42`, Runner
`246f73f7-a689-4177-a9c2-7ce711d63aed`, remote PID 669/start
23:48:27.470, provider lease/session and one submitted turn. Accepted
result `49097529-5310-4ac3-af95-5bf9b91d19f4`, publication, finalization
and accounting v2 all settled. |
| Full workflow (`7a203274`) | **FAIL, preserved.** Queued
late-requirement run `5e61f38d-c7d1-4fc1-bcbf-bce9d46ce4eb` failed
`runner did not durably suspend before checkpoint` before outage 2.
Canonical 0/1 candidate failure, cleanup passed, evidence valid. C
retention was not exercised. Independent provider absence,
controllers/PG cleanup, EC2 termination and root-volume absence all
confirmed; closed receipt SHA-256
`676a3b15eb9937b6bf995c36fb8666c88964d509e5ceb9109fb02da0aa2bc909`. |
| Retirement and stable configuration (`39a4343`) | Real Runner +
Postgres RED reproduced the exact suspension failure and nine owner
rejections. `turn.stop` is now lifecycle checkpoint maintenance under
unchanged owner/result/generation fences. GREEN25/25 proves actual
persisted suspension, exact identity and completed stop/drain/suspend;
unaccepted, foreign, stale and new-work negatives remain rejected. Two
default-home fingerprint REDs become GREEN9/9; explicit mode changes
still invalidate. No global matching exemption or historical fingerprint
rewrite. |
| Fresh full Daytona warm cell (`39a4343`) | **FAIL, preserved.**
Launched 00:15:29 UTC with automatic retries disabled. Active adoption
passed: actual offline hold 300,000.848 ms; B ready 309,922.155 ms;
natural generation 2 at 311,821 ms; authenticated 317,989 ms; resumed
330,440 ms. Same run `3e126819-e786-4b3b-b923-c975cf4f5b79`, Runner
`4079138a-8d7c-44a4-8558-07a5411d1f55`, PID 668/start and
provider/session identities. Accepted result
`08e504af-4dec-48a8-bf05-bf0a32280559` and publication passed. Queued
run `31d168c9-7ab9-4700-abb6-fafb90912237` failed the same strict
suspension error before outage 2. Its row selected a new native
identity, but no new third provider process or invocation was observed.
Recovery-time configuration still reset only `adapterConfig`; native
provider, context and workspace subtrees were unchanged. Canonical 0/1
candidate failure, evidence valid, normal cleanup passed; all campaign
provider/PG/process resources independently absent. Closed campaign
receipt SHA-256
`5e7d6281c13c812f3a946667f97f3c7d339b5ad86a3a819bd185b2fa73bc4a19`;
EC2/SSH remain temporarily held for the concrete follow-up repair.
Original 60/300/420-second limits and strict queued/second-replacement
checks remain unchanged. Direct-process path; no new live prepared-task
claim. |
| Native retirement authority (`283e05ffaa`) | Exact `39a4343` facade
regression RED3→GREEN9. The successor now uses its existing production
coordinator claim and prompt renewal before strict retirement. A real
Runner/Postgres case starts in the actual native-observed/unclaimed
phase, claims through production code, then closes and durably suspends
the predecessor in 1.657 s. All 25 real-Postgres cases pass;
unclaimed/foreign writers remain rejected. Shared attachment staging
stays after retirement; Stop and denied claims neither close nor stage.
No new claim or SQL path was added. |
| Configuration continuity (`283e05ffaa`) | Real shared/project-primary
environment selection → configuration with four skill versions
reproduces `reset: true`, changed category exactly `adapterConfig`,
after a Postgres JSONB read. Pinned restoration had reordered the
generated preference while actual runtime skills stayed identical.
Replacing existing pinned rows in place preserves the original fresh
name/key order and fingerprint. The assembled regression passes, as do
three existing pinned-version/rename/alias cases and server typecheck.
Genuine model/skill-selection changes still invalidate. No
fingerprint-format change or broad compatibility exemption. |
| Final production repair (`283e05ffaa`) | Full recursive
typecheck/build and [normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41151145106)
PASS: 48 jobs plus one expected skip. Fresh review 5/5, zero unresolved
threads. Live baseline, recovered run and queued continuation all
succeeded, each accepted/published/committed once with accounting v2
settled. Their native/provider identities and configuration fingerprint
match; both recovery and continuation report no reset, no changed
categories and task-session reuse. Active outage/authentication remains
within original timing. |
| Browser-dependent continuation (`283e05ffaa`, definition 6) | **Whole
cell FAIL, preserved.** After B was killed at 00:46:38.928 and its cache
discarded, C became ready in 9,151.410 ms. `openParent()` then hit a
30-second `page.goto`/DOMContentLoaded timeout before posting the final
request. No fourth native run or C continuation was exercised. Cause of
navigation timeout is unknown. C task had no execution lock, queued
comments, handoff or recovery blocker; all three prior runs were
committed/accounted. Normal cleanup passed; independent provider
absence, controller/PG cleanup, EC2 termination and exact root-volume
absence confirmed. Sanitized report bundle SHA-256
`2a3b841d62efb2ae30ef14f96a431ac4fa7fe3a782f1144831512f78edc65022`. |
| Native API continuation qualification (`9eb9ac6f9e`, definition 7) |
**FAIL, preserved.** The real post-C API request dispatched run
`c71fbb18-f979-4fe4-93ce-9c4be46dbf21`, which failed
`native_session_interrupted` / `native_session_recovery_failed` at
01:10:00.640 UTC. It selected the same normalized native identity,
Runner and checkpoint-provider identity as the three succeeded turns. No
C result was accepted; pending accounting remains explicit. Active
recovery passed: actual offline hold 300,001.098 ms, B ready 308,335.019
ms, natural generation 2 observed 311,818 ms, authenticated 315,540 ms,
resumed 316,612 ms. Queued continuation succeeded/accepted/published. B
was then killed/cache discarded at 01:09:34.500 and C ready in 8,783.803
ms. Original classifier `transient_infrastructure`, whole-case failure
and cleanup passed are unchanged; that classifier does not establish the
root cause. All owned provider/PG/process resources, EC2, root volume
and SSH agent independently absent. Closed receipt SHA-256
`7f21ac4a22efc07b849fa7e17866083b6127b3321af12671b4fc054ce57039a9`;
sanitized bundle
`71ffef14a30a553e008320a31840a6d6c0de85c1bfdb60f783fdb53a6aae812a`.
Exact backend detail is unavailable in approved structured evidence. |
| Current required checks (`9eb9ac6f9e`) | Recursive typecheck, standard
build and exact build stamp PASS. [Normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41152908417)
PASS: 48 jobs plus one expected skip. Fresh Greptile 5/5, latest edited
summary checked, zero unresolved threads. Exact8/512/8,192 stream probes
again report zero recovery SQL/transactions/rows and settled-idle SQL;
seven control transactions/twelve changed rows each, with 36 separate
legitimate renewals in the longest case. CI success does not supersede
the failed live C gate. |

The completion supplement observed secondary request/liveness/checkpoint
transaction p99 increases of 23.0%/45.7%/15.6% in its single pair.
Renewal p99 stayed below 0.865 s, with 900 successful renewals per arm
and no lost leases. The named pool caps stayed 10+2+2. An aggregate
activity sample peaked at 15 non-observer entries without backend-type
classification, so no claim is made that all were client connections. No
sampled lock episodes does not establish zero lock waits.

Twelve pure reader/selector cases also pass with the new adjacent
receipt. This is not another executed process rollback. Development
reader `5cf7aead` can parse the recovery state, but its old run-detail
redactor can expose receipt IDs/outcome to an authorized run reader. The
first production release containing recovery readers, complete
attachment-identity handling and both private-field redactions is the
rollback floor. The implicit-home fix prevents new recovery-time
fingerprint drift; it does not rewrite previously stored fingerprints
from failed candidates. The development probe is not a published
supported rollback target.

Latest actual `0862062` Runner tests independently passed 8/512/8,192
text chunks with zero recovery SQL, transactions and changed rows, plus
zero settled-idle SQL. All three observed seven setup/control
transactions and twelve changed rows. The 8,192-chunk case separately
recorded 31 legitimate lease-renewal statements, with no lost lease.
These are component SQL observations, not a production CPU/WAL claim.
The new adoption and identity branches affect recovery preparation only.
The `64c0b746`, `7a203274` and exact `39a4343` CI independently passed
all three stream sizes with the same zero-SQL/idle result; each latest
8,192-chunk case recorded 36 separate renewal statements. Exact
`39a4343` Runner 2/2 passed 133 files / 2,317 tests plus two skips; its
separate native group passed 150 tests plus eight skips.

The first API fixture revision omitted the mandatory final screenshot;
this was caught before paid launch and allocated no provider/model work.
The correction captures the actual retained Chromium page with an
explicit unqualified-browser label, without changing the standard report
validator.

Automatic approval review rejected exporting raw private API/workflow
snapshots because of sensitive payloads and insufficient
destination-specific authorization. Sanitized reports and
bound/hash-only receipts are retained; raw snapshots were not copied
through another channel. This does not change the failed grade.

Repair characterization also preserves its failed attempts. An
unconditional staging relocation broke three existing same-scope
scripted cases; the final implementation defers staging only for an
actual superseded warm scope, and all nine order/race cases pass. One
baseline transform overlapped an edit and is explicitly invalid; the
actual immutable `39a4343` loader then produced the three genuine REDs.
The independent task-directory hypothesis produced a RED but was
excluded by the live shared-workspace evidence; no directory policy
change was landed. A shared-workspace fixture authorization omission was
corrected separately from the four-skill semantic RED.

Setup failures are retained separately from semantic outcomes. These
include an old host port after the owned Postgres fixture restarted and
a sandbox-denied TSX IPC pipe. The supported external-Postgres run
passed, and the final exact recursive typecheck passes with normal
process permissions. Seven stale declaration errors after the master
merge cleared after compiling the actual merged SDK/Runner TypeScript
sources. No declarations were edited by hand.

<details>
<summary>Earlier qualification measurements and failed attempts
(historical, before this repair)</summary>

**Historical blocked gate (before the repair below).** Final native
implementation is `abed921e12d543b62f57a56435419c0dafb0bc33`; current
branch head is `af9e2d157dcc349e55374ad186c9be2d6cb96bf9`. The two later
commits change only qualification fixtures, their matcher and
documentation; `server/` and `packages/` are unchanged. It includes
master through `fd008979f5794fda3d5f54151d36acd31d0de140`.

The live Daytona measurement proves five-minute active recovery,
once-only completion and accounting, but **fails retained-session
identity after the second replacement**. The saved checkpoint and
recovery metadata exist. The task-session pointer remains empty, and
ordinary admission creates new native/provider sessions. The exact
skipped publication boundary remains unproven. A useful saved-file
result alone does not satisfy retained-session continuation. This
application-path gap blocks merge independently of CI.

A separate new-master CI failure is owned by the storage workstream (fix
in #18357). No storage guard, session-identity oracle, lease/grace limit
or provider result was relaxed.

| Gate | Observed result and exact scope |
| --- | --- |
| Distinct source recovery and compatible rollback | PASS at A
`5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → B
`5ea45f8d17e33c9a62bd79982dcf0375b07733d0` → fresh B → A. Four
controller processes, natural 60-second lease expiry, absent native
projections before replacements, three succeeded/committed unique
results, one provider thread, unchanged Runner identity, and strict idle
cleanup. 177.782 seconds. |
| Immutable installed native component | PASS at installed A
`5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → installed B
`eaef0d56552c087aaa4fa68a8655282ffbd12667` → fresh B → A. Each
controller imports its contained compiled server and DB. Three unique
committed results, stable Runner/thread, no extra provider restart on
same-contract rollback, strict cleanup. 106.672 seconds. This is
compiled component proof, separate from full CLI-native application
acceptance. |
| Normal public CLI startup | PASS for B
`eaef0d56552c087aaa4fa68a8655282ffbd12667`, Node 24.22.1, darwin-arm64:
fresh external DB, 415 migrations, health ready, root HTML 200, company
POST 201 plus GET readback. The installed tree remained unchanged after
execution. |
| Five-minute component outage | Actual controller absence 300,168 ms.
Natural generation-2 claim at 407.196 seconds after crash, same provider
attempt and Runner, original run succeeded/committed. The larger
540-second case timed out before its retained follow-ups, so it is an
active-adoption pass, not a complete four-controller pass. The complete
short source/installed cases above independently cover follow-up and
rollback. |
| Drain/resume review regression | Fixed in
`226f03e736d1ee99fea1762ab32ea3a8a9241c20`. Five facade-level cases
pass: overdue normal/restart waits, future timer cancellation, another
facade resuming the original owner, and transient retry. Against exact
prior scheduler `5ea45f8d17e33c9a62bd79982dcf0375b07733d0`, four fail
and one passes. The later head changes documentation only. |
| Final corrected-fixture Daytona | **PARTIAL: active recovery PASS;
retained identity FAIL**, exact live source `0b6afc499f`. A was
SIGKILLed and native cache discarded; actual offline hold 300,000.375
ms. B was ready at 308,326.961 ms, naturally claimed generation 2 at
312,059 ms, authenticated at 315,996 ms and resumed processing at
316,519 ms, within the original 420-second boundary. The same run,
Runner PID/start, session, lease and event prefix survived; one turn
submitted. Original execution succeeded and settled accounting. After
completion, B was killed and its native cache discarded; C was ready in
9,077.779 ms. C executed the saved project and returned correct
`queue_r`, with all four runs succeeded/accounted. However, native and
true provider-session identities changed: the unchanged retained oracle
fails. No whole-workflow pass is claimed. |
| Prior matched-producer Daytona | FAILED at exact `abed921e12`. The
matching current-source Runner passed normal admission, completed the
first task and settled its accounting. During the active revision, A was
SIGKILLed and its native cache discarded. Actual offline hold was
300,000.727 ms; B was ready at 308,304 ms and naturally claimed
generation 2 at 309,651 ms. At about 318 seconds, the remote process
check reported `process_not_running` for the original admitted Runner.
No authenticated after-receipt or retained follow-up passed. The current
failure is preserved; no automatic retry, lease/grace relaxation, debt
edit or provider replay is used. Bounded forensic read found preserved
exact-bound state with no recoverable failure, a transport-reset
diagnostic and no terminal delivery; this supports external process loss
without proving the exact cause. The sandbox had a five-minute auto-stop
policy, conflicting with the required surviving-sandbox assumption.
Owned cleanup precedes the separate corrected-fixture case. |

**Live retained-session boundary.** Prior completed run `3806…` and C
follow-up `8adf1df3-6744-40bf-b35d-100451f29d9f` both have database
session snapshots and `nativeControllerRecovery`. Their tool
fingerprint, provider configuration, workspace descriptor,
runtime-context digest, driver and work mode match. There is no recorded
force-fresh/workspace-selection change, and this is an ordinary task
rather than a conversation-generation reset. The same issue/agent
task-session row has `session_params_json: null` and `last_run_id: null`
from baseline time. Both native normalized UUID and actual provider
session/thread hashes differ after C. Full application admission
therefore did not select the retained source. No fallback to file
execution or relaxed identity comparison is accepted as proof of
continuity.

**Independent cleanup completed.** Full current-account provider
inventory confirms no resources owned by either final fixture, with
exact-ID absence checks. Owned controllers and campaign Postgres are
stopped/removed. EC2 `i-01551c09af098dde5` is terminated, its
delete-on-termination root volume returned `InvalidVolume.NotFound` at
22:14:08 UTC, and the temporary SSH key/agent were removed. The
canonical failed report and immutable oracle input are retained. Closed
qualification receipt SHA-256:
`b4b5fefce287ebfdbf2cc2238efaad62d8ad485aed0e9c0628ae95dbc18a8575`.

**Original measurement and grader repair remain separate.** The C reply
contains the requested marker followed by a newline and the correct
output. The old workflow waited for one literal space, so its normal
receipt collection stalled; that original attempt was closed and its
canonical `cleanup_failure`/interrupted result is preserved. Commit
`af9e2d157d` repairs only this reply predicate, with 144 focused cases,
E2E typecheck, discovery and diff checks passing. Ordinary string
matching stays literal. The corrected predicate accepts the saved reply,
but the unchanged retained-identity oracle still fails. Pure
active-recovery grading of the captured measurement passes 7/7; this is
not a rewritten passing campaign.

The sandbox idle-window conflict was corrected only for this explicit
fixture (15 minutes). Actual baseline and later sandbox metadata confirm
that policy; the original active box was disposed before a post-gate
policy read, so its 15-minute setting is established by the normal
create/environment configuration rather than an after-read of that box.
Production native lifecycle and the 60/300/420-second constants remain
unchanged.

The development compatibility reader A is a genuine backport commit
based on `59ff89a18fe1312b5f0bb908439bbdb1237b0252`, with the required
idle-goal reader fixes. It is not a previously published release. Its
older executor differs from B. The rollback floor must include this
reader capability. The source component cases admitted the Linux ARM64
Runner packaged at `59ff89a18fe1312b5f0bb908439bbdb1237b0252`; the
installed component case admitted the `5cf7aead` package. Their Runner
bytes are identical: SHA-256
`b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8`. Each
admitted artifact remained unchanged throughout its case.

[Final public package
build](https://github.com/paperclipai/paperclip/actions/runs/41121460635)
and [reader-floor
build](https://github.com/paperclipai/paperclip/actions/runs/41122170506)
both passed all four daemon builds, package assembly and both Darwin
clean consumers. Both Linux ARM consumers failed before provider work
with the same existing local Codex permission-profile error: bubblewrap
attempts `activity.fifo/.git` and reports “Not a directory.” This
matched baseline failure remains open; it is not a native remote
continuity pass. Full post-run tree and build-stamp revalidation also
passed after the compiled native case. The installed component trees are
A `c29114c1bf71c5af00f761aeb9e3d84ed3d3075b9f5600dee3ef2543eb2b6ec5` and
B `eda575505373a3bdc630fa4919862b359791fe3a7ef637961a25b73b38efe4b0`.

**Controlled PostgreSQL qualification passed.**
Baseline/candidate/candidate/baseline workloads used 300 controllers,
10,000 historical tasks, real snapshot/encrypted PRP writes and
unchanged pools. PostgreSQL and client were each limited to two CPUs and
1536 MiB on an isolated eight-vCPU Linux host.

| Workload | PG CPU per operation | WAL per operation | Database request
p99 |
| --- | ---: | ---: | ---: |
| Steady | +1.18% | +1.81% | +1.91% |
| Aligned burst | +3.40% | +2.56% | +3.81% |
| Two logical servers | +3.11% | +2.56% | +9.68% |

All arms passed absolute/correctness gates with unchanged control
transaction counts, no dropped operations and no owner loss. Overlap p99
has little margin below the relative threshold. It uses two clients in
one Node process, not two HTTP servers. [Closed measurements and
limitations](doc/plans/2026-10-30-native-controller-qualification-evidence.json)
and exact-source reproduction patches are retained. [Earlier
contended-host
failures](doc/plans/2026-10-30-native-controller-load-evidence.json)
remain unchanged. The production DB-claim probe claimed 300 owners in
2.84 seconds and restored/committed controls and snapshots in 8.74
seconds, with zero owner loss or settled-idle SQL; its provider
references were synthetic. The measured checkpoint/snapshot/renewal
paths retain the same persistence behavior after integration; later
accounting and business writes have separate owners.

Actual Runner stream tests at `da8c7f6e1837ddd896dfe83a7524812fee813487`
passed 8/512/8,192 chunks: zero recovery SQL, transactions and changed
rows during text intervals; zero settled-idle SQL. Each retained seven
control transactions and twelve changed records. Existing renewal SQL
was measured separately. The guard/scheduler fixes add no stream path,
pool or recurring idle scan.

The live fixture used current-source producer SHA-256
`f0000c90cb83880191479da639201e659aa6ce0fc7d67c859faa172ec977849e`, a
normal locked exact-head build with pinned Rust 1.97.1 in Bookworm.
Actual network-isolated, read-only ABI execution passed (GLIBC
requirement 2.34 against runtime 2.36). The immutable base image
remained
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:36069788f53c51f228764fa36180575443dfb5d1f84a342d7c0daceb28dc61cc`,
source `a0ef3ce7`. This was an unpublished development producer,
explicitly distinct from the base image and admitted before the fresh
fixture began. The earlier host build failed ABI preflight before
admission. The preceding old-producer fixture completed its initial task
but lacked the usage-basis marker required by current master; it
remained pending/unpriced and correctly paused. It was independently
cleaned without debt or budget changes. Two preceding local Linux setup
attempts OOM-killed before provider allocation, and their cleanup
passed. These attempts remain separate from the final recovery failure.

**Failures retained and resolved separately.** The earlier `98f05fc`
probe rejected a valid settled PRP v2 goal suffix; `eaef0d5` fixed that
reader and the fresh follow-up passed. Later component cleanup failures
were traced by a scoped exception observer to the fixture adapter
rejecting directory downloads. A bounded owned-directory transfer with
exact-byte/mode preflight corrected the fixture; source and installed
strict-cleanup cases then passed. No production guard was relaxed. The
initial local integrated control-port run hit its 30-second embedded
Postgres startup hook before executing 59 cases; 18 other focused cases
passed. A bounded rerun used a fresh owned external Postgres fixture,
applied normal migrations before Vitest, and passed all 59 cases with
the original timeouts unchanged. Its container and data were removed;
the failed attempt remains retained.

The additional opt-in full Product E2E Vitest run is not green: 5,313
passed, seven failed and three skipped. An actual `abed921e12` archive
with the same dependencies, Node and IPC permissions reproduces all
seven failures with the same reasons. Five are unchanged
workflow-policy/profile-pin assertions, including Pi profile version 37
versus the test pin 35. Two database suites failed during embedded
Postgres startup before their assertions; semaphore exhaustion is
confirmed in the observer log. The observer passes its supported
external-Postgres rerun with original timeouts and source unchanged. The
controller-database client-kill/postmaster assertions remain unexecuted
after `initdb` bootstrap failure; no external override is supported. The
unit launcher stopped before its Node-test phase, which was then run
independently and passed 181/181. The original failed full run is
preserved. These results remain distinct from the passing focused checks
and normal repository CI. The first local Product E2E typecheck also
encountered stale compiled Runner declarations; a genuine package-local
TypeScript compile refreshed them and the unchanged check passed,
without a dependency install or producer rebuild.

Earlier [Daytona campaign
41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900)
failed at sandbox creation before any agent, with the underlying
provider error unavailable. Its [sanitized
report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-41108162900-2/index.html)
remains a `cleanup_failure`. Both old creation names are absent in the
current authenticated account, but the old CI account identity was not
retained, so account equivalence is unproven. The subsequent exact
`eaef0d5` source Product E2E attempt passed image/agent setup and the
initial project, then held the controller offline for 300,003.7 ms. Its
Mac supervisor failed owned-process inspection before B readiness; no
adoption passed. Full current-company lease/label checks, provider
deletion, exact-ID 404s and complete inventory confirmed cleanup of that
attempt. Its local server and owned Postgres stopped. The prior AWS
campaigns and database qualification VM were terminated with root
volumes removed.

The old admitted test Runner lacks the newer usage-basis marker
introduced on master. The integrated component B turns retained
unpriced, incomplete accounting obligations with `costAccountingPending:
true` and no settled charge. The compatibility-reader A turn used its
legacy receipt interpretation but likewise retained pending accounting
and no settlement timestamp. This proves obligation preservation, not a
whole-run accounting migration. Runner/receiver release ordering remains
governed by the usage owner's contract.

At native implementation `abed921e12d543b62f57a56435419c0dafb0bc33`,
Greptile was 5/5 with zero unresolved threads and CI completed with 54
successful checks and five expected skips. At `0b6afc499f`, Greptile
again passed 5/5 with zero unresolved threads, but CI ended with 51
successful checks, five skips and three failures. Server 9/12 fails a
deterministic managed-skills maintenance assertion newly inherited from
master `198218bf` in merge `c80e00ad`; that test is absent from the PR
head, and the storage owner has the explicit-StorageService fix in
#18357. Runner 2/2 passed 2,283 unit cases but its runner received a
shutdown signal during the long stream probe; its cancelled step and
aggregate failure are preserved. No blind retry was used. Current
`af9e2d157d` checks are pending; the matcher-only change passes 144
focused cases and Product E2E typecheck. Earlier scheduling-sensitive CI
failures and the interrupted unsharded local test run remain recorded in
the dated plan. CI runs the repository's normal test partitions. Final
verification does not claim Railway rollout, uninterrupted HTTP service
or whole-webserver filesystem loss.

</details>

## Risks

- The task-pointer repair passes its assembled regression. The first
live repair exposed pre-acceptance ordering and the fresh review exposed
a replay lock-order issue; both are fixed and pass real Postgres tests
at `0862062`. The following live attempt exposed the
verified-running-versus-suspended preparation gate. Its narrow fix
passes 91 focused cases at `64c0b746`. The next live case exposed
synthetic attachment IDs; `7a203274` fixes that field selection and
passes real Runner authentication. The subsequent queued continuation
exposed strict retirement maintenance and a false default-home change;
`39a4343` fixes both with real Runner/Postgres and fingerprint
regressions. The `39a4343` live continuation failed again despite green
CI. `283e05ffaa` repairs the actual native-observed admission ordering
and reproduces/fixes the JSONB skill-order fingerprint drift. Both pass
targeted real-Postgres regressions and the live active plus queued
paths. The definition 6 post-C browser navigation timed out before
admission. Explicit definition 7 now verifies native continuation
through the ordinary comment API; its strict post-C native gate failed
after a real dispatch and remains historical failed evidence. The final
fc68584 repair now passes its fresh strict 49-check Linux/Daytona
campaign; the prior failed grades remain unchanged. Browser reconnection
is handed to its own workstream with the original failed evidence. No
prior failed campaign is reclassified as passing.
- Incorrect checkpoint/source binding or stale ownership could duplicate
external work. Missing evidence remains a hold; existing receipt and
finalization owners keep uncertain effects unresolved.
- Recovery must retain maintenance writes after accepted results without
allowing a stale business turn to write. Real lock-wait and generation
tests cover that boundary.
- The current two-connection paired burst profile fails its absolute
drain gate in both baseline and candidate, despite passing relative load
gates and all correctness checks. A causal capacity diagnostic selected
three slots; the final controlled cap-three profiles pass unchanged
thresholds. Full application replacement and two-server overlap capacity
remain separate. There is no new streaming writer, pool or recurring
idle scan.
- The remote sandbox and Runner must survive the recovery window.
Provider auto-stop is a separate lifetime bound from reconnect grace;
the explicit qualification fixture uses a longer idle window.
- Workspace and provider backup availability remain external
dependencies. This PR does not certify whole-webserver filesystem loss,
uninterrupted HTTP service or a Railway rollout.

## Model Used

Implementation and initial verification used OpenAI GPT-6 through Codex;
the initial hosted snapshot was not exposed. The follow-up qualification
used three OpenAI `gpt-6.1-sol` workers coordinated through Codex, with
repository tools, code execution, CI and scoped test infrastructure.
Context-window size is not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused and exact-source
application/cost checks pass; full forced CI passes; earlier local
embedded-PG setup failures remain recorded)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (exact fc68584 full CI and
auxiliary checks pass; forced complete test partitions)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(exact fc68584 check and edited summary pass; all six threads resolved)
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-31 07:33:52 -05:00
DottaandPaperclip 2aa0179c35 Merge qualified recovery grouping and prerequisites 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '0cb21925bf9901d818d456b8621ebf58e82ec119':
  Preserve native receipt grouping within each bounded worker pass
 2026-10-31 00:06:47 -05:00
DottaandPaperclip 0cb21925bf Merge qualified recovery grouping and prerequisites 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '6e8119550dc3dccafa165d5aebed21bd43bb4ee3':
  Preserve native receipt grouping within each bounded worker pass
 2026-10-31 00:06:47 -05:00
DottaandPaperclip 6e8119550d Preserve native receipt grouping within each bounded worker pass 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 00:06:19 -05:00
Dotta 96b516dcb6 Merge commit '499afce285c429f814f5cd22fd437bafabf530e7' into codex/sleep-final-qualified-18388 
* commit '499afce285c429f814f5cd22fd437bafabf530e7':
  test: align deferred recovery clocks with real execution
  Keep recovery acceptance timers on their owning clock
  fix(server): recheck reactive sources after status summary commit
  test: measure retention page fairness on a controlled clock
  fix(heartbeat): resume saved chat work on feature enablement
  test: await resolved interaction execution before outcome assertions
  Wake saved conversation assignments when Agent Chat becomes enabled
 2026-10-30 23:27:02 -05:00
Dotta 499afce285 Merge commit 'a0f5caa00241e9f2653992f47106bbe38fec130f' into codex/sleep-final-qualified-18401 
* commit 'a0f5caa00241e9f2653992f47106bbe38fec130f':
  test: align deferred recovery clocks with real execution
  Keep recovery acceptance timers on their owning clock
  test: measure retention page fairness on a controlled clock
  fix(heartbeat): resume saved chat work on feature enablement
  test: await resolved interaction execution before outcome assertions
  Wake saved conversation assignments when Agent Chat becomes enabled

# Conflicts:
#	server/src/services/instance-settings.ts
 2026-10-30 23:26:43 -05:00
DottaandPaperclip e6deb559ad test: align deferred recovery clocks with real execution 
Keep manual receipt cursor progress when chat is disabled. Follow saved cleanup deadlines after actual child-process and PostgreSQL work settles, and preserve the exact failure-backoff boundary.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 23:24:40 -05:00
DottaandPaperclip 8d027fd2b6 fix(heartbeat): resume saved chat work on feature enablement 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 22:47:56 -05:00
DottaandPaperclip 59da15066a feat(runner): deliver native Kimi Code (#18391) 
Deliver native Kimi Code setup, exact models, scoped credentials, verified packages and runtime packs. Preserve current upstream release attestations and restore verified same-run remote retry dispatch without weakening process ownership.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 22:26:15 -05:00
Dotta 98a68b492f Merge verified sleep prerequisites into 01 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit 'd287074fba0ab0365d61416dadc392cc3078eee5':
  Remove unused aggregate watchdog diagnostics
  test: settle delegated chat completion before plan assertions
  test: settle saved retry cleanup before idle assertion
  fix: wake retry scheduling only when retry state changes

# Conflicts:
#	server/src/services/heartbeat/retries.ts
#	server/src/services/native-runtime/native-provider-capacity-retry.ts
 2026-10-30 22:12:53 -05:00
Dotta d287074fba Merge verified sleep prerequisites into 07 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* commit '78bd3560a886c7fd3eea03baccb6cf49f6e8499a':
  Remove unused aggregate watchdog diagnostics
  test: settle delegated chat completion before plan assertions
  test: settle saved retry cleanup before idle assertion
  fix: wake retry scheduling only when retry state changes

# Conflicts:
#	server/src/services/heartbeat/retries.ts
 2026-10-30 22:12:31 -05:00
DottaandPaperclip 822b173281 test: preserve retry producer boundaries in receipt recovery 
Co-Authored-By: Paperclip <noreply@paperclip.ing>

* fix/sleep-forward-02b-security:
  fix: wake retry scheduling only when retry state changes
 2026-10-30 21:51:11 -05:00
DottaandPaperclip c2ff3eb921 fix: wake retry scheduling only when retry state changes 
Separate retry-schedule notifications from ordinary queued-run eligibility. Preserve exact deadlines and prove native producer commit/rollback behavior without global retry scans on ordinary agent or issue edits.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 21:49:38 -05:00
Dotta 2248af92a8 Merge branch 'codex/sleep-stack-07' into codex/sleep-stack-01 
* codex/sleep-stack-07:
  fix(ui): invalidate pending startup work on sign-out
  fix(server): park persisted deferred admission refusals
 2026-10-30 20:51:05 -05:00
DottaandPaperclip 11e92e3875 merge: preserve typed admission refusals and status-card source signals 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 20:51:04 -05:00
Dotta c0fe0feffb Merge commit '19a97bbe57d1ebd3691c70dd822488efe7e5d3e1' into codex/sleep-stack-c1 
* commit '19a97bbe57d1ebd3691c70dd822488efe7e5d3e1':
  fix(server): park persisted deferred admission refusals
 2026-10-30 20:50:39 -05:00
DottaandPaperclip 19a97bbe57 fix(server): park persisted deferred admission refusals 
Distinguish a budget or agent refusal saved on the existing receipt from genuine recovery failure. Preserve HTTP409 messages and details, grant each new wait reason its existing finite grace, and retain failure backoff for unsaved or unexpected errors. Prove actual Retry admission,25-minute SQL quiet, independent warm producer-to-process release, missing/cancelled receipts and real PostgreSQL write failure.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 20:35:57 -05:00
DottaandPaperclip 688f35763d merge: preserve attention, status-card and recovery source ownership 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 20:25:15 -05:00
DottaandPaperclip 9b4beff72b merge: preserve status-card and recovery producers in the review stack 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 20:24:40 -05:00
DottaandPaperclip 290d12cf4b merge: rehearse issue-recovery predecessor for saved receipts 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 20:24:05 -05:00
DottaandPaperclip fcc107739c merge: rehearse current master and UI prerequisite for dispatch 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 20:23:37 -05:00
DottaandPaperclip c74f5f12ba Retain moved receipt deadlines and wake released execution gates 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 19:45:15 -05:00
DottaandPaperclip ac6c5191ee Drive saved execution receipts from committed work and deadlines 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 19:22:39 -05:00
Nicky Leach 9e904ee519 perf(server): overlap independent heartbeat preparation reads (#18372) 2026-10-30 16:49:20 -07:00
DottaandPaperclip a12f68ef13 fix: preserve scoped status-card work across mutations and recovery 
Complete transactional source coverage, cached feature gating, startup scope reconstruction, off-page relative query expiry and busy-lock retry. Verify producer, persistence, failure, budget resume, compatibility and startup paths with isolated PostgreSQL and focused tests.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 17:50:38 -05:00
DottaandPaperclip 0fac69e167 fix(server): scope run retention hints to affected companies 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 17:27:26 -05:00
Dotta d334f59c98 Merge remote-tracking branch 'origin/master' into codex/sleep-status-card-evaluation 
* origin/master:
  fix(evals): support official TestOps job metadata (#18385)
  fix(config): require shared storage for managed cloud (#18350)
  feat(skills): make managed content independent of web-server disk (#18334)
  Add a portable agent question and answer resumption journey (#18346)
  fix(ci): isolate test selection and require current CI CODEOWNERS (#18376)
  fix(evals): capture TestOps registration after watch (#18375)
  fix(runner): allow session reset after confirmed task retirement (#18373)
  fix(evals): batch task search result readback across renders (#18370)
  fix(ui): keep task plugin views available for manual access (#18363)
  test: wait for the evidence expiry worker to settle (#18241)

# Conflicts:
#	server/src/services/company-skills.ts
#	server/src/services/native-runtime/paperclip-runner-tool-authority.ts
 2026-10-30 17:25:31 -05:00
DottaandPaperclip 4af412eaea refactor: drive status-card evaluation from committed work 
Move generation sequencing into a feature-owned deadline worker and retain persisted evaluation, eligibility and relative-search obligations. Add transactional producer hints and initial PostgreSQL acceptance coverage; follow-up completes the producer audit and remaining lifecycle gates.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 17:25:18 -05:00
DottaandPaperclip 2955e0e22e fix(server): scope run retention hints to affected companies 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 17:06:46 -05:00
DottaandPaperclip 15cc12dba6 fix(heartbeat): scope and batch dispatch wakeups by owner 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 17:06:23 -05:00
DottaandPaperclip b1104c6dd7 fix(server): scope run retention hints to affected companies 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 17:04:31 -05:00
DottaandPaperclip 0f3017c96d perf(server): archive attention from source deadlines 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 16:44:21 -05:00
DottaandPaperclip 098111c2ca test(heartbeat): retain process output and lifecycle acceptance evidence 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 16:42:12 -05:00
Dotta b57f5e5869 Merge remote-tracking branch 'origin/master' into codex/sleep-run-task-recovery 
* origin/master:
  fix(config): require shared storage for managed cloud (#18350)
  feat(skills): make managed content independent of web-server disk (#18334)
  Add a portable agent question and answer resumption journey (#18346)
  fix(ci): isolate test selection and require current CI CODEOWNERS (#18376)
  fix(evals): capture TestOps registration after watch (#18375)
  fix(runner): allow session reset after confirmed task retirement (#18373)
  fix(evals): batch task search result readback across renders (#18370)
  fix(ui): keep task plugin views available for manual access (#18363)
  test: wait for the evidence expiry worker to settle (#18241)
 2026-10-30 16:36:30 -05:00
DottaandPaperclip 922ae50f3f test(heartbeat): verify dispatch against real PostgreSQL clocks 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 16:36:26 -05:00
DottaandPaperclip 0e15c92073 feat(skills): make managed content independent of web-server disk (#18334) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use installed skills and exact version pins during execution.
> - Managed skills still depended on mutable directories on the web
server.
> - A replacement server could lose acknowledged edits or remove an
unassigned skill whose directory was absent.
> - This pull request makes immutable versions the authority for managed
content and installed repository reads.
> - The server can rebuild delivery files from durable content after
replacement.
> - This establishes skills readiness for ephemeral web servers. General
server draining and workspace recovery remain separate work.

## Linked Issues or Issue Description

**Subsystem affected**

Skills services, shared contracts, database receipts, Skill Studio, and
runtime delivery.

**Problem or motivation**

A fresh web server must discover, read, edit, and deliver protected
installed skills using the same database, object storage, secrets, and
supported application artifacts. It must not need its predecessor's
files or process memory.

**Proposed solution**

Use `managed` ownership with a null source locator. Read installed
content by immutable version. Prepare packages before a short
publication transaction. Persist execution selections before
materialization. Recover exact pinned runtime assets and reject deletion
while active or recovering pins exist.

**Alternatives considered**

Reconstructing a mutable installed directory preserves the old
source-of-truth ambiguity. Permanent dual writes create two competing
authorities. This change uses one managed model after a bounded legacy
conversion.

**Roadmap alignment**

This improves the existing Skills Manager, Studio, version history, and
Cloud execution capabilities. The maintainer supplied the implementation
contract. This does not add a separate skills product or deploy fleet
changes.

Related work: #18082 supplies immutable package storage. #17273
addresses edits outside the API for linked local skills; this PR keeps
that source category separate. #2648 is a closed earlier
ephemeral-filesystem proposal.

## What Changed

- Add managed ownership and focused content publication and execution
materialization modules. Keep the archive format and package storage
tables.
- Publish managed create, save, delete, fork, rename, restore, and
catalog content with version checks. Add durable create receipts.
Preserve file-save receipt responses.
- Return file version identities. Keep Studio drafts after a conflict
and provide an explicit reload action. A delayed reload cannot overwrite
newer typing or navigation.
- Persist exact runtime selections, including empty selections. Rebuild
pinned assets and verify their admitted digests on a replacement server.
- Preserve source access on forks and disconnected copies. Keep the
existing workspace policy category for managed skills.
- Add bounded conversion, eligibility, and independent verification
phases. Load persisted Cloud configuration in the standalone CLI without
seeding or changing it. Reject partial current versions against
installed inventories; recover only complete saved content or the exact
source. Document storage migration blockers and the compatible rollback
floor.
- Add separate-process PostgreSQL/S3 qualification and update domain,
retention, policy, and UI regressions.
- Pass the plugin worker manager into Studio execution, admit restricted
`skill_test` mode through native Runner contracts, and preserve admitted
scope across native and REST tools. Native tests cannot create ordinary
tasks, browse company data, or use external tools.
- Select the exact durable Studio revision even when unassigned or Beta
skills is off. Persist Studio output and audit atomically with native
task completion, preserving the result on replay.

## Verification

- Final revision `13e8af480b88ba6fea5737bac4bd81104d604792` passed full
CI: all 49 workflow jobs completed successfully ([run
41127396098](https://github.com/paperclipai/paperclip/actions/runs/41127396098)).
This includes typecheck, test suites, Runner checks, browser checks and
build. Greptile review 15 is 5/5 on that same revision, with zero
unresolved threads. The PR is ready for review.
- Local workspace typecheck and build passed after integrating master.
Focused checks passed: 33 execution/completion cases, 52 Runner HTTP
cases, 16 managed-content cases, 17 Studio cases, the new real-process
CLI regression, map checks and UI token gates. The CLI regression proves
inspection does not seed missing Cloud configuration. Server typecheck
passed after that fix.
- The independent-process PostgreSQL/S3 matrix passed through normal
HTTP skill routes. It covers A → B → C replacement, lost commit
responses, upload crashes, concurrent saves, binary contents and
executable delivery. Metadata listing and warm selected delivery retain
zero object downloads. Local timing measurements are recorded; they do
not establish hosted latency.
- Hosted A → B → C passed on the owned disposable test tenant with
external Postgres, S3 and a real remote Paperclip Computer. Predecessor
temporary homes were absent. C read and executed B's acknowledged
managed save. New runs used repository v1 while v2 was installed. Hashes
matched an independently checked exact upstream commit.
- Compatible image rollback to `6518b323` passed. Studio retained the
unassigned fork's C save. A fresh remote run executed the newer managed
content and historical repository pin. The actual shell event confirmed
expected hashes, stdout, exit 0 and mode 0555.
- Runtime candidate `9200e3fcb4` deployed successfully ([Cloud run
41124147876](https://github.com/paperclipai/paperclip-cloud/actions/runs/41124147876)).
The final Studio test delivered only the unassigned fork's exact v3,
executed its script and persisted a successful output snapshot.
Independent command-event inspection confirmed all three hashes, stdout,
exit 0 and mode 0555. All 14 installations were version-backed.
Independent verification checked all 25 versions: 23 remote objects, two
bundled database snapshots, zero blockers.
- The deployed image is `9200e3fcb4`. Later master integration, the
client-only delayed-reload draft guard and the CLI bootstrap have
separate automated proof. The CLI bootstrap was also exercised against
the tenant's stored configuration and storage through the same read-only
services. This PR does not claim that the deployed image is the final PR
revision.
- Live Studio checks exposed worker wiring, admission, access scope,
exact selection and completion-publication gaps. They are fixed. Failed
attempts remain in the evidence. The full local suite exceeded 80
minutes under host contention and was stopped. Later combined/isolated
local attempts also timed out; they are not counted as passes. Full CI
covers the integrated suites. Earlier CI found an obsolete restart-scope
fixture, which was corrected and passed all 52 HTTP cases.
- Reproduction and migration:
[`doc/skill-ephemeral-web-servers.md`](doc/skill-ephemeral-web-servers.md).
Source revisions, deployments, versions, run IDs, hashes and failure
history:
[`doc/plans/2026-10-30-ephemeral-web-skills.md`](doc/plans/2026-10-30-ephemeral-web-skills.md).

## Risks

- Legacy conversion requires stopped admission and retired
filesystem-authoritative writers. Pre-conversion binaries are not
supported rollback targets afterward.
- Cloud readiness requires remote object storage. Local provider
receipts and provider mismatches remain blockers until objects are
copied and verified.
- Linked local sources retain filesystem semantics and are outside the
cloud guarantee.
- Cold reads still fetch the existing archive format. Warm runtime
preparation retains the verified-cache path.
- Bundled reconciliation preserves an installed head. Existing release
snapshots remain selectable. An older server must not replace a newer
head with its shipped files.
- No production volumes, fleet defaults, or general deployment
coordination change in this PR.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
execution, and browser use. The exact backend model ID and
context-window size are not exposed in this session. No subagents were
used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 16:18:27 -05:00
DottaandPaperclip 294d8ca1c2 perf(heartbeat): schedule retry promotion from durable deadlines 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 15:53:26 -05:00
DottaandPaperclip 9f524ec1a7 fix: persist native Runner usage before transport acknowledgement (#18354) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent execution must preserve usage so Paperclip can price work and
enforce budgets.
> - Remote Runner execution can outlive its web receiver.
> - The receiver previously acknowledged Runner events before its later
usage recorder saved the accounting receipt.
> - This pull request saves required native usage receipts before that
transport acknowledgement.
> - A replacement receiver can accept the same retained report and
create one ledger charge.

## Linked Issues or Issue Description

**What happened?**

Native remote Runner usage could leave its durable outbox before the web
receiver saved its accounting receipt. The later recorder required a
local receipt directory. A replacement web receiver could lose the only
remaining report when the database write failed.

**Expected behavior**

The Runner retains unaccepted structured usage. Postgres owns accepted
receipts. Repeated delivery preserves one receipt for each source report
and one charge for the run. Missing usage keeps accounting pending.

**Steps to reproduce**

1. Start a supported built-in native remote Runner.
2. Make the receipt database insert fail after the provider reports
usage.
3. Replace the web receiver and its local directory.
4. Reconnect the same Runner and inspect the saved receipt and ledger.

**Paperclip version or commit**

The initial investigation used
`ebd05e9f4c88c66af7473db86bbaacfd1544aa0a`.

**Deployment mode**

A web server with a remote native Runner and shared Postgres.

Related work: #18327 owns output logs and provider traces. This change
has no dependency on its S3 export path. #18031 covers interrupted Codex
receipt recovery. #18005 covers legacy spool reads. #17516 covers Codex
stop-boundary retention.

## What Changed

- Add native producer receipt delivery through the existing Runner
callback before transport acknowledgement.
- Reuse `run_usage_receipts`, stable source sequences, and deterministic
report identities. Reject conflicting replay and retry a lost commit
response with the same receipt.
- Restore per-turn usage from structured receipt facts. Preserve
provider turns that share one PRP envelope turn.
- Bind retained warm transport callbacks to the current run authority
epoch, allowing old attachment control acknowledgements while preserving
accounting ownership.
- Persist Codex session baselines and cumulative totals in the existing
provider checkpoint. Mark qualified whole-run deltas, preserve them
across resume/restart/attachment, and leave old unmarked reports
incomplete and unpriced.
- Normalize the bounded final usage tail before the terminal event.
Retain later increases for explicit reconciliation, block further
attachment, and fail an affected successor without changing its
baseline.
- Complete runs from the accepted immutable receipt and its exact saved
rate-card amount; preserve existing live-budget enforcement.
- Skip the local receipt recorder for supported built-in native remote
Runner execution. Keep local, legacy, and external Dot/Muse behavior
unchanged.
- Keep pricing, reservations, ledger settlement, source fencing, and
budget holds with their existing owners.
- Put the actual Runner integration test in the existing Rust-cached
native test lane. Build before opening Postgres and give cold
compilation its own bounded budget.

## Verification

- Fresh Greptile review is 5/5 on
`e7376a50c8162770da1d581ecf25c8cffac64520`, with no unresolved findings.
Parent and independent peer review approved the final changes.
- Real PostgreSQL 18.6 and a real Rust Runner exercise failed receipt
insertion, receiver-directory loss, replay from the same producer, and
exactly one 25-cent ledger charge. The actual accounting spool path
stays absent. Optional logs/traces are disabled. The provider is
controlled; no paid request is made and the fixture never pre-saves the
expected receipt.
- Producer checks passed 32 usage/provider tests,37 terminal controls,
corrected completion, all 3 previously failing exit cases, and both full
transport goal-continuation cases. Rejected active-goal completion
retains its 120-token tail before failure. Baseline 100→successor150→33
duplicate old snapshots→old120 retains explicit incomplete
reconciliation evidence and positive budget exposure.
- Heartbeat executor checks passed 982/982. PostgreSQL tests preserve
exact 1.002 USD pricing, the 100.2-cent ledger amount, budget holds,
source takeover, partial/final receipts, cost-only corrections, equal
rate-card replay, and stale-completion interleaving. Newer incomplete
usage remains pending rather than settling an older snapshot.
- Runtime replay and strict usage-schema checks passed 2/2; generated
contracts and Rust formatting pass. Test-partition contracts passed
32/32.
- [Final combined-head
evidence](https://github.com/paperclipai/paperclip/pull/18354#issuecomment-6531526158)
records full workspace typecheck/build and 41/41 accounting tests
against all five cloud PR heads, including this final accounting commit.
It includes commands and evidence limits.
- Exact-head CI is green: 53successful checks, 6intentional skips, and
the legacy Snyk status passed. Typecheck, build, Runner, server,
browser, and aggregate gates pass. Runner Vitest1/2 initially failed an
unchanged mocked Antigravity login fixture; its focused case passed
locally, then one unchanged job retry passed 131 files/1995 tests,
including all 11 Antigravity tests. The initial failed attempt remains
recorded in CI.

Initial local attempts exposed the host SysV semaphore limit and
corrected harness path, IPC readiness, and queue-counter defects. A
temporary test loader selected an existing POSIX-semaphore PostgreSQL
18.6 build without changing repository dependencies. An optional
expanded debug provider run was interrupted at its unrelated 8192-turn
rollover stress case; no full-file local pass is claimed. Required
failure cases were run directly, and the full Runner CI remains
required.

## Risks

- The guarantee covers built-in native remote Runner execution. Local
native execution, legacy adapters, external Dot/Muse execution, and
server-originated model calls retain their existing contracts.
- This does not provide traffic switching or general web-server
failover. The existing controller identity, checkpoint, and generation
gates still apply.
- The receipt JSON gains optional structured producer facts. No schema
migration or accounting S3 spool is added. Text output and raw trace
payloads are excluded.
- Codex accounting authority requires a Runner artifact built from this
provider baseline implementation and qualified by digest. Update
receivers and strict usage-schema readers before releasing that Runner;
PRP does not negotiate this payload field. Existing unmarked Codex
reports remain pending/unpriced. No deployed Runner artifact is claimed
fixed.
- Usage arriving beyond the bounded terminal tail may require manual
reconciliation or a fresh session. Its evidence remains in the existing
producer checkpoint/outbox; no adjustment ledger or recurring scan is
added.

## Model Used

OpenAI `gpt-6.1-sol` assisted with the implementation, tests, and
documentation through code execution and repository tools. The context
window size is unavailable.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 15:20:42 -05:00
DottaandPaperclip dbff3747b8 feat(server): persist run logs and provider traces in S3 (#18327) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators use run logs and restricted provider traces to inspect
agent work.
> - Local log and trace files depend on the web server's persistent
volume.
> - That dependency prevents overlapping volume-backed Railway
deployments.
> - This PR reuses the instance S3 store for batched log and trace
payloads with conditional publication.
> - Existing Postgres rows retain their contracts; native export
recovery reuses events already committed before Runner acknowledgement.
> - S3-configured servers automatically copy and verify retained history
before a later deployment cutover.

## Linked Issues or Issue Description

**Subsystem affected**

Server run logs, restricted provider traces, storage providers, and
migration tooling.

**Problem or motivation**

Replacing the web server and its disk must preserve committed log output
and retained traces. The current S3 mirror can repeatedly upload a
growing file. Moving those payloads into Postgres would add database
writes, WAL, replication, and backup work.

**Proposed solution**

Reuse the existing instance storage configuration, S3-compatible
provider and credential chain. Store new run logs under run-logs/ and
opted-in raw traces under provider-traces/, below the existing prefix.
Cloud keeps its existing bucket and bucket-scoped credentials per stack.
Add no new configuration switches. Store immutable payload segments and
bounded binary range indexes directly in S3. Publish each batch with an
ETag-conditional head. Keep existing database ownership and lifecycle
records. Recover canonical native exports from their existing validated
control event source, checkpointing export progress in S3. Add no new
payload writes to Postgres. Preserve local defaults and migrate settled
history with verified byte counts and digests.

**Alternatives considered**

Postgres payload chunks add control-plane load. Repeated whole-file
mirrors amplify transfer work. The first index prototype left superseded
index nodes; the final binary index keeps all normally published nodes
reachable.

**Roadmap alignment**

This is a storage prerequisite for replacing the main web/API server. It
does not enable deployment overlap, detach a volume, or change native
Runner protocol acknowledgements. The maintainer approved this
implementation scope. ROADMAP.md has no duplicate implementation item.

Related PR: #16770 improves the existing in-flight mirror. This PR
preserves that legacy backend and uses the existing instance S3 profile
for new journals.

## What Changed

- Automatically select S3 segments for new logs and opted-in traces when
instance file storage uses S3. Reuse bucket, endpoint, region, prefix,
path style and credentials. Remove separate backend switches. Keep
local-disk instances local.
- Share the existing storage configuration resolver with product files.
Preserve config-file settings, environment precedence and the existing
root prefix. Keep pre-existing run-log mirror settings for legacy
history.
- Add bounded buffering, conditional ownership fencing, immutable range
reads, stable byte offsets, and completion hashing.
- Dispatch all existing log consumers through the recorded backend.
Coalesce output progress writes and publish live log messages after
storage receipts.
- Recover interrupted canonical native exports from their existing
durable source. Exclude separate credential, cleanup, and reprojection
records. Replay missing callback ranges and deduplicate late delivery.
Repair settled archives after controller lease expiry. Bound shared
repair work and stop it when all readers cancel. Serve the published
prefix when best-effort repair fails.
- Add encrypted optional trace channels, replacement reads, durable
deletion tombstones, and deadline-driven local/external expiry cleanup
independently of heartbeat scheduling. Stream historical source and
remote verification. Propagate worker cancellation through trace open,
conditional-write probing, upload, restore and reads. Fence deletion
even before the first object head exists; abort and drain owned
captures.
- Retry transient storage errors, distinguish 409 from failed
preconditions, and backpressure bounded log admission. Track interrupted
trace imports explicitly so ordinary local deletion stays independent of
S3.
- Automatically migrate retained history through the existing delivery
coordinator. Process one log and one trace per batch. Retry unresolved
history with backoff and stop scans after completion. Keep the
inventory/import command as optional diagnosis and repair. Verify source
and remote bytes before switching references. Verify finalized native
size and digest before setting its export cursor. Require ordered source
coverage when completeness metadata is missing. Preserve original trace
expiry. Block active, missing, corrupt, and unproven sources.
- Preserve identity and run-secret redaction in live and recovered
native logs. Register runtime log redaction secrets once through the
existing encrypted registry. Keep agent private keys in their existing
owner.
- Add storage, migration, retention, UTF-8, and route-privacy coverage,
plus operating documentation and qualification tooling.
- Add no schema, payload table, per-record SQL, production dependency,
or lockfile change.

## Verification

- Final fixes at `b3a44c898e` passed all 116 selected startup,
retention, attribution and cancellation controls. The disabled-scheduler
cleanup and concurrent-repair defects reproduced red before the fixes. A
timed-out repair still prevents later storage reads.
- The merged source passed 124 storage and real-Postgres Cloud
configuration tests. These include automatic history migration, existing
environment/file vault keys, source-directory removal, five real-SDK
stalled-request cancellation cases, shared capability-probe cancellation
and multi-chunk trace ordering. Saved Cloud S3 settings work after the
legacy config file is gone.
- Workspace typecheck and full build passed at the merged parent
`19f702d780`. Server typecheck and build passed again at `b3a44c898e`.
- Actual app acceptance: finish a process-agent run through the normal
API, stage its retained local-file input, then restart. Automatic
migration reports one migrated item and zero blockers. Remove the
original web home and restart at `b3a44c898e` with an empty home, the
existing S3 profile and the existing vault key file. The normal log API
preserves all five producer lines and exactly 788 bytes, with SHA-256
`74d9ae469da28bfe89b9b50a0e132aec24fe9e6d686496ed5e4ffa6167151036`. No
log-specific storage settings are present.
- Actual expiry acceptance: with heartbeat scheduling disabled, a trace
remains unexpired after startup and later reaches its fixture deadline.
The worker marks the row deleted and tombstones both S3 channel heads.
The retained run log stays byte-identical. The process fixture emits no
provider protocol frames, so nonempty encrypted payloads are qualified
separately through the real SDK round trips.
- Isolation tests use identical run identities in two stack buckets.
Cleanup in one existing prefix leaves another prefix, another company
and product files intact. Read-only Cloud source inspection confirms a
bucket and bucket-scoped credentials per stack. Four existing Cloud
storage environment-contract tests passed. No Cloud provisioning changed
and no new live vendor permission probe was performed.
- Earlier qualification passed 1,014 affected storage/native tests and
27 privacy-route tests with isolated Docker Postgres. An abrupt process
test commits 1,000 records, receives receipts, dies by SIGKILL and loses
its directory; another process reads every byte and continues. The real
Rust Runner experiment acknowledges 32 records while the export cursor
is zero, then continues the same Runner PID after controller loss. All
64 records export once with exactly 64 existing control rows. Repeated
snapshot/Stop commands completed in 162/81 ms.
- Emulator workload: 1,000 burst records used two batch PUTs. Eight
steady streams produced 240 records with 104 conditional PUTs and 72
GETs over 6.9 seconds. These are synthetic operation counts, not fleet
sizing or provider prices.
- Failure evidence is retained. Compact native events over 64 KiB,
missing file-key support, disabled-scheduler retention and repair
failure handling all reproduced before their fixes. Fixture repairs
addressed cold module setup, localhost/IPC permissions, the S3
bucket-name limit and a mode-0700 crash-test directory. Full local `pnpm
test:run` was stopped after host PostgreSQL semaphore exhaustion;
affected suites used isolated Docker databases instead. Native package
library symlinks were restored with the package's standard hydration
helper.
- The `19f702d780` browser session-recovery case timed out while saving
a task title. All eight browser shards passed on `b3a44c898e`, including
that case, with no UI behavior or assertion change. Fresh-head CI passed
at `b3a44c898e`: 55 successful checks, three skipped checks and one
neutral check. The branch is conflict-free.
- Map and module-boundary checks passed. Current-head Greptile is 5/5
with zero unresolved review threads.

Reproduce the opt-in storage suites with
`PAPERCLIP_LOG_TEST_S3_ENDPOINT` and a dedicated
`PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL` ending in `_test`. See
`doc/run-evidence-storage.md` for operating limits, automatic migration,
optional diagnostics and rollout qualification.

## Risks

- Keep the existing per-stack bucket, prefix and vault key stable across
replacements. Relocating that storage is a separate whole-instance
operation.
- This release automatically changes new log storage for existing
S3-configured instances. Local-disk instances remain local. Canonical
native source replay is qualified with a deterministic real Runner;
actual remote execution and production-provider qualification remain
rollout gates. Generic server-only diagnostics and optional raw trace
tails without a surviving source can be incomplete after a crash. This
does not establish full zero-downtime deployment.
- Before deploying this release, qualify the actual S3 provider. It must
enforce conditional writes and pass read, version, deletion, workload,
and latency qualification. A capability probe rejects providers which
ignore preconditions. Moto is emulator evidence only.
- Dirty output is bounded and unconfirmed until S3 publication. Storage
failure leaves persistence incomplete while task results and usage can
settle.
- Trace expiry is not yet represented by Cloud's external wake contract.
Outstanding trace deadlines conservatively block sleep.
- Automatic history copying requires the original source volume. Keep it
until all retained resources pass verified import and clean-filesystem
reads. Stop old binaries from creating new local-file runs. Missing or
corrupt history remains blocked and visible in server logs. Rollback
requires a compatible reader against the same external state. Ordinary
object deletion does not erase retained historical object versions or
backups.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, code execution,
and browser verification. The runtime did not expose a context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 15:17:35 -05:00
DottaandPaperclip f5e5b05a33 Preserve native workspace recovery across Stop and sandbox reuse (#18332) 
Land the thirteen-file native recovery fix independently from #18245.

Retain verified Stop evidence and queued user comments. Retire stale
preparation ownership atomically and require paired sandbox lease
handoff receipts before workspace recovery.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 10:57:53 -05:00
Nicky Leach fba0ab8162 fix(runner): preserve GitHub helpers and Codex probe policy (#18212) 2026-10-30 07:42:10 -07:00
DottaandPaperclip ed8fae8dbb fix(workspaces): preserve completed shallow sandbox exports (#18221) 
## Thinking Path

> - Paperclip coordinates agent work and preserves its results.
> - Remote execution copies completed workspace changes back to the
host.
> - An agent can fetch a published branch with shallow Git history.
> - A full Git bundle omits the shallow boundary metadata and can fail
host connectivity checks.
> - This pull request carries the boundaries through the full-bundle
fallback and imports through normal Git fetch.
> - Recovery must also recognize completed native work on a reusable
lease without treating it as ordinary executor reuse.
> - Completed work can be saved without another provider turn, while
existing host history remains connected.

## Linked Issues or Issue Description

**What happened?**
Sandbox finalization can fail with a missing parent commit after an
agent checks out a separately fetched shallow branch. Re-exporting a
full bundle repeats the failure because Git bundles do not encode
shallow boundaries.

**Expected behavior**
Import the completed branch and saved files. Preserve existing host
ancestry and concurrent host edits. Keep normal Git connectivity and
workspace merge checks.

**Steps to reproduce**
1. Prepare a sandbox from one host commit.
2. Fetch a different published branch with depth one and check it out.
3. Save a result file and restore the sandbox workspace.

**Paperclip version or commit**
Reproduced against master before this repair. The regression fails with
the missing-parent error before the production change.

Related public context: #15268 covers rewritten history integration.
This repair handles shallow bundle transport before integration.
Searches found no equivalent open repair.

## What Changed

- Export shallow boundaries beside the full bundle when the delta import
lacks history.
- Unpack into a temporary shared bare repository. Keep parent
connections already available on the host and import with Git's shallow
fetch protocol.
- Reject malformed boundary values and retain existing connectivity,
branch, and concurrent-work merge guards. Keep boundary probes within 15
seconds without consuming the separate 60-second Git import budget.
- Let source-bound preservation recover reusable leases only for the
finalizer-selected, accepted successful native result and matching
completion contract. Retain provider-stop, physical ownership and
host-writer checks; ordinary reusable sources remain blocked.
- Classify invalid Git metadata separately from omittable unsafe
archives. Keep native finalization failed and preserve service-owned
file recovery; carry the sender's dedicated rejection across the command
boundary without transient retries.
- Share the failure allowlist across bounded run/activity projections
and cancelled-preservation selection. Migration `0416` rebuilds the
matching partial index; task chat retains the non-retryable recovery
notice.
- Add real Git, filesystem sandbox and database recovery regressions.
Update execution semantics and affected verification maps.

## Verification

- Red: real filesystem sandbox restore failed with the same
missing-parent/revision-walk error before the export repair.
- Green: 240 focused Git, workspace restore, sandbox transport and
native-finalization tests pass. Coverage includes both transfer paths,
malformed/oversized metadata, a genuinely shallow host with its original
boundary intact, and concurrent host work with both merge parents
verified.
- Red/green: a 14-second simulated probe left the final real Git fetch
only 45,893 ms before the budget split. It now retains over 55 seconds
after the probes; a 16-second probe exhausts its own budget before host
import.
- Red/green: an oversized read progress report was retried three times
before its distinct metadata classification. It now aborts the first
read before payload consumption, leaves the host tip unchanged and does
not retry.
- Red/green: the real metadata error was swallowed by best-effort export
and native finalization incorrectly returned success. Both regressions
now pass; all 240 focused Git/restore/native-finalization tests pass. A
PostgreSQL/filesystem fault-injection test also verifies retained files,
failed run, service-owned recovery barrier, no preservation receipt and
no new wake.
- Red/green: duplicate metadata produced 47 Git process starts before
deduplication; the bounded-work regression now passes. Oversized byte
and boundary inputs fail before Git probes.
- Recovery red: both reusable lease policies exhausted recovery despite
a selected accepted successful result. The assembled
PostgreSQL/filesystem path now passes for retained and both reusable
policies, including idempotent re-admission after exhausted automatic
recovery, saved files, no agent wake, and once-only existing-result
finalization.
- Source verification controls reject ordinary reusable leases,
rejected/failed/unselected results, another physical allocation owner,
and an active host writer without changing recorded state.
- The 78-test PostgreSQL/filesystem recovery/source suite passed before
the metadata classification correction. Its new persisted
metadata-failure regression also passes. Full workspace typecheck and
build pass at final head `69f10daab2`.
- `node scripts/check-maps.mjs` and `git diff --check` pass.
- Full local `pnpm test:run` attempts did not finish. They are not
counted as passes. Early focused runs exceeded 5-second and 30-second
fixture deadlines; the final serial suite passed all 176 tests with a
30-second per-test budget in 56.8 seconds.
- Red/green: both bounded PostgreSQL projections dropped the new code in
oversized results. All 19 source/projection cases now pass, plus an
actual index-predicate test that excludes a metadata repair hold and
still selects ordinary cancelled work.
- Red/green: an oversized sender `.git/shallow` lost its error type
before host validation. The real transport test now verifies one typed
rejection, no shallow-payload read, and an unchanged host tip. All 465
command/sandbox/task-chat tests pass, and token/migration gates pass.
- Full typecheck/build pass after these final contract corrections at
`69f10daab2`; all 20 PostgreSQL source/projection/index contracts pass.
All current-head CI checks passed; Greptile is 5/5 with no unresolved
review threads. Merged #18230 supplies the CI-cache and Linux
shell-fixture baseline repairs; their duplicate changes have been
removed from this PR. The final Runner lane passed after one justified
rerun of an unchanged authentication fixture timeout (all 11 fixture
tests also pass locally). Previous CI failures and incomplete local runs
remain recorded; live deployment and retained-run recovery remain
separate verification.

## Risks

- The fallback creates a temporary shared bare clone. Metadata is
limited to 64 KiB and 256 unique boundaries, duplicate/parent probes are
cached, and Git import has a 60-second command budget plus a separate
15-second boundary-probe budget. It runs only after a missing-history
import failure.
- Incorrect boundaries could hide real host ancestry. The importer
retains available parent connections, and regressions cover that case.
- The full-bundle fallback can transfer a bundle twice. This preserves
the existing fast delta path; removing that extra transfer is outside
this correctness repair.
- Preservation eligibility expands only for completed native exports on
reusable leases. It creates no provider turn and leaves ordinary
reusable leases ineligible. Database/filesystem regressions cover result
selection and competing physical/host ownership.
- The migration rebuilds one partial index; no tables, columns or user
records change. There are no agent execution admission, model, trust,
recovery permission, workflow or lockfile changes.

## Model Used

OpenAI Codex, GPT-6, with reasoning, code execution, and browser tools.
The exact serving variant and context-window size are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 20:02:53 -05:00
DottaandPaperclip 2aa3ea47e3 Reduce restored workspace overhead and trace cache publication (#18240) 
## Thinking Path

> - Paperclip runs agents in isolated workspaces.
> - A native sandbox turn must restore its workspace before the provider
starts.
> - Restored Git metadata can cause expensive work in the provider's
first snapshot.
> - Cache replay also hashes the same archives twice, and workspace
helpers open sessions that do not need output.
> - This pull request removes that repeated work and refreshes the
restored index before OpenCode starts.
> - It also adds cache-publication timings so the next measurement can
separate quota checks, lock waiting, and archive IO.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Native sandbox workspace preparation and copyback.

**Current behavior**

Cache replay validates shared archives, copies them, and validates them
again. Exit-status-only helpers use persistent sessions. A restored
checkout can make OpenCode's first snapshot recheck much of the
repository. Cache publication has an unexplained interval after archive
verification.

**Proposed behavior**

Validate the private archive copy that staging will consume. Use the
existing one-shot command path for helpers that only need an exit
status. Refresh the Git index after workspace transfer for a new native
OpenCode sandbox. Keep the refresh inside the awaited inbound barrier
and use the existing command timeout. Add nested publication spans.

**Reason and benefit**

Two matched local component pairs reduced preparation plus provider time
from 8.563/8.553 seconds to 7.313/7.511 seconds, including the refresh
cost. Two alternating remote copyback pairs reduced the median from
9.110 to 7.943 seconds. These are component results. They do not
establish an end-to-end hosted improvement.

Related: #18181 and the earlier closed #18195. This implementation uses
the existing fully awaited transfer boundary. It does not use an
optional preparation deadline.

## What Changed

- Materialize cache hits into caller-owned staging and validate those
exact bytes once. Remove incomplete copies on failure.
- Select the existing one-shot path for exit-status-only run and removal
helpers. Output-reading helpers keep their existing behavior.
- Refresh restored Git stat metadata before OpenCode starts. Enable this
only for newly acquired native OpenCode host-current workspaces. Skip
recovery and retained sandboxes. Keep dirty files unstaged, accept Git's
dirty-worktree status, and propagate real command errors.
- Trace cache capacity checks, eviction, the body after lock
acquisition, archive copy, sync, and hashing. Keep the operator endpoint
gate and bounded phase names.

## Verification

- On the rebased branch: 202 focused tests pass: 152 workspace
seed/cache/runtime tests, 27 managed command tests, and 23 native
workspace/tracing tests. Adapter TypeScript build and server typecheck
pass.
- Tests cover corrupted copies, cache changes after private
materialization, dirty working trees, quoted paths, recovery, the
awaited refresh barrier, real command failure, and company-lock
contention.
- A real-filesystem probe through the server phase mapper verified
parentage and endpoint-off behavior. A controlled lock wait stayed
outside the locked-publication span. No external export was used.
- A full local Core/Daytona replay on the pre-rebase implementation
saved its synthetic reply. It retained the pinned large public
workspace, all 209 skill files, positive Git context, and 66 tool
schemas. Both model calls stayed on sandbox loopback. Index refresh took
0.644 seconds and the first provider snapshot took 0.499 seconds. The
run took 162.004 seconds, including large laptop uploads; it is not a
hosted acceptance result or a matched end-to-end comparison. The failed
installer attempt remains recorded separately. Normal cleanup removed
all owned sandboxes.
- The current head passed 53 CI checks, with two optional skips.
Greptile is 5/5 on the exact head, with no open review threads. No
deployment or hosted latency claim is included.

## Risks

- Index refresh adds one awaited command. It uses the existing timeout.
A real Git failure stops preparation.
- Cache hits need temporary disk space for a private copy. Existing
corruption fallback and per-run recovery archives remain.
- One-shot command transport must preserve exit status and timeout
behavior across providers. The agent's native session and output-reading
helpers are unchanged.
- Local component gains are small relative to the remaining hosted
latency. Batch the fixes with the diagnostic spans; measure the actual
deployed path before claiming success.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, and code
execution. The session does not expose the exact model SKU or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 19:32:28 -05:00
DottaandPaperclip 61bd4ac691 feat(agents): set up Antigravity with saved Google subscriptions (#18189) 
## Thinking Path

> - Paperclip manages agents and their work.
> - Users choose providers and managed accounts while adding agents.
> - Antigravity was missing from the ordinary additional-agent picker
and its saved Google path did not complete native readiness.
> - A user needs a successful Test, an enabled task action, and a stable
account binding.
> - This pull request connects the ordinary setup flow to the native
candidate.
> - It also preserves retired credential refreshes across database
failures and server restarts.

## Linked Issues or Issue Description

Built on merged #18188 and #18187. Refs: #12503, #14938 for related
legacy CLI proposals.

**Agent or provider**

Google Antigravity ACP 1.4.3.

**Why this adapter is useful**

Users need to set up an Antigravity agent and run real work through
their managed Google subscription.

**How the agent is invoked**

Native Paperclip Runner uses the pinned Antigravity ACP runtime. Exact
operator admission remains required; the local candidate result does not
enable general release.

## What Changed

- Show Google Antigravity in the ordinary adapter grid and preserve the
draft choice when setup resumes.
- Reuse the same user’s saved personal default Google subscription and
discover the exact low model through the native account.
- Run a prompt-free native Test and wait for authoritative readiness
before enabling task assignment.
- Carry the managed account through native runs and recover retired
refreshes through the existing encrypted, fsynced journal with exact
generation checks.
- Retain the latest rotated credential through verification, storage,
and partial home-removal failures. Retry the exact account’s retained
cleanup before another provider use.
- Reconcile cleanup before taking the credential mutation lock, fence
owners arriving during lock acquisition, and support nested lock holders
without a join cycle.
- Bound sign-in retirement outside database transactions and preserve
claims for the existing cleanup owner; one pending sign-in does not
block unrelated accounts.
- Cover Gemini and Pi API-key recovery, keep API-key bulk manual-model
editing usable, and keep adapter discovery available after candidate
expiry.
- Skip taskless native timer work before run or credential admission.
Preserve current master monitor and deadline scheduling.

## Verification

- The registered local saved-Google-subscription journey passed all 16
matchers in 196,520 ms using eval source `12aba` against actual serving
source `4a37ee7c`. Both turns stayed in one conversation, produced exact
downloaded files of 61 and 119 bytes after reload, and used distinct
native primary processes. All observed owners retired, leases released,
and credential fences settled. Independent readback confirmed Done work,
no active owned runs, a connected saved account, and the retained agent
paused for inspection.
- The measured runtime is production-equivalent to pre-restack
`65ef31dc` except its test-only transport fixture correction.
Current-master head `ff7c9249` preserves all 99 product paths: 98 exact
blobs and only eight inherited plugin-panel lines in `agents.ts`. Its
native binary was rebuilt for inherited Codex-only diagnostic changes;
the changed Rust behavior is outside Antigravity's ACPX execution path.
Shared native/UI changes have focused regression coverage. The recorded
live result is not represented as a measured replay of `ff7c9249`.
- Full workspace build and typecheck passed on `ff7c9249`. All 376
focused tests in 14 files passed, covering Google credential/retirement
controls and the inherited native-tool, redaction, route, and UI
recovery changes. Token gates, maps, and diff checks passed. The
required-close fixture was separately demonstrated red, then passed all
317 selected provider-wiring cases.
- [Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/40959387972)
is green: 55 successful checks, two skipped workflow checks, and an
optional neutral review check. Required `ci / verify` and `ci / e2e`
passed. Greptile and Paperclip Review both report 5/5 on exact
`ff7c9249`, with zero unresolved review threads.
- This qualifies the selected operator-authorized local
saved-subscription candidate. The original exploratory journey remains
historical evidence; the repeatable fixed-driver eval is in the next PR.

## Risks

- Google refresh recovery must preserve the exact grant, account, and
credential generation. An uncertain native owner or failed encrypted
handoff retains the private home and blocks cleanup.
- A successful local saved-subscription candidate does not qualify fresh
OAuth, API keys, Cloud, Linux, other models, or general release.
- Timer filtering changes only taskless native Runner admissions.
Existing task-bound monitors and legacy adapters retain their owners.

## Model Used

OpenAI Codex (GPT-6). The host did not expose the exact model revision
or context limit. The model used tools for code edits, tests, and
review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 16:51:50 -05:00
DottaandPaperclip f1911a4c53 perf(skills): move repository packages out of Postgres (#18082) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/205/changes/paperclip/205
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents receive selected company skills before each turn.
> - Imported skills can need files from the whole source repository.
> - The current path stores those files in Postgres and reads large JSON
payloads during preparation.
> - This pull request stores verified packages in the configured file or
object store and keeps preparation queries small.
> - It also converts old rows automatically and restores missing files
from the exact saved commit.
> - The result reduces database traffic while preserving skill versions
and repository dependencies.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Company skill import, version storage, and preparation before agent
turns.

**Current behavior**

Repository snapshot bodies live in JSONB. Skill preparation can read
those bodies again even when the files already exist on disk. Historical
packages and native assets also need a bounded cleanup path.

**Proposed behavior**

Store package archives and manifests through the existing storage
service. Keep version identities, receipts, hashes, and recovery
metadata in Postgres. Convert legacy rows after the stored package
passes read-back verification. A missing package can use legacy bytes or
its exact Git commit under current repository authorization. No separate
data-migration command is required.

**Reason and benefit**

The warm path sends no repository bodies through SQL and reads no object
bytes. Cleanup runs through the existing deadline coordinator. Current
versions, active runs, explicit pins, and authored content retain their
protection.

**Breaking changes**

The normal application schema upgrade is required. Single-version and
skill-file responses keep their existing shape. Version lists,
current-version summaries, and test-run version summaries now contain
file names and kinds without file bodies. Diff and restore fetch
selected versions through the existing single-version endpoint. Older
server builds must not write after conversion unless they support the
new package format. The storage override and rollback limits are
documented in `doc/skill-package-storage.md`.

**Additional context**

Related work: #15731 preserves repository dependencies; #17923 changes
version-cache scheduling; #18080 changes delivery to remote Codex homes.
This change owns package persistence, acquisition, and expiry. It does
not replace those delivery or scheduler changes.

## What Changed

- Add company-scoped archive receipts, exact-commit recovery recipes,
migration deadlines, and a durable object-deletion queue.
- Add bounded archive upload, read-back verification, extraction,
concurrent request coalescing, and verified local caches.
- Convert old snapshots and versions automatically. Keep legacy bytes
when verification fails.
- Recover missing files through current source authorization and the
original commit. Validate repository identity, file hashes, modes, and
the aggregate hash.
- Expire unused recoverable Git packages and native assets after 30
days. Preserve active pins and the durable copy of authored or unproven
content.
- Keep warm preparation queries metadata-only. Bound file digest caches
and avoid repeated body hashing when file identities are unchanged.
- Extend the existing storage contract with cancellation. Register
package work with the existing coordinator. Optional conversion retries
retain the original bytes and may wait for ingress without blocking
sleep.
- Add regression tests, operating guidance, a deep trace, benchmark
receipts, and an optional maintenance command.

## Verification

### Final master integration and review repair — October 29, 2026

Current head: `a6065424249e763dfeac37161f1b52a9f25bd9b7`. Commit
`f8bbdd3dbd` resolves the conflicts with master `76a796e964`; the final
two commits repair the Apex deletion findings below.

- Keep both package-maintenance and version-retention notifications in
their write transactions.
- Keep the shared run-pin predicate and master's controller-lease
deadline metadata. Package expiry reads the updated protected-version
result.
- Preserve every master migration entry. Generate migration **0415
(`0415_zippy_zemo.sql`)** after master's 0414. Its skill SQL is
byte-for-byte unchanged from the previously tested idempotent migration.
Retain only the five newest schema snapshots. Migration numbering and
safety checks pass.
- Repair failed file/folder deletion: preserve a complete temporary
subtree until transaction commit, restore on upload or SQL failure, and
restore callback failures before releasing the mutation lock. Both
PostgreSQL regressions reproduced missing files before the fix and now
pass. They cover binary data, executable permissions, empty folders,
unlisted files, unchanged database/version rows, successful retry after
upload-claim expiry, and historical reads.
- Reconcile lost commit replies under the same advisory and row locks.
Check the saved version and source identity before restoring. Keep
committed deletions intact and preserve backups when the database
outcome is unknown or a newer edit exists. Four real-PostgreSQL outcome
tests pass; the prior code failed three of them.
- Full workspace typecheck and production build pass on the final
commit. All **204** focused PostgreSQL package/cache/retention/authoring
and filesystem-boundary tests pass, with none skipped.
- Apex review **18 passes 5/5**, with zero new comments on the exact
final head. All **34** review conversations are resolved.
- [Final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/40914116508)
completes successfully on attempt 2. The head has **55 successful
checks**, two neutral reports, two skipped checks, and a successful Snyk
status. No checks are pending or failing. All eight browser shards pass.
- Runner Vitest 2/2 timed out once in the unchanged ACPX
controller-replacement fixture. The same job passed on the preceding
commit. One unchanged CI retry passes all five ACPX recovery cases,
**2,253** Runner package cases and **126** native server integration
cases; their standard one and five skips remain. The initial timeout's
root cause is not established. Local diagnosis also hit the fixture's
eight-second session.open timeout with an authenticated Runner; no local
Runner pass is claimed.
- Both full local `pnpm test:run` attempts were stopped when Apex
identified deletion defects and source repair became necessary; neither
is claimed as a passing full local run. The final **204** focused tests,
full workspace typecheck, and production build pass. Full-suite
acceptance uses the final-head CI jobs. The superseded merge-head CI
included runners terminated by shutdown signals; each source repair
started a new full run.

- Auto-merge is enabled for squash merge. The final head is
conflict-free and all checks pass. GitHub's active master ruleset still
requires a code-owner approval for the dependency files. The existing
review requests are with `devinfoley`, `nickyleach`, and `forgottendev`;
no policy bypass was used.
- The earlier hosted stress and full native receipts below remain tied
to `a2e597d800`; they are not relabeled as a deployment of this final
integration. The earlier local PostgreSQL startup failure remains
recorded below; PostgreSQL ran successfully for this integration
verification.

### Full native skill execution — October 29, 2026

The full native-agent acceptance gap is closed on candidate
`a2e597d800eb989d565780e7aa947d5368676444`. Two qualifying native turns
used OpenAI API / `gpt-5.4-mini` in real Daytona sandboxes, on the same
disposable hosted TEST tenant as the stress test. A follow-up on the
same task consumed a newly saved skill version.

The task text asked for the assigned skill workflow. It did not contain
the proof nonce or expected receipt. The assigned authored skill
contained a Python verifier and a random package-local nonce. The agent
read the actual installed skill files and executed that verifier against
the imported, automatically converted 1,360-file repository package. It
checked the repository file count and reference/binary bytes, and
directly executed the repository's executable probe. Independent
verification matched the emitted JSON to the nonce and expected byte
hashes. The agent persisted the same receipt as its task response;
native finalization committed, workspace finalization succeeded, and
task state became Done.

| Qualifying turn | End-to-end run time | Recorded skills.prepare |
Repository files | Binary hex | Shell receipt SHA-256 |
|---|---:|---:|---:|---|---|
| Initial task | 47.881 s | 1822 ms | 1360 | `00ff07800a` |
`a1deb30dc095f2306c7aa9cc5ec396da12c68961cf5b809f4d7c20f7fc68493b` |
| Corrected follow-up, new nonce/version | 37.974 s | 377 ms | 1360 |
`00ff07800a` |
`dbcdc33fa1376b4edc136feae1cdf770560a73292467e93619efdda20e4282a7` |

Both successful shell receipts reported `probe: STRESS18082` and
`assetSha256:
e8ea7a2f15f7bb3b332a46adbef848465884c8996595aa234748befd544d1f3a`. Their
proof nonces, selected version IDs and receipt hashes differed, and both
ran on the same task. These are observed timings, not latency SLOs; they
include real provider work. The service stress-test timings remain a
separate measurement boundary.

Earlier failures remain in the evidence:

- The original five-minute Daytona allocation timeout never reached
model execution. After normal owned-stack restore, its durable cleanup
record was `expired` / `success`. The cause of that original allocation
timeout remains unknown; a later success establishes recovery only.
- Initial continuation setup used unsupported generic agent-update
fields and returned HTTP 409 before dispatch. Setup was corrected to use
the supported skill-assignment and versioned instruction commands.
- The intervening completed native turn read repository bytes and ran
the probe, but omitted the verifier JSON under the older test agent's
overly-specific instruction file. Its strict receipt validator failed;
it is not counted as a qualifying turn. Correcting the test agent's
instruction file and sending an explicit task-scoped wake produced the
qualifying follow-up above.
- The pinned application's immutable rate card omits `gpt-5.4-mini`, so
completed usage triggered the normal block-on-unpriced budget policy.
Supported append-only accounting corrections estimated the three
completed runs using the [official OpenAI model rate
card](https://developers.openai.com/api/docs/models/gpt-5.4-mini): $0.75
input, $0.075 cached input, $4.50 output per million tokens, assuming
standard processing. Estimates totaled 12.482655 cents. Original usage
and missing-price records are preserved; the $1 hard stop and
unpriced-usage guard stayed active.
- Model shell attempts included a missing `xxd` command and incorrect
guessed file paths. Those failed calls and subsequent successful actual
verifier calls remain visible in complete event streams. No
zero-tool-error claim is made.

No product source changed during this continuation. The previously
measured contention, cold-start and memory limits remain. The later
merge resolution is covered by the separate integration checks above;
these hosted receipts qualify their recorded candidate.


### Hosted stress test — October 29, 2026

**Hosted acceptance:** full native-agent execution and a changed-skill
follow-up pass on the recorded hosted candidate. The final integration
section above records the subsequent conflict resolution and its
separate checks.

Tested baseline `929e840595c8933ca485ac7a2e99413e1e11ca64` and this
exact candidate sequentially on one disposable TEST tenant, with
external Neon PostgreSQL and actual S3. The fixture contained **1,360
files / 22.4 MB / ten skills sharing one repository**. Measurements
cover deployed skill-preparation and native-context services in an
adjacent diagnostic process, not a complete agent turn. The container
had **24 vCPU / 24 GB**.

| Warm workload | Median before → after | p95 before → after | DB
returned value bytes/preparation before → after |
|---|---:|---:|---:|
| One skill, serial | 1,060 → 125 ms | 1,303 → 155 ms | 22,610,389 →
1,771 |
| One skill, four concurrent | 4,415 → 297 ms | 5,143 → 328 ms |
22,610,389 → 1,771 |
| Ten skills, serial | 5,723 → 535 ms | 6,057 → 600 ms | 224,739,030 →
10,671 |
| Ten skills, four concurrent | 24,410 → 1,986 ms | 24,833 → 2,221 ms |
224,739,030 → 10,671 |

- Candidate warm preparation performed **zero S3 GETs**. Ordinary-pool
queries fell from 16 to 9 for one skill and 61 to 9 for ten, excluding
dedicated lock connections. DB numbers measure returned values before
Drizzle decoding, not wire bytes. Baseline/candidate sample counts were
30/100, 32/100, 5/50 and 8/48 respectively; the small baseline ten-skill
samples are comparative evidence, not SLO estimates.
- **Application-UID soak:** 800 ten-skill preparations over 10.3
minutes, four workers paced by 2.5 seconds; **zero failures**, median
**559 ms**, p95 **775 ms**, p99 **910 ms**, zero warm S3 reads. All 726
accompanying health/company-list requests returned HTTP 200. Their
end-to-end p95 was 569/502 ms, including network latency. The probe
overlaps most of the soak. Soak DB-response capture was limited to its
first 60 seconds.
- **Important remaining costs:** an unpaced 16-way ten-skill burst had
p95 **8.32 seconds** because company-level preparation serializes. Cold
preparation took approximately **1.1–2.5 seconds** for one skill and
**7.4–10.0 seconds** for ten. Ten-skill diagnostic RSS remained about
**1.17 GiB** (baseline 1.18 GiB), and the soak peaked at **1.35 GiB**. A
concurrent whole-container sample was 3.45 GB. This does not establish
suitability for a small-memory host or eliminate per-entrypoint native
asset duplication.
- **Upgrade without a manual data migration passed:** legacy snapshot
JSON shrank from 22,465,691 bytes to `[]`, with a stored package and
zero pending conversions.
- **Recovery passed:** sixteen concurrent cold requests coalesced into
two S3 GETs; full bytes/binary/executable modes matched; same-size
corruption repaired; an injected storage failure preserved warm access
and failed cold access safely. A disabled public-Git source denied
recovery, and after the actual five-minute backoff its exact SHA
reconstructed all 229 files. Cross-company descriptor access was denied.
Private-Git credential revocation was not tested.
- **Retention passed:** reclaimed 27,754,496 allocated cache bytes,
deleted an unreferenced package row and both objects, preserved aged
authored history, and expired/reacquired reconstructible Git content. A
version pinned by the failed native run was correctly retained. Only
owned fixture timestamps were aged.
- **Browser UI passed:** authored skill creation, saved edit surviving
reload, version history, and restoration of earlier content.
- **Initial native attempt (superseded by the qualifying turns above):**
one real run failed at Daytona allocation after five minutes
(`setup_failed`), before any model execution. Provider cleanup also
reported failure. That attempt does not count as a passing native turn.
Its receipt and all evidence remain preserved; its cleanup later
recorded success after normal restoration.
- Initial A/B commands ran as SSH root. Diagnostic-owned lock/cache
permissions caused one setup HTTP 500 and the first UID-1000 soak to
fail; exact fixture paths were repaired. The final soak, retention and
recovery ran as application UID 1000. A fresh UID-1000 one-skill batch
also passed (30 samples, median 116 ms, p95 321 ms, 1,771 DB value
bytes/preparation). Earlier harness-export, SSH, npm provenance, backoff
and UI-wait failures remain recorded alongside their successful retries.
- Private evidence includes raw timings, assertions, HTTP probes,
screenshots and failure/cleanup receipts. Initial failed-campaign
cleanup suspended and retained the owned TEST tenant and released the
shared QA account lease at **15:22:41 UTC**. The tenant was subsequently
resumed under its existing ownership for the native execution
qualification above. Final cleanup completed at **15:53:09 UTC**: the
account lease is released and the tenant is suspended/retained under the
original October 30, 12:24:52 UTC retention deadline. Its separate stack
lease remains held; no active or pending-cleanup environment lease
remained. The original retention window is deliberately preserved even
after successful continuation.

- Earlier hosted-tested head:
`a2e597d800eb989d565780e7aa947d5368676444`, integrated with master
`929e840595`. The earlier merge conflicts were resolved at this head.
[Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/40884087629)
passed; all checks are complete with 55 successes, two neutral reports,
two skips, and no failures. [Apex review
15](https://github.com/paperclipai/paperclip/pull/18082#issuecomment-6480257258)
passed 5/5 with zero findings on this exact head. All review
conversations are resolved.
- Ten real-filesystem boundary cases and server typecheck pass locally
for the security fix. Coverage includes cache roots, descendants,
individual files, symlink aliases, recursive ancestors, a future
concurrent cache fill, and allowed sibling directories. [Current-head
private-project integration
tests](https://github.com/paperclipai/paperclip/actions/runs/40884087629/job/123483306405)
passed all 29 cases, including authorized and unauthorized imports of a
cached private package. Full typecheck, build, server suites, all eight
browser shards, and Runner checks passed in current-head CI. Local
database tests could not run because PostgreSQL could not initialize;
this limitation is recorded below.
- The earlier generated migration preserved the newest five snapshots.
The final integration renumbers its unchanged SQL to 0415. Migration
safety/numbering checks pass. Its SQL is byte-for-byte identical to the
previously verified idempotent skill migration; repeated application on
this latest schema was blocked by the same local database startup
failure.
- The broader verification before the final migration-only rebase passed
1,025 package, lifecycle, remote-launch, native-session, and
asset-retention tests, 52 focused UI tests, full workspace
typecheck/build, token gates, and map checks. The linked complete CI run
passed before the latest focused fixes.
- A disposable MinIO instance passed multipart upload, verified
read-back, and cold restoration of an 18 MB archive.
- Twenty warm samples with a synthetic 1,349-file, 22.8 MB repository
retained five metadata queries and zero object reads. Median preparation
was 256 ms for one selected skill and 691 ms for ten, with 5 ms emulated
network delay each way. These are local measurements, not production
latency guarantees.
- No production instance was changed. Earlier verification failures and
their disposition are retained below.

<details>
<summary>Earlier failures, review findings, fixes, and rerun
evidence</summary>

- The first CI pass found a stale inventory-span expectation and a
missing repository-read span. Both are fixed; all three observability
tests pass. One browser test copied page text into its YAML parser after
reload; its first unchanged-head retry passed, but the race recurred
after rebasing. The test now waits for the renamed URL and completed
reload before copying the editor, retaining its original assertions. The
earlier full local suite did not complete cleanly: it reported setup
timeouts and assertions, while isolated relevant reruns passed. The
complete current-head CI run closes that broad verification gap. No
production instance was changed.

- The first Apex review scored 2/5 and identified six issues. The fixes
protect authored history even when it shares an in-flight Git upload,
recheck upload expiry during orphan deletion, bound failed-upload
cleanup, retain per-version kind/encoding metadata, add sanitized
failure diagnostics, and correct the migration guide. The final focused
run passes all 140 tests; the preceding eight-file run passed all 149
tests, including local storage and S3 diagnostics. Server typecheck and
migration safety/numbering checks pass after the fixes. The second Apex
review scored 1/5 and found unbounded maintenance storage calls and
mismatched package/Git batch limits. Those are fixed with a 30-second
cancellable cleanup deadline, shared 100 MiB/10,000-file scanner and
storage limits, and bounded Git read batches. The proposed manual
lockfile edit was rejected with evidence of the enforced CI-owned
lockfile policy. All 19 storage/recovery unit tests and server
typechecking pass. The new database-backed cleanup/conversion tests were
locally skipped because the host exhausted its PostgreSQL semaphore
capacity (confirmed directly by initdb). That gap is closed by
current-head CI: [package-store
tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109300889)
pass all 18 cases and [package-worker
tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109301157)
pass all 7, with no skips in either suite. The corrected YAML browser
shard passes. A separate planning-fixture test failed its cleanup DELETE
with an activity-log foreign key violation after its behavior assertions
passed; its one unchanged-head retry passed. The third Apex review
scored 1/5 and identified a one-slot pool deadlock, projected path
limits, cleanup holding the preparation lock during archive reads, and a
publication/expiry race. These are fixed with transaction-local reads,
prefix-aware bounds, isolated verification, and shared package row locks
with a publication grace. The separate Paperclip review also found stale
historical inspection, eager history downloads, and conversion retries
blocking sleep. Those are fixed. Local PostgreSQL capacity recovered:
the new database regressions ran, including the one-slot fork, paused
storage during a warm turn, and a locked publication during expiry. The
expanded run passed 144/145 tests; its remaining assertion still
expected file bodies from the version list and was updated to fetch the
selected version. The corrected company-skill suite passes all 81 tests,
and the final package/projection run passes all 27. All 52 focused UI
tests pass after updating their query-provider fixtures; the first UI
run exposed a fixture remount, and the follow-up exposed an editing
syntax error, both corrected. Full workspace typecheck and build, token
gates, and map checks pass. The sandbox initially blocked the build
tool’s local IPC socket; the authorized build rerun passed. Master then
claimed migration 0401. The branch is rebased on f8226b0515, the
unchanged SQL is regenerated as 0402, and repeated application passed
twice on the combined schema. Post-rebase workspace typecheck passes.
All 1,025 package, lifecycle, remote-launch, native-session, and
asset-retention tests pass on the combined head. All CI checks passed
for head 9d80a3b52d. The fourth Apex review scored 3/5 and found a
save/edit lock-order cycle, a retry retaining an uncommitted source ID
in its recovery recipe, and a summary read racing conversion. Commit
b7d9237c6a fixes all three. The new concurrency and retry tests pass
within all 61 package/source tests, and server typecheck passes. Master
then claimed migration 0402. The branch is now rebased on 65fa1cd617 and
the unchanged SQL is regenerated as 0403. All 68 package, source, and
worker regressions, server typecheck, migration safety/numbering, and
repeated migration application pass on current head 62553671a0.
Subsequent verification and review outcomes are recorded below.


- The fifth Apex review scored 4/5 and found cached version reads
unnecessarily depending on object storage and conversion reads missing
the coordinator cancellation signal. Commit 1eb90d80be fixes both:
cached reads verify bytes without the preparation lock; manifest and
archive reads carry cancellation through conversion. The first focused
run passed 70/71 tests, including all three new cases, but the existing
diagnostic test observed no warning. Its fixture now sets an explicitly
expired migration deadline; the final worker suite passes all nine
cases. Server typecheck passes.


- The sixth Apex review scored 4/5 with one finding: concurrent cached
inventories were unbounded. Commit 4b7589e14c gives cached inventories
and previews a two-reader/32-waiter limit independent of cold storage.
The first expanded pressure test observed a third reader before the
preview path had the same bound; both paths now share the local limit.
All 73 focused cases pass, followed by all 29 package cases after the
final overload adjustment; server typecheck passes. The separate
Paperclip review found no must-fix or should-fix issues. Its optional
suggestion to persist durability verification was considered: cleanup
intentionally verifies current external bytes before discarding the last
local authored copy, and the missing-object regression requires that
check. Shared-package eviction happens on cold fills with at most two
attempts and a ten-second budget.


- The seventh Apex review reached 5/5 with one remaining P2:
timeout-bearing manifest reads bypassed request coalescing. The security
review also found repeated archive acquisition after a cold
administrative read. Commit 79a80a39d2 shares manifest loads with
per-caller cancellation and cancels the underlying work when the last
caller leaves; coalesces identical cold version/file requests; and
publishes verified restored packages into the shared cache. Publication
uses a short nonblocking lock, while eviction verification runs outside
it. The first expanded run passed 73/74 tests and exposed a missing
originating DB for cache publication inside read-only inspection;
transaction readers now pass that owner explicitly, keeping metadata on
the transaction. The corrected run passed 75/75. A final eviction test
initially expected the shared package to remain after only its separate
runtime projection was reused; it now explicitly accesses the shared
package during verification. All 32 final package tests pass, including
independent caller cancellation, one acquisition across
concurrent/repeated reads, and warm preparation during stalled eviction
verification. Server typecheck passes.


- The eighth Apex review scored 4/5 with one finding: decoded cold
inventories outlived the storage semaphore while waiting for cleanup.
Commit 9d7921d1ec adds a separate two-response/32-waiter lifetime bound
held through cleanup, preserving the independent warm-read queue and
forwarding cancellation through eviction verification. The new
regression pauses cleanup after two responses decode, verifies a third
archive stays queued, and completes warm preparation before releasing
cleanup. All 77 package/source/worker cases and server typecheck pass.
On the preceding head, serialized server shard 6/9 was interrupted with
“The operation was canceled” after 57 seconds of tests; the log contains
no failing assertion. Fresh latest-head CI reruns that shard along with
the other gates.

- The ninth Apex review scored 4/5 with one P1: cold responses could own
both permits while waiting on a one-slot ordinary database pool held by
a queued transaction reader. Commit a94f0e409f reads caller-scoped
metadata before admission and isolates all acquisition/cleanup SQL on a
private connection. The new real-PostgreSQL regression reproduces the
deadlock against the previous head and passes with the fix; all 78
focused cases and server typecheck pass. The previous CI run completed
with 54 successful checks, including both formerly interrupted/stuck
serialized shards.

- The tenth Apex review scored 4/5 with one P1: preloading legacy bodies
before admission could retain many full repositories in queued requests.
Commit 276e353a37 removes that preload and also bounds legacy manifest
derivation/fallback, using private connections inside the I/O slot. All
80 focused tests pass; the final synchronized package suite passes all
36 tests and server typecheck passes.

- Apex review 11 returned 5/5 with zero comments on 276e353a37. While it
ran, master 8bc051245f added migration 0403, creating a metadata
conflict. Merge commit e1783764d9 integrates master and generates
0404_clumsy_leo.sql; its skill SQL is byte-for-byte identical to the
previously verified migration. The combined schema passes migration
safety/numbering, repeated SQL application twice, all 80 focused
PostgreSQL tests, full workspace typecheck, and full build. The
preceding a94f0e409f CI run was superseded and cancelled while its final
runner job was active; no old-head success is claimed for that run.

- Final acceptance: Apex review 12 returned 5/5 with zero comments on
e1783764d9. All 28 review threads are resolved. The final head has 55
successful checks, two neutral reports, two skipped checks, and no
pending or failing checks; the PR workflow completed successfully.
Superseded run 40789121902 was cancelled to free its workflow slot for
the final head; normal cancellation did not finish promptly, so force
cancellation completed it.

- October 29 conflict resolution (867c00673b): master 929e840595 added
installed-selection reuse and bundled relocation, routine cleanup
notifications, a hardened YAML copy helper, and migrations through 0413.
The merge retains those behaviors together with package read scoping and
the public-selection fast path; restricted selections still reauthorize
and the full row read rechecks scope. Generated migration 0414 retains
the existing SQL unchanged. Workspace typecheck/build and migration
checks pass. Local database verification is unavailable: initdb reports
semget / “could not create semaphores: No space left on device”; all 168
cases skipped, and repeated migration application failed at startup.
Docker returned HTTP 500 during container creation, so no Docker test
pass is claimed. Current-head CI must close this gap.

- Apex review 14 scored 3/5 with one P1: a local import could copy
cached private package files into a company-wide skill. Commit
a2e597d800 rejects cache trees and ancestors through a canonical
filesystem boundary, including aliases and ancestors checked before a
concurrent fill. Ten local filesystem tests and server typecheck pass.
The existing private-project integration case now rejects package-cache
imports for both authorized and unauthorized users. The preceding merge
commit’s complete CI run passed, and logs confirm all 168 locally
skipped cases executed successfully: package store 36, sources 35,
package worker 9, runtime selection 4, observability 3, authoring 81.

- Apex review 15 returned 5/5 with zero new findings on a2e597d800. The
separate Paperclip review found no must-fix or should-fix issues. Its
optional suggestions were considered: unexpected filesystem cleanup
failures remain visible, the durable marker cannot replace checking that
bytes still exist before deleting a last local authored copy, and
administrative history batching remains outside the pre-turn
optimization. The cache-path literal centralization suggestion is also
optional; the current producer and guard use the same directory, covered
by the new integration regression. All review conversations are
resolved.

- Earlier October 29 CI/Apex acceptance: a2e597d800 was mergeable at
that check, Apex review 15 passed 5/5 with zero findings, and all review
conversations were resolved. The hosted verification above records the
later acceptance limits and renewed conflicts. Workflow 40884087629
completed successfully, with 55 successful checks, two neutral reports,
two skips, and no pending or failed checks. The project-skill-sources
job log confirms all 29 cases ran and passed, closing the latest
integration gap. The preceding merge workflow also confirmed all 168
focused database cases ran and passed. No local PostgreSQL or Docker
pass is claimed.

</details>

## Risks

- First use after eviction can require storage or Git access. Revoked
grants, removed sources, and unreachable commits can prevent
reconstruction. Recovery never substitutes a newer commit.
- Historical packages without a proven recipe keep durable bytes.
Cleanup favors retention when a run pin cannot be interpreted safely.
- New uploads and conversion use bounded work and persisted retries.
Large initial conversions still add temporary storage and database load.
- Optional cleanup can wait until the next server wake. Supported
notifications assume one owning server process.
- Per-entrypoint native tree duplication and cold preparation cost
remain. The final ten-skill cold sample took 47.6 seconds on the shared
host.
- Generic authored edits can still hold an existing edit transaction
while a package uploads.
- Explicit file/folder deletion uses temporary disk space for rollback.
An uncertain commit with a changed skill or unavailable database retains
that backup and logs its location for recovery.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
execution, and tests. The exact served model ID and context window are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 12:19:37 -05:00
Nicky LeachandPaperclip 247de89eae fix(runner): preserve warm tasks during recovery (#18169) 
## Thinking Path

> - Paperclip manages agent work and the lifetime of its execution
resources.
> - A warm native task can keep one Runner alive between successful
turns.
> - Foreground completion selects whether that resource stays running or
stops.
> - Background recovery must use the same selected policy.
> - Recovery currently reads that policy only from workspace copy-back
records.
> - A task without such a record can be stopped immediately after
foreground retention.
> - This pull request saves the resource policy with the native runtime
and permits the retained task to refresh its control connection.

## Linked Issues or Issue Description

**What happened?**

Background native finalization can stop a successful warm Runner when
its run has no Core workspace-sync record. Foreground completion retains
that same task. The stopped Runner cannot receive the later drain and
suspend commands. Cold resume then correctly rejects its unproven
checkpoint.

Hosted diagnostics confirmed the recovery reconciler attempted
environment release immediately after foreground retention. The provider
connection returned HTTP 404 within the same second. A database-backed
regression reproduced the stop without a workspace-sync record.

A separate authorization gap also prevents the current retained task
from refreshing its connection endpoint after the run succeeds.

**Expected behavior**

Foreground completion and recovery must use the selected resource
policy. A successful warm task stays running until its idle shutdown can
suspend it. Failed or cancelled work still stops. The current retained
task can refresh its existing control route, but cannot prepare or
submit new execution.

**Steps to reproduce**

1. Select a warm native runtime on a reusable remote execution
environment.
2. Complete a turn without a Core workspace-sync record.
3. Let native finalization recovery settle the still-active run lease
before foreground retention finishes.
4. Before this change, recovery defaults to `stop_and_retain` and
invokes the provider stop path.
5. Separately, request the retained task's connection endpoint after
successful completion. Before this change, Core rejects that request.

**Paperclip version/commit**

The hosted failure used Core 31b927e301.
Regressions also reproduced on master at 6d59f5ef7.

**Deployment mode**

A remote native execution environment with reusable warm tasks. The
resource policy is generic and applies across harnesses and providers.

Related recovery coverage: Refs #18161. This change preserves the
suspended-checkpoint admission guard. Existing tasks without the newly
saved policy retain the previous conservative fallback; deployments must
drain or recover them explicitly.

## What Changed

- Save the selected native resource disposition in the existing
runtime-selection transaction. Add no startup database round trip or
network request.
- Read this small policy in background settlement and recovery when no
workspace-sync record exists. Preserve valid workspace-sync policies and
export-retry behavior.
- Keep failed and cancelled runs out of warm retention.
- Permit connection refresh for a successful, dispatched warm task on
its current retained reusable lease. Keep retirement, termination,
expiry, saved-run identity, and plugin capability checks.
- Add database-backed retention and selection regressions,
failure/cancellation coverage, and bounded projection checks. Document
the lifecycle and access contracts.

## Verification

- Before the source edits, the no-workspace-sync regression invoked the
driver with `stop_and_retain` instead of retaining the lease. Both
runtime-selection cases failed because the resource policy was absent.
- After the source edits, those regressions and both
run-state/runtime-selection files passed. The three-file run had 77
passes and two new fixture expectation failures. Those fixtures were
corrected. Final-head GitHub CI passed, including the server suites,
Runner checks, typecheck, build, and E2E gates. The local rerun was
deferred at the user's request.
- Commands: `node_modules/.bin/vitest run
server/src/__tests__/native-environment-settlement-work.test.ts
server/src/services/heartbeat/runtime-selection.test.ts
server/src/services/heartbeat/run-state.test.ts`.
- The connection authorization regression failed before its edit. All 34
tests in `server/src/__tests__/environment-task-runtime.test.ts` then
passed.
- `node scripts/check-maps.mjs` and `git diff --check` passed.
- No comprehensive local suite was run. Further local tests and VM
reproductions were stopped at the user's request to conserve battery and
mobile data. GitHub CI verifies the final head.
- Hosted verification of this patch requires deployment after merge.
Passing pre-patch retries did not establish repair of the original
failure.

- Standard Greptile and the subsequent requested Apex review both
returned **5/5** on `360bcd7e789d65950c27484af237faec02553666`, with no
inline findings.

## Risks

- This adds one server-owned field to the existing run profile JSON; no
migration is required. Invalid or absent policy keeps the previous
conservative fallback.
- Valid workspace-sync disposition remains authoritative. Pending export
and failed finalization still use stop-and-retain safety behavior.
- Retained connection access requires a successful run, matching saved
run identity, dispatched warm task, unexpired reusable lease, and no
retirement or termination. Superseded leases and new execution remain
denied.
- The cold-resume checkpoint proof is unchanged. This patch does not
make the already-stopped task's unsuspended checkpoint resumable.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, and code
execution. The host does not expose a more specific serving model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run targeted local checks; final verification passed in
GitHub CI (local rerun deferred at the user's request)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 09:19:49 -07:00