Commit Graph
2200 Commits
Author SHA1 Message Date
DottaandPaperclip eb8044bccc fix: separate workspace origin from replaced process authority
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 12:36:30 -05:00
DottaandPaperclip ce107d4710 fix: validate persisted warm origin run identity
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 12:27:56 -05:00
DottaandPaperclip cfa2b81539 fix(computers): allocate listeners outside guest ephemeral ports
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 12:27:30 -05:00
DottaandPaperclip d91b4fef7b fix(computers): keep shared desktop input outside agent ownership
Launch repaired IBus in a collected app.slice service so native owner retirement cannot kill the shared desktop daemon. Presence renewal refreshes credentials without repeating input readiness; explicit Connect still prepares the desktop.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 12:21:34 -05:00
DottaandPaperclip d4047f2fef fix(computers): reserve lifecycle traffic capacity during workspace setup
Route project setup and file operations through ordinary SSH admission, retaining reserved control capacity for owned-port inspection and lifecycle operations.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 12:19:06 -05:00
DottaandPaperclip d3fc9db89e fix: verify original workspace lineage after warm retirement
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 12:13:40 -05:00
DottaandPaperclip 43b818a39a test: synchronize remote upload concurrency assertions
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 12:11:49 -05:00
DottaandPaperclip 546dcb9daf Keep remote instructions usable in large persistent folders
Use bounded partial directory pages with a global metadata limit and explicit listing/export warnings. Always inspect the configured entry directly so it remains readable beyond the first page.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 12:10:52 -05:00
DottaandPaperclip fe619aa13a fix(computers): prune retired desktop viewers during reconciliation
Keep active viewer leases and retired runner process proof, while removing expired or disconnected viewer records from the host ledger.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 12:08:58 -05:00
DottaandPaperclip 6726f78238 fix(computers): serialize shared project checkout setup
Hold a confined per-project flock across clone, repository validation, checkout, and worktree creation so another agent cannot observe partially created Git state.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 12:08:58 -05:00
DottaandPaperclip 469da8b2c2 fix(boat): stage immutable context per bootstrap attempt
Bind generated instructions and skills to fresh attempt-owned snapshot paths, retaining old immutable snapshots without touching persistent user files or provider HOME. Record possible process dispatch only at the actual launch RPC boundary so preparation failures can retire their exact owner safely.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:56:53 -05:00
DottaandPaperclip 7f646066e8 Document provider cache computer-owner capability requirement
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:52:47 -05:00
DottaandPaperclip 3233b079cd test(boat): retain cache acceptance coverage in stacked PR
Keep the immutable cache suite and Boat secret configuration acceptance assertion in the combined feature PR while preserving the standalone core review size.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:50:19 -05:00
DottaandPaperclip 5b00bdb95e Reuse immutable provider packs across computer task sessions
Verify shared cache hits before archive preparation. Publish privately staged, verified packs without replacing shared bytes and fence retry uploads with exact ownership receipts.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:50:02 -05:00
DottaandPaperclip 94ddd82b9f fix(boat): settle runner authority before idle retirement
Reserve exact computer generations for graceful suspension, keep provider checkpoints on their persistent remote authority, and fence failed prelaunch attempts before retry. Recover previously retired quiescent Codex checkpoints only with exact ownership and unchanged-byte evidence.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:45:58 -05:00
DottaandPaperclip 8096727117 test(boat): keep remote acceptance extensions in stacked PR
Preserve the two existing additive runtime acceptance suites in the stacked feature PR while keeping browser synchronization fixes in the independently checked core branch.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:44:36 -05:00
DottaandPaperclip 00f92a6d05 fix(computers): reserve bounded graceful runner retirement
Fence admission and ordinary commands at warm expiry, reserve exact-generation process control for at most thirty seconds of shutdown, and always retire the owned slice after graceful close or timeout.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:44:06 -05:00
DottaandPaperclip b6f8c44f9f Reclaim abandoned computer home uploads from durable receipts
Persist exact upload ownership before staging, renew a five-minute inactivity lease per chunk, and reclaim verified expired uploads on seed or home admission. Preserve active uploads and unrelated directories, and fence delayed writes and aborts after replacement.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:31:56 -05:00
DottaandPaperclip cd1bfc1aa0 Reserve Boat SSH capacity for lifecycle and process control
Bound each physical Boat to eight SSH operations with six ordinary slots and four bulk transfers. Preserve command deadlines while queued and always release admission after transport completion or failure.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:23:47 -05:00
DottaandPaperclip db3a9aecf4 fix(boat): resolve prior workspace authority to recorded lease
Preserve historical PRP workspace coordinates and require the recorded lease run plus exact retired owner before rotating authority.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:16:40 -05:00
DottaandPaperclip 8242d89916 Allow bounded Boat connection probes before saving an agent
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:16:40 -05:00
DottaandPaperclip 26d404cbfe Move bootstrap and instruction acceptance coverage to stacked PR
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:08:49 -05:00
DottaandPaperclip ab5bb8552a fix(boat): bound provider-pack cold bootstrap separately
Allow thirty minutes for computer ACPX pack staging without extending turn, recovery, finalization or warm-idle deadlines.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:06:31 -05:00
DottaandPaperclip cebfa8b09f Bound initial computer home uploads and exclude runtime state
Stream initial personal files through bounded chunks into private staging and atomically publish without replacing an existing home. Keep the home-level runtime directory out of instruction enumeration and seeding while preserving nested user content.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 11:00:36 -05:00
DottaandPaperclip 421f7f0600 Resolve Boat input readiness in the desktop user context
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:59:29 -05:00
DottaandPaperclip d6363b41d4 fix(runtime): verify retired computer owners after warm expiry
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:53:10 -05:00
DottaandPaperclip ff26b3af9b Reconcile superseded Boat stop operations
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:50:20 -05:00
DottaandPaperclip 03962e7189 Expose exact read-only computer retirement proof
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:48:33 -05:00
DottaandPaperclip 3308a116d4 Restore Boat desktop input before computer access
Probe IBus without replacing a healthy daemon. Rebind a failed restored socket to the same user private runtime directory before viewer access or Cua MCP startup; headless execution stays independent.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:47:31 -05:00
DottaandPaperclip 6e4bd77b58 Merge upstream workspace preparation extraction into Boat runtime
Preserve Boat workspace and credential policies in prepareHeartbeatWorkspace. Move computer route acceptance coverage to the stacked UI PR without dropping coverage from the full feature tree.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:39:19 -05:00
DottaandPaperclip 357d122705 test(runtime): preserve policy fields in legacy workspace checkpoint
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:34:38 -05:00
DottaandPaperclip 46ec1f81de Reap retired command spools before resuming computer work
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:34:38 -05:00
DottaandPaperclip 07fee85eaf refactor(heartbeat): extract workspace preparation (#15807)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The heartbeat service prepares the workspace and dispatches each
agent run.
> - After the runtime extraction, `heartbeat.ts` still had 6,417 lines.
> - Workspace setup occupied one contiguous 857-line block inside
`executeRun`.
> - This pull request moves that block into one workspace preparation
module.
> - The executor keeps dispatch gates and terminal cleanup. Reviewers
can inspect workspace setup as one phase.

## Linked Issues or Issue Description

Refs #15803 and #15578. This continues the heartbeat refactor series.
The search found no open duplicate workspace preparation extraction.

**What existing behavior does this improve?**

It makes heartbeat workspace preparation easier to maintain. It does not
change the workspace policy or execution behavior.

**Current behavior**

`executeRun` contains workspace resolution, freshness decisions,
provisioning, persistence, issue binding, environment acquisition, and
realization inline.

**Proposed behavior**

`prepareHeartbeatWorkspace` owns that sequence in
`server/src/services/heartbeat/workspace-preparation.ts`. `heartbeat.ts`
calls it at the same point and uses its returned workspace, lease,
target, and metadata.

**Reason and benefit**

The main file falls from 6,417 to 5,597 lines. The workspace phase has a
typed boundary and direct regression tests.

**Breaking changes**

None. There are no schema, API, feature-flag, or execution-policy
changes.

## What Changed

- Extract the 857-line workspace preparation block without changing its
operations or their order.
- Preserve native recovery bindings, workspace persistence failure
cleanup, issue rebinding after realization, and the cloned restore
lease.
- Keep environment selection, dispatch admission, terminalization, and
final lease release in `heartbeat.ts`.
- Add 12 boundary tests with disposable PostgreSQL, real Git
repositories, and real local environment leases.
- Make the existing Stop and plan-only recovery tests await executor
completion instead of short polling deadlines. Assert exactly one
continuation for the specific plan-only source run.
- Keep server-only Node preload hooks out of the deterministic process
adapter used by Stop browser acceptance.
- Refresh workspace recovery browser acceptance for the current task
composer, and give its disposable test-drive an explicit 10-second
snapshot deadline. Production snapshot settings stay unchanged.
- Document the workspace preparation boundary in `doc/DEVELOPING.md`.

## Verification

- AST comparison: all 857 moved lines match the original block. The
remaining module body differs only at the extraction call.
- `pnpm build`: passed.
- `pnpm -r typecheck`: passed.
- Focused regression coverage: 651 distinct cases across eight suites
passed across targeted runs, including all 12 new boundary tests. The
complete 400-test recovery suite passed together on the final commit.
- Workspace bootstrap browser acceptance: both journeys passed (4.2
minutes), including preserved source edits, ignored-file exclusion,
bounded retries, and persistence across reload.
- Stop/queue browser acceptance: both process and native runner journeys
passed (2.2 minutes), covering queueing, Stop, subtree pause/cancel,
resume, and unrelated-task isolation.
- [Full repository
CI](https://github.com/paperclipai/paperclip/actions/runs/38062408589):
all 47 jobs passed, including the general and serialized test suites,
eight browser shards, build, typecheck, native runner checks, and canary
dry run.
- The local `pnpm test:run` was interrupted to prioritize the recovery
regression; the complete repository suite was verified in CI.
- Greptile: 5/5 on `c28e698b885cf5fd12f36f7b4052e7a2e6f879d7`, with no
open review threads.

## Risks

- This code writes durable workspace bindings and acquires environment
leases. A changed operation order could break recovery or cleanup after
partial setup. The extraction preserves the original statements and
exception boundaries.
- The new tests cover creation, reuse, immutable recovery bindings,
persistence failure, acquisition and realization failure, controller
ownership loss, isolation, execution policy, and lease snapshot
independence.
- Browser acceptance uses real application services with deterministic
local workers and a fake Codex protocol provider. It does not qualify
live cloud providers.

## Model Used

OpenAI GPT-6 through Codex, with code editing, shell execution, and test
tools. The exact serving model identifier and context-window size are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-10 10:30:17 -05:00
DottaandPaperclip a9882c9dee Keep owned stdin valid until command completion on Boat
Boat invalidates descriptors for unlinked backing files, causing Node startup to abort. Retain the0600 stdin spool until normal cleanup, and reap abandoned command spools only after exact-owner retirement stops its cgroup.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:28:14 -05:00
DottaandPaperclip af93abd90a Use isolated computer storage paths during restart reattachment
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:23:59 -05:00
DottaandPaperclip 5a16ef77c0 Isolate persistent computer runtime assets by agent session
Root runner binaries, harness installs, provider packs, and session state under the trusted persistent agent home and normalized session. Keep the shared project working directory unchanged and preserve existing sandbox paths.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:21:55 -05:00
DottaandPaperclip 4d2113e188 fix(runtime): retain safe private Codex installation diagnostics
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:17:35 -05:00
DottaandPaperclip 00d1d4ea61 fix(runtime): bind persistent computer homes into Claude runtime context
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:17:23 -05:00
DottaandPaperclip ec80163f2f fix(computers): delete large agent files without downloading
Hash remote files through confined descriptors in bounded chunks before editor mutations. Preserve the remote compare-and-swap removal check and read-size cap; cover a real 17 MiB file, stale deletion, and path confinement.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:16:15 -05:00
DottaandPaperclip 03542b0845 Authenticate Boat initial checkouts with run-scoped Git credentials
Pass the existing workspace credential provider through computer realization separately from durable metadata, and supply its URL-scoped helper only to the clone subprocess. Verify authenticated clone behavior and credential non-retention.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:16:15 -05:00
DottaandPaperclip 25c12848a2 fix(runtime): allow supervised identity reads within startup deadline
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:11:01 -05:00
DottaandPaperclip f6aade3bae Add persistent Boat computer environments and runner ownership
Introduce company-scoped computer attachments, bounded runner and viewer ownership, persistent files and worktrees, Boat ingress, and native and legacy runtime integration behind the experimental Boat flag. Keep desktop UI and extended acceptance cases in the stacked feature branch.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-10 10:07:51 -05:00
DottaandPaperclip 19e9facc5b perf(chat): wake delivery queues from commits and deadlines (#15802)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat connectors store incoming messages and outgoing publications in
durable queues.
> - The app checks these queues every second, even when they are empty.
> - Empty scans keep the database active and hinder instance sleep.
> - This pull request wakes three queue workers after transactions
settle and at known retry or recovery deadlines.
> - Empty queues stop scheduling checks. Existing rows retain their
ordering, retry, and recovery rules.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Idle database activity from incoming chat deliveries, outgoing
publications, and Slack upload receipt recovery.

**Subsystem affected**

`server/` — chat delivery and background work scheduling.

**Current behavior**

The app's one-second chat timer scans the three queues together with
provider and action maintenance. Empty queues continue to query
PostgreSQL.

**Proposed behavior**

Register three independent workers with the existing delivery
coordinator. Producers announce work inside their database transactions.
Commit wakes the corresponding worker. Startup restores outstanding
work; persisted retries and stale claims supply the next deadline. An
empty queue has no recurring scan.

**Reason and benefit**

Remove avoidable queue reads without requiring a new table, an external
scheduler, or a persistent PostgreSQL listener.

**Breaking changes**

No schema or public API change. Work notifications are scoped to the
app's shared database owner. Direct SQL and independent database roots
do not send these notifications; startup still scans durable work.

**Additional context**

Refs #15625, #15750, and #15757 for the preceding sleep work. Related:
#15153 reduces chat query allocation while retaining polling. This PR
changes when the three queue workers run; it does not add query caching.

## What Changed

- Register independent incoming delivery, publication, and Slack receipt
workers on the shared scheduler.
- Add transaction work signals at queue writes, endpoint state changes,
and action transitions that unblock publications. Keep provider I/O
outside the added transactions.
- Restore deadlines from durable retry times, abandoned claims, and
publication lease expiry. Share publication and receipt eligibility
rules between dispatch and deadline queries. Respect incoming
conversation order and lifecycle dependencies.
- Exclude publications blocked by unresolved Slack tool sends. Wake them
when the tool receipt changes.
- Wake publication dispatch when an endpoint send finishes so free slots
refill. Preserve per-endpoint concurrency, conversation order, and
conservative handling of uncertain sends.
- Wake receipt recovery when a stale publication becomes
`delivery_unknown`, and wake queues when a provider connection recovers.
- Commit interaction publications with their callback tokens so dispatch
cannot see an incomplete card.
- Keep provider runtimes, other action outboxes, milestone and
wake-notice projection, Teams protocol maintenance, GitHub recovery, and
Slack session synchronization on their existing maintenance cadence.
This PR does not make the whole chat subsystem idle.
- Update the instance sleep inventory with this partial migration and
its remaining work.

## Verification

- All 1,225 PostgreSQL chat integration cases passed across the three
hosted chat shards. These cover retry and lease deadlines, blocked
followers, unresolved Slack sends, provider recovery, commit visibility,
and endpoint slot refill.
- 32 local app-wiring, queue-worker, and delivery-coordinator tests
passed after updating the wiring assertion for event-driven dispatch.
Fake timers cover an hour of idle time without queue scans.
- Related publication, interaction, receipt, and wake-notice tests
passed locally. The 13 interaction integration cases include atomic
rollback when callback-token storage fails.
- Local `pnpm typecheck`, `pnpm build`, and final server typecheck
passed. All hosted checks passed on the final commit, including build,
typecheck, test shards, release registry checks, and the canary dry run
(54 successful checks; two Storybook checks skipped for this backend
change).
- The last local PostgreSQL rerun was skipped because the embedded
database startup probe failed after five attempts. The hosted PostgreSQL
cases ran successfully; their skips only partition cases across shards.
- `git diff --check` passed. Greptile is 5/5 on the final commit with no
unresolved findings.

## Risks

- Every producer that makes queued work eligible must send a transaction
work signal. The tests exercise commit visibility, concurrent wakes,
worker failures, and restart recovery. Future direct SQL writers must
participate in this contract.
- The existing database work-signal layer reconciles uncertain
transaction outcomes. This PR adds no work-signal table or schema
migration.
- Provider and action maintenance still polls. Further changes are
required before an idle chat-enabled instance can sleep.
- Tests use real PostgreSQL with simulated provider responses; they do
not send messages through live provider accounts.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, and code
execution. The exact serving model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-10 09:13:44 -05:00
DottaandPaperclip 1fb3f9746e refactor(heartbeat): extract prepared runtime execution (#15803)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Heartbeat prepares, dispatches, and completes each agent run.
> - After the runtime-selection extraction, the main service still had
7,295 lines.
> - Provider dispatch and workspace finalization formed one large block
with shared cleanup rules.
> - This pull request moves that block into a typed runtime-execution
module.
> - The service now has 6,417 lines. Readers can follow admission and
outer cleanup without reading the provider execution details.

## Linked Issues or Issue Description

Follow-up to #15746. I searched open heartbeat PRs and issues. I found
no competing runtime-execution extraction.

**What existing behavior does this improve?**

The organization and test coverage of native and legacy heartbeat
execution.

**Current behavior**

The heartbeat service contains a 912-line block for provider dispatch,
usage capture, workspace finalization, and instruction cleanup.

**Proposed behavior**

Move that block to `server/src/services/heartbeat/runtime-execution.ts`.
Keep the atomic dispatch gate, admission, terminal run status, and outer
lease cleanup in the service. Preserve run behavior.

**Reason and benefit**

This gives runtime execution a named boundary and direct tests. It
removes 878 lines from the main service in one bounded change.

**Breaking changes**

None intended. There are no schema, API, or policy changes.

## What Changed

- Add `executeHeartbeatRuntime` for prepared native and legacy
execution, usage capture, workspace finalization, and
gateway/instruction cleanup.
- Return an explicit dispatch result. Update outer ownership state
through immediate callbacks, including before exceptions. Read
instruction state through live getters.
- Move the two internal recovery error classes with their existing
behavior. The caller imports the same classes for `instanceof` checks.
- Add 15 database-backed execution tests. Keep the existing heartbeat
regression suites.
- Repair the native Stop acceptance fixture: installed Codex now takes
precedence over `PATH`. A test-only loader pins the fake provider. An
isolated home and dummy key keep readiness offline. Add four loader
tests.
- Document the module boundary and fixture setup.

## Verification

- Passed: eight focused heartbeat suites (483 distinct tests, including
15 new execution tests). The boundary suite passed again after adding
the failed-restore retention case from review.
- Passed: `node --test
tests/e2e/fixtures/composer-stop-codex-loader.test.mjs` (four tests).
- Passed: `pnpm -r typecheck` and `pnpm build`.
- Passed: AST comparison of all 912 moved lines after the explicit
boundary substitutions. SQL template text, recovery class bodies, public
exports, and the rest of the executor match the baseline.
- Passed: browser Stop journeys for legacy process and native Paperclip
Runner (two tests). The native fixture log proves provider `turn/start`
before Stop and `turn/interrupt` after Stop.
- Passed: all three ACP continuation browser journeys (same-session
queued input, no replay of an interrupted write, and a new direction
after composer Stop). Five relevant browser journeys passed in total.
- Passed: the full CI test groups, including all server, chat,
workspace, serialized-server, runner, and browser shards.
- Local full-suite limitation: `pnpm test:run` first hit the macOS
shared-memory segment limit while starting embedded PostgreSQL. After
releasing only this task’s unused acceptance-server segments, the retry
progressed without reported failures. I stopped that duplicate run after
the full CI test groups passed.

- Passed: all 56 PR checks/statuses on
`8694c116e8d43ed8ac11bc4c58623ea3af34a675`. Greptile is 5/5 with no
unresolved review threads.

## Risks

- Cleanup depends on state written before an error. The module reports
dispatch, retention, and recovery ownership immediately. Tests cover
early returns and exceptional cleanup.
- Instruction callbacks can replace saved metadata during execution.
Live getters preserve those changes.
- Native acceptance uses a real server, database, browser, and Rust
runner with a deterministic provider. It does not verify a live provider
service.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and repository/tool access.
The exact serving model ID and context-window limit were not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-10 09:10:53 -05:00
DottaandPaperclip 2a2071340d refactor(server): extract heartbeat runtime selection (#15746)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Heartbeat execution prepares a runtime, dispatches work, and records
the result.
> - Earlier refactors separated workspace services, scheduling, and
completion.
> - The executor still builds native inputs and records runtime
selection inline.
> - This preparation must preserve admitted inputs and serialize with
Stop.
> - This pull request moves that phase into a typed runtime-selection
module.
> - Reviewers can inspect runtime preparation separately from dispatch
and cleanup.

## Linked Issues or Issue Description

Refs: #15732.

**What existing behavior does this improve?**

Maintainability of heartbeat runtime preparation and selection.

**Subsystem affected**

server/ — orchestration services.

**Current behavior**

The main heartbeat service has 8,128 lines. `executeRun` builds or
restores native inputs, selects sessions and sandbox lifecycle, and
records native or legacy selection inline.

**Proposed behavior**

Move this phase into
`server/src/services/heartbeat/runtime-selection.ts`. The executor calls
`selectHeartbeatRuntime` at the existing boundary and receives an
explicit selection result. The main file becomes 7,275 lines, a
reduction of 853 lines.

**Reason and benefit**

This gives runtime selection its own typed boundary and direct tests. It
preserves existing decisions, database transactions, and cleanup
ordering.

**Breaking changes**

None intended. All public heartbeat exports remain available.

## What Changed

- Move native input assembly, completion contract selection, checkpoint
lookup, runtime persistence, and native workspace-sync preparation into
`selectHeartbeatRuntime`.
- Preserve the row lock and controller lease check that serialize native
selection with cancellation.
- Report lifecycle and ownership changes to the executor immediately,
including when a later preparation step throws.
- Move sandbox lifecycle and model fallback helpers with compatible
public re-exports.
- Add 12 database-backed boundary tests. Update the existing
session-recovery source test and developer guide.

## Verification

- Structural AST comparison against current `master` confirms the
784-line selection block, moved helpers, and remaining service bodies
are unchanged after normalizing boundary callbacks. All 140 public
exports are preserved.
- On final head `2eddaa485f`, all 12 new database-backed tests and 83
existing lifecycle, selection, fallback-warning, and session regression
tests pass (95 total).
- Five executor integration cases passed before the rebase. Their
final-head rerun could not start because the local checkout was removed
during verification; no assertions ran in that last attempt.
- `pnpm -r typecheck` and `pnpm build` both pass on final head.
- Local verification limitations: the full `pnpm test:run` was
attempted, then stopped after database/process startup timeouts in
concurrent local runs. Two unchanged optional real-runner session cases
timed out on the initial run and one isolated retry; they were excluded
from the 95-test focused run. CI also skips those two binary-dependent
cases.
- All CI gates pass on final head `2eddaa485f`: typecheck, build,
general and serialized tests, runner checks, E2E, and canary dry run.
- Greptile reviewed final head at 5/5 with no actionable findings.

## Risks

This boundary controls native admission and cleanup ownership. Losing an
update after a later exception could change resource cleanup. Immediate
callbacks preserve those updates. Tests cover Stop races, lost
controller leases, immutable admitted inputs, competing runtime
selection, and workspace-sync failures. The change adds no schema or API
behavior.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, code editing, and command
execution. The session does not expose the exact model variant or
context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (95 focused tests on final
head; broader local limitations documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-10 07:31:15 -05:00
DottaandPaperclip d6e6d1c78a Stream OpenCode immediately, scope reasoning to turns, and deduplicate instructions (#15708)
## Thinking Path

> - Paperclip helps people manage AI agents and observe their work.
> - The native runner carries provider output into the task transcript.
> - OpenCode sends incremental text in `message.part.delta` events.
> - The driver ignored those events and waited for a full part snapshot.
Accepted chats also waited after lifecycle verification was complete.
> - This pull request forwards each valid chunk as it arrives and
removes unnecessary isolated startup work.
> - Valid chunks reach Paperclip sooner without changing completion or
permission rules. The matched browser experiment below does not
establish a reliable end-to-end latency improvement.

## Linked Issues or Issue Description

**What happened?**

The native OpenCode driver withheld text until OpenCode sent a full
snapshot. A local delayed-provider probe with the pinned OpenCode
1.18.34 runtime showed a 1,496 ms gap between the first native text
delta and the first Paperclip text delta (1,510 ms from provider text).

**Expected behavior**

Forward valid assistant text and reasoning deltas immediately. Preserve
repeated identical tokens, message roles, turn ownership, and final
response selection.

**Steps to reproduce**

Send an empty text part, then text deltas, and withhold the final full
snapshot. The new fixture requires the consumer to acknowledge its first
text delta before releasing the snapshot. The previous driver cannot
pass this handshake.

**Paperclip version or commit**

Baseline: `011e5669bad61c435a4745fb10812bef70376841`. The patch is
rebased onto current master.

**Deployment mode**

Local native OpenCode 1.18.34 on macOS arm64. Live diagnostics use
`openrouter/deepseek/deepseek-v4-flash-0731`. These are narrow
diagnostics, not full Product E2E qualification.

## What Changed

- Keep the macOS verified OpenCode launch pathname until child exit,
with guarded reuse and cleanup on failed launches. This fixes a
reproduced asynchronous executable-lifetime failure.
- Dispatch accepted queued chats immediately after lifecycle readiness
commits, through normal scheduler admission. Preserve ready state if
dispatch fails.
- Validate native OpenCode model access with the real hello probe
without a redundant catalog scan. Keep legacy adapter discovery
unchanged and isolate unspecified native probe directories.
- Map incremental text and reasoning to the existing canonical event
path, using observed part and message identities.
- Keep additive chunks without IDs distinct. Preserve ID-based event
deduplication.
- Clear current streaming part metadata when a new turn starts. Reject
mismatched session/message identities and unknown parts or fields.
- Disable automatic updates, remote model refreshes, and default plugins
in each isolated OpenCode process. Keep assigned MCP tools and default
model reasoning intact.
- Retain the latest cumulative text/reasoning part while its message
role is unknown, including completion metadata after more than 100
chunks.
- Expose `startTurn({ message, reasoningMode: "disabled" })` for one
OpenCode/OpenRouter turn. Carry the selection in the durable command
through Rust and the proxy. An omitted/default option restores provider
defaults on the next turn, including warm and recovered sessions. There
is no environment-level setting or automatic selection policy.
- Group optional Rust turn inputs in `CodexTurnOptions` and dispatch
through `start_turn_with_options`; retain `start_turn` for defaults.
Future turn settings can be added without another feature-specific
method.
- Expose a native-session capability for per-turn reasoning and reject
explicit unsupported selections before dispatch, including direct ACPX
and Codex backends. Runnerd checks the durable selected model as well as
the provider.
- Remove duplicate agent instructions from the shared OpenCode proxy
task envelope for all users. Send one system copy on every turn,
including prepared and resumed turns, because OpenCode reads the latest
user message’s system field. Preserve task constraints and completion
contracts.
- Document startup and streaming behavior. Add regression coverage for
role ordering, repeated chunks, final snapshot deduplication, and stale
turns.

## Verification

### Lower-load browser retry — 2026-10-10

**All 8/8 trials succeeded. In this batch, median Send → first visible
reply fell from 30.586 s to 12.476 s (59.2% lower).** Four fresh-agent
trials per variant, phase order before/after/after/before with two
trials per phase. Every adjacent-phase pair favored the candidate,
although one difference was only 0.301 s. This is evidence for the
readiness/probe changes on this host, not a production percentile or a
guarantee across providers.

| Phase/trial | Variant | Send → reply (s) | Observed ready → run (s) |
Session open → turn start (s) | Model requests |
|---|---|---:|---:|---:|---:|
| 0/0 | before | 45.042 | 25.337 | 3.629 | 2 |
| 0/1 | before | 13.689 | 0.000 | 2.443 | 2 |
| 1/0 | after | 14.696 | 0.000 | 2.526 | 1 |
| 1/1 | after | 13.389 | 0.000 | 2.470 | 2 |
| 2/0 | after | 10.871 | 0.000 | 2.706 | 2 |
| 2/1 | after | 11.564 | 0.004 | 2.443 | 2 |
| 3/0 | before | 41.582 | 21.518 | 2.376 | 2 |
| 3/1 | before | 19.589 | 9.753 | 2.707 | 1 |

Observed ready-to-run median: **15.636 s → 0 s**; zero means execution
began by the first ready observation, not literally zero latency.
Polling interval is nominally 250 ms plus HTTP time.
Session-open-to-turn-start includes controller handling, not just child
startup. All eight ran within the existing startup timeouts; no timeout
or reasoning policy was changed. Host load at fixture creation stayed
**5.47–8.73 on 18 CPUs**, versus 68.9–100.8 in the prior campaign. No
builds/tests ran concurrently with measurement.

Same merged source **b0a0158729a7913882a0e756fe7e27228d72bf53**, release
runner, rebuilt static UI, OpenCode **1.18.34**, model
**openrouter/deepseek/deepseek-v4-flash-0731**, provider-default
reasoning, and empty isolated workspaces in both variants. Only the
readiness worker and explicit-model verification source were toggled;
the launch-lifetime fix and all proxy streaming changes were held
constant. Thus this does not separately measure the original streaming
changes. One browser per phase, two fresh companies/agents/chats per
phase. Measurement begins at capture-phase Send click and ends on
visible Hi text plus two animation frames. Status/thinking is excluded;
emoji can arrive just after the first text. Successful screenshots
inspected. Each variant had three two-request turns using
`paperclip_finish` and one one-request turn; no status lookup was
observed. Reported usage includes settlement after the first visible
reply.

Runner SHA-256
`57f658c220eecd2370ffd3333cd7e5d06d16bb2798af0cf43c84e01c8207c1c3`;
proxy
`bcc02fda2bf35b38a116cd880841aa6c534f6e0b497a96d97369a1e06dcc72ac`;
rebuilt UI tree
`eee238f2eda628c4e8c3660707986da4f8dd9100c814e26938fce3756df2eae4`
(sorted relative path, NUL, binary per-file SHA-256). Worker/probe
variant source hashes are unchanged from the preserved comparison below.
Measured turn cost **$0.00612574**, plus verification probes not
included in that figure. All eight attempts retained; all source/proxy
files restored and test supervisors exited.

Current commit validation: **53 successful CI checks, 4 skipped, zero
failing; Greptile 5/5 on b0a0158729 with no actionable findings.** The
previous 20 targeted post-merge tests passed. No source edit was made
for this rerun. Earlier failed and inconclusive cohorts below remain
historical evidence and are not pooled with this batch.


### Master update and failure diagnosis — 2026-10-09

Merged `origin/master` at `417db14e7d` into this branch without
conflicts; pushed merge commit
`b0a0158729a7913882a0e756fe7e27228d72bf53`. The incoming changes do not
alter OpenCode process startup or its timeout budgets. Historical
measurements below remain attributed to their original commits.

The four last-campaign failures never issued `turn.start`, so these were
**cold-session admission failures, not model replies failing**. Two
retained `session.open` commands eventually completed after controller
timeout/cleanup had begun. The other two ended with the inner
`thread/start` response timeout, and their diagnostics recorded OpenCode
startup retries. Startup event timestamps are buffered publication times
and must not be interpreted as exact process-spawn times.

On the merged source, two credential-free startup-only checks used the
real pinned OpenCode 1.18.34, empty temporary workspaces, a placeholder
key, and **no model turns**:

- Existing pinned executable: spawn at 1.083 s; failed health
attempt/retry at 11.311 s and 21.914 s; third process healthy at 32.210
s; session opened at **35.168 s**.
- Verified private executable path: snapshot preparation took **0.497
s**; process spawned at 1.588 s; healthy at 9.967 s; session opened at
**11.120 s**.

These are sequential diagnostic samples, not evidence that one
executable path is faster. Host load was **56.5–64.5 on 18 logical
CPUs**. The local health wait permits 10 s per attempt and the driver
retries up to three times; the outer controller's cold admission budget
is 30 s, and the Rust facade also waits 30 s for a response. Thus
retries plus setup/session creation can outlast the caller's budget. The
isolated 35.168 s success demonstrates that a session can eventually
open yet fail the app's admission deadline. Heavy load aggravates the
timing; its exact contribution is not isolated. No timeout was increased
and no new model-latency claim is made.

Merge validation: **20/20 targeted lifecycle-policy and heartbeat
run-completion tests passed**, including temporary PostgreSQL cases. The
first sandbox invocation skipped database cases; the authorized local-DB
rerun executed all 20. The package-manager launcher could not verify its
version via the unreachable registry, so validation used the
already-installed Vitest binary. Full CI/review subsequently passed on
this merge commit; see the October 10 retry above. Older results below
apply to their named commits.


### Follow-up: reproduced macOS launch failure — 2026-10-09

The retry exposed a startup bug rather than only host-load noise. The
macOS proxy deleted the private verified executable path immediately
after asynchronous spawn. A credential-free test of the actual pinned
OpenCode `--version` reproduced a fast SIGKILL and a 15-second timeout
in three immediate-cleanup attempts; three deferred-cleanup controls
succeeded. The timeout was killed by the diagnostic's own deadline. This
experiment does not establish the OS-internal reason for the fast
SIGKILL.

Commit `c30451e0ebf3f74e9ef70d37bbe4d9fe7fb56359` retains the private
launch path until child exit, cleans failed spawns, and waits for forced
shutdown before reuse. Source descriptor authentication, ownership,
mode, inode checks, and the prohibition on ambient executable fallback
remain. Overlapping reuse fails closed. The existing synchronous shell
test was changed to exercise the actual asynchronous boundary.

**Verification:** 59 focused proxy/driver tests passed; runner
TypeScript check passed. All six subsequent real OpenCode version
launches succeeded without timeout (4.11, 4.33, 5.73, 6.91, 4.95, 6.70
seconds). The native Rust binary is unchanged from the retry; no model
calls were made by these diagnostics.

Retained interrupted retry: before 42.56/18.69 seconds; after one
180-second timeout (`provider_process_exited` / health SIGKILL) and one
37.33-second success. The latter overlapped the credential-free
diagnostic and is not a clean matched timing. The next phase was stopped
before Send to diagnose the reproduced bug. Reported measured-turn cost:
$0.0016874, verification probes additional. These attempts are not
pooled with other campaigns.

<!-- launch-fixed-proof:start -->
**Completed retry, stopped by the declared failure rule after 6 of 8
planned trials. No reliable end-to-end speedup demonstrated.** At commit
`c30451e0ebf3f74e9ef70d37bbe4d9fe7fb56359`, the launch fix above was
held constant in both variants; only the readiness handoff and
explicit-model verification path changed. Planned phase order:
before/after/after/before, two fresh agents/chats per phase, one new
browser per phase. The second after phase had two consecutive failures,
so the final before phase was not run.

| Phase/trial | Variant | Send → first visible reply | Observed ready →
run | Outcome |
|---|---|---|---|---|
| 0/0 | before | No reply within 180 s | 2.777 s | session.open timeout
|
| 0/1 | before | 47.654 s | 9.383 s | Succeeded |
| 1/0 | after | 50.051 s | 0.087 s | Succeeded |
| 1/1 | after | No reply within 180 s | 0.151 s | session.open timeout |
| 2/0 | after | No reply within 180 s | 3.337 s | session.open timeout |
| 2/1 | after | No reply within 180 s | 1.434 s | session.open timeout |

Both successful screenshots show “Hi 👋”. Timing starts at the browser's
capture-phase Send click and ends at the first visible answer text plus
two animation frames; the first fragment can be “Hi” before the emoji
arrives. Thinking/status text is not counted. All four failures were
`runner_local_connect_failed: PRP command session.open timed out`
(`native_session_interrupted`), before a reply; none reported the
earlier health-stage SIGKILL. They are retained, not discarded as
outliers. One success per variant does not support a speedup estimate or
median comparison. Observed ready-to-run median was **6.080 s before
(n=2) vs 0.793 s after (n=4)**; this is sampled every nominal 250 ms
plus HTTP latency and is a lower bound, not precise tracing. Host load
at fixture creation ranged **68.9–100.8 on 18 logical CPUs**, a
substantial confound that worsened over the campaign. No local
build/test was run concurrently with this campaign.

Provenance: pinned OpenCode **1.18.34**, model
**openrouter/deepseek/deepseek-v4-flash-0731**, default reasoning in
both variants, empty synthetic workspaces and isolated probe homes.
Release runner SHA-256
`57f658c220eecd2370ffd3333cd7e5d06d16bb2798af0cf43c84e01c8207c1c3`;
current proxy SHA-256
`bcc02fda2bf35b38a116cd880841aa6c534f6e0b497a96d97369a1e06dcc72ac`;
built UI tree SHA-256
`7f0c2b7f586cb753408351574b2239e22311944add99af4f6f3562cd6d47bbcb`. This
holds the rebuilt runner and static UI constant. Provider-reported
measured turn cost **$0.00120649**; verification probes are additional.
Including the earlier interrupted retry, this follow-up reports
**$0.00289389** in measured turn usage; missing usage is not assumed
free. Prior cohorts remain separate below.

Validation on `c30451e0e`: **59 focused tests passed**, runner
TypeScript check passed, **6/6 real credential-free pinned OpenCode
launches passed**, fresh Greptile **5/5 with zero unresolved threads**.
CI: **51 success, 4 skipped, 1 failed**. The failed native compilation
job never reached compilation: Docker's auth endpoint timed out during
Buildx setup, then failed the same way on one retry
([job](https://github.com/paperclipai/paperclip/actions/runs/37994181674/job/114039737503)).
Full local checks were not rerun while measuring. The PR remains draft:
the launch defect is reproduced and fixed, but cold session startup
still exceeds the controller's admission budget on this host. Raising
that budget would not establish a faster first response.
<!-- launch-fixed-proof:end -->

### Readiness investigation and live rerun — 2026-10-09

**Measured queue fix; no reliable end-to-end speedup claim. Keep this PR
in draft.** Accepted chats could wait for the periodic queue sweep after
verification had already committed `ready`. The lifecycle worker now
dispatches through the existing scheduler immediately after that commit.
The normal company, pause, budget, concurrency, and task-ownership
checks still apply. Dispatch failure preserves readiness and leaves
recovery to the queue sweep.

Native OpenCode verification also avoids enumerating the full model
catalog before running its real hello request. Identifier validation
remains. Authentication failures, unavailable models, and timeouts still
prevent readiness. An unspecified local probe directory uses an empty
temporary directory.

**Queue-only experiment:** four before and four after attempts, in
before/after/after/before server phases, two fresh agents per phase.
Median observed ready-to-start delay fell from **19.429 s to 0.058 s**.
State polling was nominally 250 ms plus HTTP time, so the latter means
within observation resolution; it is not a precise 58 ms latency claim.
After timings were 100.20, 65.29, 63.78, and 76.97 seconds. Before had
one 59.65-second success and three 120-second response timeouts (one
provider health SIGKILL, one late success, one still running after
settlement and canceled). These censored results do not support a
success-only median speedup comparison. Focused tests overlapped the
last baseline phase, which further limits total-time interpretation.
Reported measured-turn cost: $0.0055383; verification probes and
incomplete/canceled usage are additional.

**Final matched experiment:** built static UI in both variants to avoid
pre-Send Vite compilation; current proxy, release runner, selected
model, default reasoning, and explicitly isolated empty probe
cwd/HOME/XDG held constant. Both worker handoff and model-probe changes
vary. Fresh browser/company/agent/chat per attempt. ABBA order, two
attempts per variant, 180-second response limit declared before the
batch, four-turn bound. Same Send-to-rendered-Hi metric as the earlier
experiment.

| Order | Variant | Send → visible Hi | Created → execution | Observed
ready → execution | Outcome |
| --- | --- | ---: | ---: | ---: | --- |
| 1 | Before | 62.66 s | 23.467 s | 2.830 s | Succeeded |
| 2 | After | >180 s | 14.200 s | Within polling resolution | Provider
health SIGKILL |
| 3 | After | >180 s | 13.877 s | Within polling resolution | Provider
initialization timeout |
| 4 | Before | >180 s | 144.256 s | 14.067 s | Still running after
settlement; canceled |

Only one of four attempts produced a timed reply. **There is no valid
overall speedup estimate from this batch.** The dispatch delay is
removed, but provider startup failures and heavy shared-host load
remain. A recorded host sample was load 93.55 on 18 logical CPUs. This
is a confound, not proof that every failure was caused by host load.
Provider-reported measured-turn cost was $0.00032752, with no reported
usage for the other three attempts; verification probes are additional.

Retained earlier setup attempts: one shared-memory startup failure
before model calls, and a separate development-UI batch with a
60.06-second baseline followed by a navigation timeout before Send.
These attempts are retained separately, not counted as successful
response timings. No live attempts were silently dropped.

Provenance: source base `53580f8fbd40b963f3fb7ba5172ff3ede2e9ef1f`,
measured patch committed as `40b54eafb`, then rebased as
`c590770dcc2bb1debce14037935a7031e5c3c1ac`. **These live timings predate
the rebase.** Source and binary hashes are retained below. The inner
harness label `after` selects the identical current proxy in every
attempt; the outer campaign handoff label is the actual before/after
variant. Owned benchmark processes stopped and original proxy restored.

<!-- readiness-check-status:start -->
Post-rebase validation at `c590770dcc2bb1debce14037935a7031e5c3c1ac`:

- Full local `pnpm -r typecheck`: **passed**, including the native
release build (1,098.6 seconds on the loaded host).
- Focused lifecycle and adapter tests: **26 passed**. The 14 database
integration tests did not run locally because their hard-coded 20-second
database setup hook timed out. A retry with a CLI timeout override
encountered the same explicit hook limit. The earlier pre-rebase focused
batch passed 77 tests; these results are not substituted for post-rebase
execution.
- Local `pnpm test:run` was started, then stopped to avoid duplicating
CI on the saturated host. Local `pnpm build` was not run separately.
**Current-head CI passed Typecheck + Release Registry, Build, all
general and serialized test groups, Rust checks, both runner Vitest
groups, and all eight browser E2E shards.** The last general shard
passed 1,179 tests (72 files).
- Fresh [Greptile
review](https://github.com/paperclipai/paperclip/pull/15708#issuecomment-6085742217):
**5/5**, reviewed this exact head, no actionable defect.
- Final CI rollup: **49 success, 4 skipped, 4 failure**. The failures
are [Compile isolated native
Runner](https://github.com/paperclipai/paperclip/actions/runs/37990274515/job/114024329242),
[Docker context
integrity](https://github.com/paperclipai/paperclip/actions/runs/37990274623/job/114024196511),
[Canary Dry
Run](https://github.com/paperclipai/paperclip/actions/runs/37990274623/job/114024196424),
and the dependent `ci / verify` aggregate. All three root failures are
Docker Hub HTTP 429 / unauthenticated pull-rate limits while fetching
Node base images. The native-image job was retried once and hit the same
limit. No source compiler/test failure was reported by those gates. CI
maintainers need authenticated pulls or to rerun after the registry
limit clears.
- Owned benchmark servers stopped. Local validation commands completed
or were stopped. The PR remains a draft: user-visible speed/reliability
is not qualified, and the Docker-dependent CI gates are not green.
<!-- readiness-check-status:end -->

<details><summary>Sanitized readiness evidence with all attempts and
source hashes</summary>

```json
{"campaign":{"order":["before","after","after","before"],"trialsPerPhase":1,"maxMeasuredTurns":4,"maxCampaignUsd":1,"sourceHead":"53580f8fbd40b963f3fb7ba5172ff3ede2e9ef1f","reasoning":"provider default, no override","proxySha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","purpose":"Final matched comparison of both optimizations; distinct from earlier queue-only campaign.","responseTimeoutMs":180000,"timeoutRationale":"Declared before this final campaign because earlier unchanged baselines sometimes exceeded 120 seconds; earlier timeouts remain retained.","ui":{"mode":"built static UI","treeSha256":"7f0c2b7f586cb753408351574b2239e22311944add99af4f6f3562cd6d47bbcb"},"priorCampaign":"combined-proof stopped after a development-UI navigation timeout before Send; no failed attempts erased."},"complete":true,"statistics":{"before":{"attempted":2,"successfulMeasurements":1,"medianTtfrMs":62657.2000002861,"rangeTtfrMs":[62657.2000002861,62657.2000002861],"medianObservedReadyToRunMs":8448.5,"medianRunStartToVisibleMs":37290.199951171875},"after":{"attempted":2,"successfulMeasurements":0,"medianTtfrMs":null,"rangeTtfrMs":null,"medianObservedReadyToRunMs":0.0,"medianRunStartToVisibleMs":null}},"measuredTurnCostUsd":0.00032752,"costCoverage":"Provider-reported measured chat turns only; harness verification probes are additional and not included.","readinessMetricLimit":"First observed ready state sampled every 250 ms plus HTTP duration; clamped to zero if execution started before the observation. This is a lower bound on ready-to-start delay, not a precise sub-millisecond metric.","results":[{"phase":0,"trial":0,"handoff":"before","ttfrMs":62657.2000002861,"answer":"Hi","statuses":["succeeded"],"error":null,"runErrorCode":null,"providerHealthKilled":false,"queueMs":23467.0,"queuedBeforeObservedReady":true,"observedVerificationMsAfterFixture":26390.0,"observedReadyToRunMs":2830.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":37290.199951171875,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00032752,"costUsdExact":"0.000327520","inputTokens":26864,"outputTokens":46,"cachedInputTokens":0,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326"},{"phase":1,"trial":0,"handoff":"after","ttfrMs":null,"answer":null,"statuses":["failed"],"error":"TimeoutError: page.waitForFunction: Timeout 180000ms exceeded.","runErrorCode":"provider_process_exited","providerHealthKilled":true,"queueMs":14200.0,"queuedBeforeObservedReady":true,"observedVerificationMsAfterFixture":24028.0,"observedReadyToRunMs":0,"readyAndStartedInSameSample":true,"runStartToVisibleMs":null,"usage":[{}],"workerSha256":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7"},{"phase":2,"trial":0,"handoff":"after","ttfrMs":null,"answer":null,"statuses":["failed"],"error":"TimeoutError: page.waitForFunction: Timeout 180000ms exceeded.","runErrorCode":"provider_initialize_timeout","providerHealthKilled":false,"queueMs":13877.0,"queuedBeforeObservedReady":true,"observedVerificationMsAfterFixture":18107.0,"observedReadyToRunMs":0,"readyAndStartedInSameSample":true,"runStartToVisibleMs":null,"usage":[{}],"workerSha256":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7"},{"phase":3,"trial":0,"handoff":"before","ttfrMs":null,"answer":null,"statuses":["running"],"error":"TimeoutError: page.waitForFunction: Timeout 180000ms exceeded.","runErrorCode":null,"providerHealthKilled":false,"queueMs":144256.0,"queuedBeforeObservedReady":true,"observedVerificationMsAfterFixture":136901.0,"observedReadyToRunMs":14067.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":null,"usage":[{}],"workerSha256":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326"}],"hostLoadSample":{"at":"2026-10-09T20:48:40.031Z","parallelism":18,"load":[93.546875,82.68017578125,75.9091796875]},"measuredChangesCommit":"40b54eafb","postRebaseCommit":"c590770dcc2bb1debce14037935a7031e5c3c1ac","postRebaseLimit":"Live benchmark predates the rebase. No post-rebase live result is claimed.","variantSourceHashes":{"before":{"server/src/modules/agent-lifecycle/application/worker.ts":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326","packages/adapters/opencode-local/src/server/test.ts":"4c10bacb55f2c7589e7b5c79d74781206ae075db8955f5b998065d6db3c0335a","server/src/adapters/registry.ts":"4a858763f69947d5745c7b8cb251a62fa404bc9683ade0ffb76282ed12761394"},"after":{"server/src/modules/agent-lifecycle/application/worker.ts":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7","packages/adapters/opencode-local/src/server/test.ts":"cd2ea3d51c3f1b4fb42261be701fb7f6b85704ad8e641e6db73d7c58e47364d3","server/src/adapters/registry.ts":"4a858763f69947d5745c7b8cb251a62fa404bc9683ade0ffb76282ed12761394"}},"model":"openrouter/deepseek/deepseek-v4-flash-0731","prompt":"Reply with Hi and an emoji and nothing else.","runnerBinary":{"profile":"release","sha256":"6134c17debda14a76da69ac9631a2d10dad482ebd58dbc9a4e4480ed652211c4"},"probeIsolation":"Explicit empty cwd and HOME/XDG folders under the synthetic fixture workspace in BOTH variants. No server checkout or ambient provider home is used for the hello request.","comparisonLabelNote":"Inner run.mts variant after selects the identical current proxy in every trial; outer campaign readinessHandoff controls before/after source changes."}
```

Queue-only campaign:

```json
{"campaign":{"order":["before","after","after","before"],"trialsPerPhase":2,"maxMeasuredTurns":8,"maxCampaignUsd":1,"sourceHead":"53580f8fbd40b963f3fb7ba5172ff3ede2e9ef1f","reasoning":"provider default, no override","proxySha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04"},"complete":true,"statistics":{"before":{"attempted":4,"successfulMeasurements":1,"medianTtfrMs":59654.5,"rangeTtfrMs":[59654.5,59654.5],"medianObservedReadyToRunMs":19429.0,"medianRunStartToVisibleMs":8843.5},"after":{"attempted":4,"successfulMeasurements":4,"medianTtfrMs":71128.7000002861,"rangeTtfrMs":[63778.90000009537,100195.59999990463],"medianObservedReadyToRunMs":58.0,"medianRunStartToVisibleMs":26542.849975585938}},"measuredTurnCostUsd":0.0055383,"costCoverage":"Provider-reported measured chat turns only; harness verification probes are additional and not included.","readinessMetricLimit":"First observed ready state sampled every 250 ms plus HTTP duration; clamped to zero if execution started before the observation. This is a lower bound on ready-to-start delay, not a precise sub-millisecond metric.","results":[{"phase":0,"trial":0,"handoff":"before","ttfrMs":59654.5,"answer":"Hi 👋","statuses":["succeeded"],"error":null,"queueMs":50590.0,"observedReadyToRunMs":25709.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":8843.5,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00087852,"costUsdExact":"0.000878520","inputTokens":25644,"outputTokens":264,"cachedInputTokens":28416,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326"},{"phase":0,"trial":1,"handoff":"before","ttfrMs":null,"answer":null,"statuses":["failed"],"error":"TimeoutError: page.waitForFunction: Timeout 120000ms exceeded.","queueMs":41651.0,"observedReadyToRunMs":14656.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":null,"usage":[{}],"workerSha256":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326"},{"phase":1,"trial":0,"handoff":"after","ttfrMs":100195.59999990463,"answer":"Hi 🙂","statuses":["succeeded"],"error":null,"queueMs":65893.0,"observedReadyToRunMs":116.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":33243.60009765625,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00075782,"costUsdExact":"0.000757820","inputTokens":27014,"outputTokens":175,"cachedInputTokens":26368,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7"},{"phase":1,"trial":1,"handoff":"after","ttfrMs":65288.60000038147,"answer":"Hi 👋","statuses":["succeeded"],"error":null,"queueMs":43511.0,"observedReadyToRunMs":0,"readyAndStartedInSameSample":true,"runStartToVisibleMs":21227.60009765625,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00085326,"costUsdExact":"0.000853260","inputTokens":25166,"outputTokens":250,"cachedInputTokens":28160,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7"},{"phase":2,"trial":0,"handoff":"after","ttfrMs":63778.90000009537,"answer":"Hi 👋","statuses":["succeeded"],"error":null,"queueMs":39218.0,"observedReadyToRunMs":0,"readyAndStartedInSameSample":true,"runStartToVisibleMs":23479.89990234375,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.000839,"costUsdExact":"0.000839000","inputTokens":27068,"outputTokens":238,"cachedInputTokens":26368,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7"},{"phase":2,"trial":1,"handoff":"after","ttfrMs":76968.80000019073,"answer":"Hi 👋","statuses":["succeeded"],"error":null,"queueMs":46257.0,"observedReadyToRunMs":185.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":29605.800048828125,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00030582,"costUsdExact":"0.000305820","inputTokens":24694,"outputTokens":32,"cachedInputTokens":1792,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"ceffb90a6bc97ffaaf8ec7d0267c7bfa7738d321aaf2a618b5efb9b114bc88e7"},{"phase":3,"trial":0,"handoff":"before","ttfrMs":null,"answer":null,"statuses":["succeeded"],"error":"TimeoutError: page.waitForFunction: Timeout 120000ms exceeded.","queueMs":100979.0,"observedReadyToRunMs":3872.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":null,"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00190388,"costUsdExact":"0.001903880","inputTokens":25524,"outputTokens":852,"cachedInputTokens":55808,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"workerSha256":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326"},{"phase":3,"trial":1,"handoff":"before","ttfrMs":null,"answer":null,"statuses":["running"],"error":"TimeoutError: page.waitForFunction: Timeout 120000ms exceeded.","queueMs":129391.0,"observedReadyToRunMs":24202.0,"readyAndStartedInSameSample":false,"runStartToVisibleMs":null,"usage":[{}],"workerSha256":"02adc7e1162fb60439869cec62b75d8fb22fec71155f80535c8e00211cde8326"}]}
```
</details>

### Matched live browser latency evidence — 2026-10-09

**Conclusion: the overall first-response speedup is not established.**
Twelve successful real browser runs produced a lower after median, but
the updated path won only 3 of 6 adjacent pairs. Median paired
improvement was -0.17 seconds (effectively tied). Do not treat the
deterministic streaming-forwarding improvement as proof that this PR
fixes end-to-end first-response latency.

- **Actual path:** local Paperclip browser → real server/database →
current optimized Rust runner → pinned OpenCode 1.18.34 → OpenRouter
`deepseek/deepseek-v4-flash-0731`. Default reasoning in both variants;
no disabled-reasoning override.
- **Prompt:** `Reply with Hi and an emoji and nothing else.` Fresh
company, agent, chat, and per-turn native provider process for each
trial; same QA fixture instructions and selected model.
- **Comparison:** current product/runner at
`53580f8fbd40b963f3fb7ba5172ff3ede2e9ef1f` held constant. The baseline
substitutes the exact `opencode-app-server-proxy.ts`,
`opencode-proxy-task-envelope.ts`, and `opencode-server-driver.ts`
contents from pre-PR `41fec8c8921f9fb19c13fe11e2c9b71bf1017317`. This
isolates those changed modules; it is not an entire old-checkout
comparison.
- **Metric:** browser capture-phase Send click to a rendered `Hi`
markdown answer plus two animation frames, with a screenshot of the
answer. Thinking, tool labels, and status indicators do not count. This
is a DOM/render measurement, not instrumented compositor pixel timing.
- **Order/bound:** predeclared `before, after, after, before` repeated
three times; six trials per variant; 120-second response timeout; $1
campaign guard. All 12 measured trials returned `Hi` and an emoji and
finished successfully; none omitted.
- **Result:** before median **48.08 s** (37.42–66.40); after median
**42.50 s** (27.68–63.32). The 11.6% lower median alone is not
persuasive evidence of a reliable improvement given the mixed paired
results and broad overlap.
- **Remaining delay:** retained queue timestamps for 10/12 trials show
**9.76–41.65 seconds from run creation to execution start**. This is
before the OpenCode path changed here. Its cause is not established by
this benchmark. The final two queue reads were unavailable after
automatic server shutdown; all twelve primary browser timings were
retained.
- **Cost:** $0.010517369 provider-reported across the 12 measured turns;
$0.011371499 including the successful setup pilot. This is separate from
the older diagnostics listed below.
- **Retained setup attempts:** two failures before Send (composer
locator and transpiler helper); one baseline startup failure before any
observed model request (`SIGKILL` while probing OpenCode health); one
successful current-code pilot whose selector missed the visible reply.
These are retained separately, not silently included/excluded as valid
timings. The measured batch used the corrected selector and the same
release binary for both variants.
- **Limits:** local development server and headless Chromium on macOS
arm64, small sample, variable queue timing/provider routing/cache. This
does not qualify production/cloud performance or isolate the
contributions of each changed module. Server, browser, and owned runs
were stopped; original proxy build artifact restored; no product source
files changed for this experiment.

| Pair | Before: Send → visible Hi | After: Send → visible Hi | Faster
variant |
| --- | ---: | ---: | --- |
| 1 | 66.40 s | 27.68 s | After |
| 2 | 37.42 s | 28.00 s | After |
| 3 | 39.75 s | 55.22 s | Before |
| 4 | 47.32 s | 63.32 s | Before |
| 5 | 48.85 s | 29.79 s | After |
| 6 | 51.31 s | 61.07 s | Before |

<details>
<summary>Sanitized measurement records, source/binary hashes, usage, and
retained attempt classification</summary>

```json
{"manifest":{"candidate":"53580f8fbd40b963f3fb7ba5172ff3ede2e9ef1f","baseline":"41fec8c8921f9fb19c13fe11e2c9b71bf1017317","comparison":"Same current product and runner; replace the three OpenCode proxy/driver source modules with exact pre-PR contents for baseline. Reasoning uses defaults in both.","model":"openrouter/deepseek/deepseek-v4-flash-0731","prompt":"Reply with Hi and an emoji and nothing else.","samplesPerVariant":6,"order":["before","after","after","before","before","after","after","before","before","after","after","before"],"maxTurnMs":120000,"maxCostUsd":1,"variants":{"before":{"sources":{"src/cli/opencode-app-server-proxy.ts":"e9b46c937d7ce3b51dc44181a2a3ad82befda9ebe184adf80c0a6b30da140fd6","src/cli/opencode-proxy-task-envelope.ts":"fc23f8a515c2c4d1551f6b95e5b38e6254849dec23f7396e76b646e170a6e006","src/drivers/opencode/opencode-server-driver.ts":"99de415e8fc3bb5fb6f8d7512d751c4c37d2aa77a18b0546b9fb232f1a6fb6c5"},"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b"},"after":{"sources":{"src/cli/opencode-app-server-proxy.ts":"fa800e9bccce02f05f1af83e225d3b5a1300d0a7d4c35ac548f400d6c0a83f79","src/cli/opencode-proxy-task-envelope.ts":"5c387614fc9832322686326b8482eb00d88699d491b05cf59cadcd94a41194ea","src/drivers/opencode/opencode-server-driver.ts":"a944f3bf7f1bf897daec3a5dc2aa34ee9d1a2133efcb3cf150b002095a0ec586"},"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04"}},"runnerBinary":{"profile":"release","sha256":"6134c17debda14a76da69ac9631a2d10dad482ebd58dbc9a4e4480ed652211c4"},"metric":"Browser Send capture-phase click to visible Hi answer, sampled on animation frames, plus two paint opportunities. Excludes reasoning/tool/status text.","setupAttempts":["setup-failure-1","setup-failure-2","startup-failure-3","pilot-selector-failure"]},"summary":{"before":{"attempts":6,"measured":6,"medianMs":48083.050000190735,"minMs":37420.5,"maxMs":66397.69999980927,"totalCostUsd":0.004945229,"toolCalls":[4,1,0,0,0,1]},"after":{"attempts":6,"measured":6,"medianMs":42504.049999952316,"minMs":27680.099999904633,"maxMs":63320.90000009537,"totalCostUsd":0.00557214,"toolCalls":[0,1,0,4,0,1]},"medianReductionPercent":11.602841334350234,"medianPairedImprovementMs":-168.59999990463257,"reportedCampaignCostUsdIncludingPilot":0.011371498999999998},"pairs":[{"indices":[0,1],"beforeMs":66397.69999980927,"afterMs":27680.099999904633,"improvementMs":38717.59999990463},{"indices":[2,3],"beforeMs":37420.5,"afterMs":27997,"improvementMs":9423.5},{"indices":[4,5],"beforeMs":39751.60000038147,"afterMs":55217.199999809265,"improvementMs":-15465.599999427795},{"indices":[6,7],"beforeMs":47318.90000009537,"afterMs":63320.90000009537,"improvementMs":-16002.0},{"indices":[8,9],"beforeMs":48847.2000002861,"afterMs":29790.900000095367,"improvementMs":19056.300000190735},{"indices":[10,11],"beforeMs":51311.7000002861,"afterMs":61072.40000009537,"improvementMs":-9760.699999809265}],"trials":[{"index":0,"variant":"before","startedAt":"2026-10-09T19:15:23.172Z","measurementError":null,"ttfrMs":66397.69999980927,"domMs":66390.59999990463,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.002612595,"costUsdExact":"0.002612595","inputTokens":55962,"outputTokens":1344,"cachedInputTokens":83456,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b","toolCalls":["paperclip_get_task_context","paperclip_comment_on_task","paperclip_finish","paperclip_finish"],"eventCounts":{"turn.started":1,"tool.execution.started":4,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:15:58.194Z","turn.submitted":"2026-10-09T19:15:58.311Z","turn.started":"2026-10-09T19:15:59.063Z","firstTextDelta":"2026-10-09T19:16:36.039Z","turn.completed":"2026-10-09T19:16:36.118Z"},"completedRunDurationsMs":[42915.0]},{"index":1,"variant":"after","startedAt":"2026-10-09T19:16:36.598Z","measurementError":null,"ttfrMs":27680.099999904633,"domMs":27636.599999904633,"answer":"Hi","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00030429,"costUsdExact":"0.000304290","inputTokens":26845,"outputTokens":28,"cachedInputTokens":0,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","toolCalls":[],"eventCounts":{"turn.started":1,"item.delta":3,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:17:00.717Z","turn.submitted":"2026-10-09T19:17:00.841Z","turn.started":"2026-10-09T19:17:01.508Z","firstTextDelta":"2026-10-09T19:17:05.934Z","turn.completed":"2026-10-09T19:17:06.205Z"},"completedRunDurationsMs":[14435.0]},{"index":2,"variant":"after","startedAt":"2026-10-09T19:17:08.762Z","measurementError":null,"ttfrMs":27997,"domMs":27984.599999904633,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00101016,"costUsdExact":"0.001010160","inputTokens":27544,"outputTokens":366,"cachedInputTokens":26624,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","toolCalls":["paperclip_finish"],"eventCounts":{"turn.started":1,"tool.execution.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:17:26.192Z","turn.submitted":"2026-10-09T19:17:26.305Z","turn.started":"2026-10-09T19:17:26.936Z","firstTextDelta":"2026-10-09T19:17:40.688Z","turn.completed":"2026-10-09T19:17:40.826Z"},"completedRunDurationsMs":[17763.0]},{"index":3,"variant":"before","startedAt":"2026-10-09T19:17:41.376Z","measurementError":null,"ttfrMs":37420.5,"domMs":37364.40000009537,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.000693018,"costUsdExact":"0.000693018","inputTokens":24236,"outputTokens":282,"cachedInputTokens":27648,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b","toolCalls":["paperclip_finish"],"eventCounts":{"turn.started":1,"tool.execution.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:18:12.381Z","turn.submitted":"2026-10-09T19:18:12.616Z","turn.started":"2026-10-09T19:18:13.441Z","firstTextDelta":"2026-10-09T19:18:21.746Z","turn.completed":"2026-10-09T19:18:22.262Z"},"completedRunDurationsMs":[29216.0]},{"index":4,"variant":"before","startedAt":"2026-10-09T19:18:23.000Z","measurementError":null,"ttfrMs":39751.60000038147,"domMs":39705.7000002861,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.000265747,"costUsdExact":"0.000265747","inputTokens":26123,"outputTokens":77,"cachedInputTokens":0,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b","toolCalls":[],"eventCounts":{"turn.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:19:01.000Z","turn.submitted":"2026-10-09T19:19:01.163Z","turn.started":"2026-10-09T19:19:01.893Z","firstTextDelta":"2026-10-09T19:19:07.169Z","turn.completed":"2026-10-09T19:19:07.384Z"},"completedRunDurationsMs":[15843.0]},{"index":5,"variant":"after","startedAt":"2026-10-09T19:19:10.050Z","measurementError":null,"ttfrMs":55217.199999809265,"domMs":55204.89999961853,"answer":"Hi 🙂","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00030421,"costUsdExact":"0.000304210","inputTokens":4294,"outputTokens":28,"cachedInputTokens":22543,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","toolCalls":[],"eventCounts":{"turn.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:19:59.877Z","turn.submitted":"2026-10-09T19:19:59.990Z","turn.started":"2026-10-09T19:20:00.713Z","firstTextDelta":"2026-10-09T19:20:07.290Z","turn.completed":"2026-10-09T19:20:07.658Z"},"completedRunDurationsMs":[15104.0]},{"index":6,"variant":"after","startedAt":"2026-10-09T19:20:09.550Z","measurementError":null,"ttfrMs":63320.90000009537,"domMs":63313.09999990463,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00260723,"costUsdExact":"0.002607230","inputTokens":137971,"outputTokens":951,"cachedInputTokens":1024,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","toolCalls":["paperclip_comment_on_task","paperclip_get_task_context","paperclip_comment_on_task","paperclip_finish"],"eventCounts":{"turn.started":1,"tool.execution.started":4,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:20:59.730Z","turn.submitted":"2026-10-09T19:20:59.859Z","turn.started":"2026-10-09T19:21:00.501Z","firstTextDelta":"2026-10-09T19:21:19.416Z","turn.completed":"2026-10-09T19:21:19.608Z"},"completedRunDurationsMs":[26519.0]},{"index":7,"variant":"before","startedAt":"2026-10-09T19:21:20.519Z","measurementError":null,"ttfrMs":47318.90000009537,"domMs":47246.800000190735,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.000257965,"costUsdExact":"0.000257965","inputTokens":4700,"outputTokens":71,"cachedInputTokens":21407,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b","toolCalls":[],"eventCounts":{"turn.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:22:03.924Z","turn.submitted":"2026-10-09T19:22:04.439Z","turn.started":"2026-10-09T19:22:04.772Z","firstTextDelta":"2026-10-09T19:22:09.915Z","turn.completed":"2026-10-09T19:22:10.343Z"},"completedRunDurationsMs":[16702.0]},{"index":8,"variant":"before","startedAt":"2026-10-09T19:22:11.124Z","measurementError":null,"ttfrMs":48847.2000002861,"domMs":48802.59999990463,"answer":"Hi 🎉","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.000327654,"costUsdExact":"0.000327654","inputTokens":44796,"outputTokens":32,"cachedInputTokens":0,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b","toolCalls":[],"eventCounts":{"turn.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:22:57.182Z","turn.submitted":"2026-10-09T19:22:57.281Z","turn.started":"2026-10-09T19:22:57.988Z","firstTextDelta":"2026-10-09T19:23:02.915Z","turn.completed":"2026-10-09T19:23:03.035Z"},"completedRunDurationsMs":[9541.0]},{"index":9,"variant":"after","startedAt":"2026-10-09T19:23:03.579Z","measurementError":null,"ttfrMs":29790.900000095367,"domMs":29748.800000190735,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00034054,"costUsdExact":"0.000340540","inputTokens":24710,"outputTokens":59,"cachedInputTokens":1792,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","toolCalls":[],"eventCounts":{"turn.started":1,"item.delta":2,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:23:28.840Z","turn.submitted":"2026-10-09T19:23:28.954Z","turn.started":"2026-10-09T19:23:29.666Z","firstTextDelta":"2026-10-09T19:23:35.306Z","turn.completed":"2026-10-09T19:23:35.531Z"},"completedRunDurationsMs":[12732.0]},{"index":10,"variant":"after","startedAt":"2026-10-09T19:23:37.247Z","measurementError":null,"ttfrMs":61072.40000009537,"domMs":61061.300000190735,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00100571,"costUsdExact":"0.001005710","inputTokens":25307,"outputTokens":368,"cachedInputTokens":28160,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6691475ed55f10179ec6ee0023a1f6462982f763848da7cd5b1d0baf38633b04","toolCalls":["paperclip_finish"],"eventCounts":{"turn.started":1,"tool.execution.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:24:33.721Z","turn.submitted":"2026-10-09T19:24:33.832Z","turn.started":"2026-10-09T19:24:34.601Z","firstTextDelta":"2026-10-09T19:24:43.135Z","turn.completed":"2026-10-09T19:24:43.278Z"},"completedRunDurationsMs":[17387.0]},{"index":11,"variant":"before","startedAt":"2026-10-09T19:24:44.710Z","measurementError":null,"ttfrMs":51311.7000002861,"domMs":51300.5,"answer":"Hi 👋","runCount":1,"statuses":["succeeded"],"usage":[{"model":"openrouter/deepseek/deepseek-v4-flash-0731","costUsd":0.00078825,"costUsdExact":"0.000788250","inputTokens":24316,"outputTokens":356,"cachedInputTokens":27648,"cacheWriteTokens":0,"usageSource":"per_run","costStatus":"reported"}],"bundleSha256":"6dd8726e72545341f6772e54ec5aeb6264dec74db67290df44930d87135efb6b","toolCalls":["paperclip_finish"],"eventCounts":{"turn.started":1,"tool.execution.started":1,"item.delta":1,"turn.completed":1},"eventTimes":{"session.started":"2026-10-09T19:25:31.347Z","turn.submitted":"2026-10-09T19:25:31.463Z","turn.started":"2026-10-09T19:25:32.208Z","firstTextDelta":"2026-10-09T19:25:39.405Z","turn.completed":"2026-10-09T19:25:39.563Z"},"completedRunDurationsMs":[15446.0]}],"additionalAttempts":[{"attempt":"setup-1","stage":"before_send","result":"non-editable composer locator","modelRequestObserved":false},{"attempt":"setup-2","stage":"before_send","result":"transpiler helper absent in browser","modelRequestObserved":false},{"attempt":"startup-pilot-before","runnerProfile":"debug","result":"no visible reply within 120s; OpenCode health startup SIGKILL; retained infrastructure failure","modelRequestObserved":false},{"attempt":"selector-pilot-after","runnerProfile":"debug","result":"Hi reply succeeded; browser selector missed it; no valid TTFR","providerReportedCostUsd":0.00085413}],"queueTimingCoverage":{"measuredTrials":10,"totalTrials":12,"records":[{"index":0,"runs":[{"createdAt":"2026-10-09T19:15:28.998Z","startedAt":"2026-10-09T19:15:53.558Z","queuedMs":24560.0}]},{"index":1,"runs":[{"createdAt":"2026-10-09T19:16:38.520Z","startedAt":"2026-10-09T19:16:54.157Z","queuedMs":15637.0}]},{"index":2,"runs":[{"createdAt":"2026-10-09T19:17:10.587Z","startedAt":"2026-10-09T19:17:23.561Z","queuedMs":12974.0}]},{"index":3,"runs":[{"createdAt":"2026-10-09T19:17:43.807Z","startedAt":"2026-10-09T19:17:53.564Z","queuedMs":9757.0}]},{"index":4,"runs":[{"createdAt":"2026-10-09T19:18:28.113Z","startedAt":"2026-10-09T19:18:54.011Z","queuedMs":25898.0}]},{"index":5,"runs":[{"createdAt":"2026-10-09T19:19:12.343Z","startedAt":"2026-10-09T19:19:53.993Z","queuedMs":41650.0}]},{"index":6,"runs":[{"createdAt":"2026-10-09T19:20:14.258Z","startedAt":"2026-10-09T19:20:53.866Z","queuedMs":39608.0}]},{"index":7,"runs":[{"createdAt":"2026-10-09T19:21:23.019Z","startedAt":"2026-10-09T19:21:54.298Z","queuedMs":31279.0}]},{"index":8,"runs":[{"createdAt":"2026-10-09T19:22:14.671Z","startedAt":"2026-10-09T19:22:53.933Z","queuedMs":39262.0}]},{"index":9,"runs":[{"createdAt":"2026-10-09T19:23:06.155Z","startedAt":"2026-10-09T19:23:23.941Z","queuedMs":17786.0}]}],"limitation":"The server shut down before a final API read; queue timing is retained for the first ten trials only."},"conclusion":"No reliable end-to-end speedup established. The after median is lower, but only three of six adjacent matched pairs were faster; median paired improvement is approximately zero. The deterministic streaming-forwarding fix remains separately supported.","limitations":["Local dev server and headless Chromium on macOS arm64; six fresh chats per variant; not a production/cloud latency guarantee.","Current product and current release runner held constant; only three OpenCode modules replaced by their exact pre-PR contents for baseline.","Default model reasoning unchanged; no no-reasoning override or decision policy.","Upstream routing/cache and queue time vary; trials run serially in predeclared counterbalanced ABBA order.","Render proxy: capture-phase Send click to visible nonempty Hi markdown, then two animation frames; screenshot verifies the rendered answer. This is not instrumented compositor pixel timing.","Two setup-only failures, one startup failure, and one invalid-measurement successful pilot are retained separately, not silently treated as successful timings."],"reproducibility":{"harnessSha256":"f93996b306c7d31c19bfcb9cc197653f867944ffb900d81a23a34f831e17c0c5","variantBuilderSha256":"4e88a7bbdc0de5d72601194ebe3b96d9c6f6eda5a33d8e0f00cf2d5d14db34c2","analyzerSha256":"2d129177a1b8033f768aad3c82dc8fff70ba0ba76e7270b2d8612ee760d4aaf4"}}
```

</details>

- Latest options-struct refactor: 33 provider tests and 86 backend tests
passed. Four port-dependent fixtures initially failed inside the sandbox
and passed with loopback access. Latest check snapshot: 51 successes, 4
skips, and 2 failures (`ci / e2e` and `ci / e2e shard (5/8)`). This
latency experiment does not make the PR merge-ready.

- Focused TypeScript suites passed: OpenCode driver, per-turn reasoning,
prepared-context recovery, proxy envelope, live sessions, Codex context
control, and native instruction measurements. The real runnerd/proxy
integration test confirms disabled followed by default in the same
session. A stale staged binary initially hid the new parameter;
rebuilding the release runner resolved it.
- Capability-guard regression suites passed all 53 tests; the real
runner/proxy boundary still accepts the per-turn selection. Both
OpenRouter and non-OpenRouter real runner/proxy cases passed, along with
62 existing OpenCode fixture tests. Runner TypeScript build passed after
the final correction.
- Affected Rust modules passed: 33 provider tests and 86 durable backend
tests. Malformed or unsupported reasoning selections do not dispatch a
provider turn or poison admission state.
- Pinned OpenCode 1.18.34 with a local OpenRouter-compatible fixture
sent three successive provider requests with reasoning omitted,
`enabled: false`, then omitted. Each request contained one copy of the
agent instructions. No paid provider call was needed.
- The reconstructed first-turn task envelope shrank from 4,356 to 460
characters. This is a payload-size measurement, not a claimed token or
latency reduction. Existing stored conversation history is not
rewritten.
- Prior head `8c07fe2e70267a7c8a3118f1071eca73ecdc6c8a`: all 53 checks
passed, with four intentional skips. Greptile is 5/5 with no open
findings. One bounded retry cleared first-test timeouts, a legacy
signoff lease failure, and CI runner shutdowns. The timed-out suites
also passed locally (14 auth-signal tests, 40 device-login tests, and
the runner resume case).
- Known reported diagnostic usage totals about $0.0029. Two prematurely
closed baseline warm-turn attempts are excluded from latency comparisons
and their usage is not fully captured; background provider usage is also
not fully accounted. No further paid runs are active.
- `pnpm -r typecheck`: passed. Runner TypeScript typechecks and build
also passed after the review correction.
- `pnpm test:run`: attempted and stopped after a reproducible failure in
the untouched Codex recovery test `archives a damaged prior epoch before
completing guarded same-task replacement with real runnerd`
(`server/src/services/native-runtime/native-session-resume.test.ts:1072`).
It returns `needs_review` / failed instead of done / succeeded,
including when rerun alone. The broad local suite is not green. The two
initial collection failures disappeared after building; their isolated
rerun passed all 1,565 tests.
- `pnpm build`: passed.
- Pinned real OpenCode plus a delayed local provider: first native delta
to Paperclip delta fell from 1,496 ms to less than 1 ms at millisecond
measurement resolution. This isolates forwarding delay, not
network/model inference.
- Two live driver samples after the patch reached first text in
2.28–2.62 seconds including session startup. Baseline first responses
took 12.37–13.43 seconds. Small unmatched samples had different
reasoning/cache outcomes, so these are observations, not an attributed
speedup ratio.
- Full compiled runnerd protocol diagnostic: completed with `Hi 👋`;
first assistant delta at 9.19 seconds and cleanup complete at 9.44
seconds. The seeded mock-control-plane harness used 27,512 input tokens
across two provider requests, split into 13,521 and 13,991. The extra
`answer_status_question` call posted `Hi 👋` to the mock task in 12 ms,
then caused another model request. A reconstruction contains 25 tools
(35.3k JSON characters), 12.8k system characters, and a 4.4k task
envelope; 3.9k instruction characters appear in both system and task
context. Provider-reported cost was $0.00073976; the pricing-table
estimate was $0.00395332. This is one integration diagnostic, not a
production latency claim.
- A separate full-runner sample with existing lazy tool discovery also
completed, but took 14.64 seconds through cleanup and called
`paperclip_finish`. Its usage ledger recorded 12,888 input and 11,776
cached-input tokens, with $0.00096344 provider-reported cost. This did
not establish a latency improvement, so the default tool exposure was
not changed.
- Four direct OpenRouter requests separately measured default reasoning
at 2.45/3.90 seconds to first text and disabled reasoning at 0.66/0.53
seconds. Default reasoning remains unchanged; disabling it requires the
explicit per-turn option.

## Risks

- Immediate readiness dispatch changes scheduling timing. It uses
existing admission checks and preserves the periodic sweep as recovery.
- Native model verification relies on the real request rather than
catalog enumeration. The final live batch had provider startup failures;
overall speed and reliability are not qualified.
- Incremental chunks without IDs cannot safely be deduplicated by
content. Repeated identical chunks are valid. Known-ID duplicates remain
suppressed.
- OpenCode uses its bundled model catalog plus the explicit
selected-model entry. New remote catalog metadata will not refresh
during isolated startup.
- Default provider plugins are disabled. This runner supplies
credentials and assigned MCP tools through its own controlled
configuration.
- Disabled reasoning is supported only on the OpenCode/OpenRouter path
and requires a compatible model. It applies to one turn and is not an
automatic latency policy.
- The matched live browser sample is small and does not establish a
reliable end-to-end improvement. It includes local queueing/startup and
browser rendering, but does not qualify production or remote sandbox
latency.

## Model Used

OpenAI GPT-6 (Codex). Tool use, code editing, and command execution were
used. The session does not expose a more specific model ID or
context-window size. Live diagnostic target: DeepSeek
`deepseek/deepseek-v4-flash-0731` through OpenRouter and OpenCode
1.18.34.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-10 07:15:41 -05:00
DottaandPaperclip 35511366dc perf(server): schedule browser cleanup only while sessions exist (#15757)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Hosted instances must become quiet when they have no work.
> - Browser Use queried its session table every three seconds, even when
empty.
> - Existing session rows already store the next poll time and cleanup
lease.
> - This pull request uses commit signals and those saved deadlines to
schedule cleanup.
> - Empty instances stop browser database scans. Live sessions keep
their safety checks.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Browser Use background cleanup and recovery. This follows #15625 and
#15750.

**Subsystem affected**

server/ — REST API and orchestration services.

**Current behavior**

The app scans for due sessions every three seconds, including instances
that have never used Browser Use.

**Proposed behavior**

Scan once at startup. Wake after session creation, provider-create
settlement, stop requests, or credential-removal cleanup. Schedule the
next check from `nextPollAt` and `leaseUntil`. Stop scheduling when all
sessions are terminal.

**Reason and benefit**

Remove empty database traffic so idle stacks and their database
connections can become quiet.

**Breaking changes**

No API or schema change. Running and idle sessions retain provider
polling, cost accounting, access checks, expiry checks, and cleanup
retries.

## What Changed

- Register browser cleanup with the existing work coordinator. Remove
the app's three-second interval.
- Signal existing session transactions before writes. No new tables or
provider infrastructure.
- Restore deadlines from live sessions and their leases. Wake after a
credential-removal sweep releases its lease.
- Wait for concurrent cleanup instead of returning while it still runs.
Drain and standby preserve deferred work.
- Add scheduler tests and database-backed commit and restart coverage.
Update the connector guide and sleep inventory.

## Verification

- Passed: focused browser worker and shared coordinator tests, 17 tests.
- Passed: browser lifecycle, route-scope, viewport, and transport tests,
50 tests. The lifecycle suite uses real local Postgres and a simulated
provider.
- Passed: `pnpm -r typecheck` and `pnpm build`.
- Passed: all hosted test groups for this commit. Started full local
`pnpm test:run`, then stopped its duplicate serial run after the
complete hosted sharded suite passed; no full local-suite pass is
claimed.
- Passed: direct server TypeScript check. Greptile reviewed
`99c3217a2dea505be94081e729941e542e6fbc03` at 5/5 with no findings. All
54 applicable hosted checks passed; the two Storybook checks were
skipped because this PR has no UI changes.

## Risks

- Process-local signals cover services that share the instrumented
database owner. Independent processes and direct SQL writes require
startup recovery; this does not add cross-process notification.
- Live sessions still poll. This removes empty scans; it does not make a
running hosted browser sleep.
- Cleanup uses the existing provider protocol and lease fences. Tests
simulate the provider; no paid live browser was created for this
scheduling change.

## Model Used

OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The
exact deployment model ID and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-10 07:06:31 -05:00
Devin FoleyandPaperclip 7e10dbe652 fix(server): retain stopped ephemeral workspaces after failed copy-back (#15790)
## Thinking Path

> - Paperclip manages work performed by agents.
> - A remote adapter must copy workspace changes back after execution.
> - Cancellation can stop and release an ephemeral sandbox before
copy-back fails.
> - The recovery recorder previously considered only active or already
held leases.
> - This change carries the original host lease into the recorder and
checks both stop receipts.
> - The exact stopped source receives a board repair obligation without
replaying work.

## Linked Issues or Issue Description

**What happened?**

A legacy remote run can report `workspaceRestoreFailure` after its
dedicated stop has released the source lease. The recorder misses that
source and does not create its file-repair action. Related prior work:
#15637 adds bounded transfer diagnostics; #15718 protects native
accepted finalization. This change addresses the separate legacy
recording order.

**Expected behavior**

Preserve the exact eligible source and record the existing board repair
obligation. Preserve cancellation, later task progress, and newer
execution ownership.

**Steps to reproduce**

1. Acquire an ephemeral remote lease for a legacy run.
2. Complete the dedicated stop-and-retain lifecycle and persist both
stop receipts.
3. Report a workspace copy-back failure from the original adapter
invocation.
4. Observe that the old recorder omits the released source. The
PostgreSQL regression reproduces this order.

**Paperclip version or commit**

de042e79ef.

**Deployment mode**

Server with PostgreSQL and a sandbox provider. The regression uses
disposable local PostgreSQL and synthetic provider receipts.

## What Changed

- Clone the realized host lease before adapter dispatch and carry it
into restore recording.
- Admit a released source only when its original and current policies
are ephemeral, its full ownership and workspace identity match, and its
dedicated retention receipt agrees with the independent
stopped-execution receipt.
- Reject reusable, expired, destroyed, rebound, competing, or
insufficiently proved sources. Preserve release state and receipts when
recording the existing no-replay repair action.
- Cover the actual stop-before-copy-back lifecycle, cleanup retries,
built-in dispatch, rollback, repeated recording, newer owners, and
ownership/receipt failures. Document the narrow scope.

## Verification

- `pnpm exec vitest run
server/src/__tests__/legacy-workspace-restore-recovery.test.ts
server/src/services/legacy-execution-recovery.test.ts`: 80 passed.
- Existing Grok confirmed-stop/copy-back boundary test: passed.
- The new PostgreSQL regression fails on the baseline because the
recovery marker is absent.
- Guard-removal controls fail when either the ephemeral-policy gate or
original-source identity checks are removed.
- `pnpm --filter @paperclipai/server typecheck`: passed.
- Additional competing-owner/lifecycle checks: 6 passed, including older
stopped and destroyed allocation owners.
- `pnpm -r typecheck` and `pnpm build`: passed.
- Independent review approved the final source. The exact ephemeral
regression fails on the baseline; policy/identity guard-removal controls
fail their intended assertions.
- Compatibility with #15789 was checked in a disposable combined
checkout: both commit orders yield the same tree. The earlier broad run
passed 136 cases; the final guard/SDK delta passed 9 cases.
- Greptile: 5/5 on the current head, with no review threads.
- Exact-head CI: 53 successful checks and 2 expected skips; no failures
or pending checks. This includes all test shards and Canary Dry Run.
- The duplicate local `pnpm test:run` aggregate was stopped after full
CI passed. It is not reported as a completed local suite.
- No live provider calls, historical data repair, run replay, or
deployment was performed.

## Risks

No schema or API change. A qualifying source remains held for manual
file repair and eventual removal, which can retain provider storage
longer. This fix does not recover historical files or establish that the
provider still holds the bytes. Reusable sources remain excluded from
this new path because provider resume can start before the database
handoff. Existing active-source and previously recorded recovery
behavior is unchanged.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, and
independent agent review. The exact model revision and context-window
size are not exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-10 01:27:03 -07:00
Devin FoleyandPaperclip caa7895f9a fix(opencode): report timeouts without claiming deadline expiry (#15789)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The OpenCode adapter reports the outcome of a local or remote
execution.
> - A provider can report a timeout before the configured execution
deadline.
> - The adapter treated every timeout as proof that the configured
deadline expired.
> - This pull request uses a generic timeout message and retains the
resolved policy.
> - Operators can compare the run duration with its actual configured
limit without a false cause claim.

## Linked Issues or Issue Description

**What happened?**

A remote execution can return `timedOut: true` after a short interval
while the sandbox policy is four hours. OpenCode reported `Timed out
after 14400s`. It also dropped the resolved policy on that result. Run
finalization then used the raw `timeoutSec: 0` configuration, so
diagnostics incorrectly reported an unlimited timeout.

**Expected behavior**

Keep the timeout outcome. Report the resolved policy separately from the
failure message. Do not infer which timer fired from the provider's
boolean or stderr.

**Steps to reproduce**

Call the OpenCode adapter with a sandbox execution target and
`timeoutSec: 0`. Return a timed-out process result before four hours.
Inspect the message and the policy that finalization sends to the
run-failure collector. The new tests reproduce this with local fixtures
and an in-memory Sentry transport.

**Paperclip version or commit**

Reproduced on `de042e79efbb7bb88e9716835407882e3ebab188`.

**Deployment mode**

Legacy OpenCode execution on a local instance or remote sandbox.

Refs #14832 for related timeout result preservation; this PR does not
change session state. Related PRs #14964, #15659 and #15715 cover
silent-run limits, gateway requests and OpenCode v2 support. They do not
fix this policy/message path.

## What Changed

- Retain the complete existing timeout resolution on both OpenCode
result branches.
- Report `OpenCode execution timed out` without copying provider output
into exception text.
- Document that configured policy and the existing `timeoutFired`
outcome field do not prove timer origin.
- Test policy overrides, sandbox defaults, opt-out, ordinary exits, real
local timer expiry, cleanup and real Sentry SDK delivery.

## Verification

- 142 focused adapter, stop-metadata and diagnostic tests passed on Node
25.6.1.
- The real Sentry SDK contract suite passed 11/11 with the audited
optional peer 10.71.0 and an in-memory transport. The new test runs the
actual adapter, finalization policy merge and collector. It verifies
that raw HTML and credential canaries stay out of the event and
unrelated captures stay isolated.
- Full workspace `pnpm -r typecheck` and `pnpm build` passed. All hosted
CI checks passed on `f4df5cba1d2223d444559de960bbf88dfa985bbf` (54
passed, 2 expected skips).
- The local aggregate is not a pass: 15 existing PostgreSQL fixture
cases initially failed because this fresh macOS install lacked bundled
ICU symlinks. The dependency's existing hydration script repaired the
isolated install; all 15 affected cases then passed with real
PostgreSQL. No test or production source was changed for that setup
issue. The aggregate was stopped after hosted test coverage passed; it
is incomplete, not a local full-suite pass. All owned test processes
exited.
- Greptile scored this exact head 5/5 with no review threads.
Independent review found no issues. It ran 58 positive cases and adapter
typechecking. Three negative controls failed as expected when it
restored the old producer, removed timeout policy, or copied raw stderr
into the message.

## Risks

The timeout message changes for local and remote OpenCode runs. The
resolved policy is additive result data. The numeric timeout, grace
period, process result, retry rules, session handling, cleanup and
recovery behavior are unchanged. This fixes reporting and does not
identify or fix an upstream provider timeout. No schema change or
migration is required.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and
independent agent review. The exact service model identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-10 01:18:17 -07:00
32745f4e84 feat(issues): keep follow-up parents and show the creation source (#15762)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents delegate work by creating follow-up tasks through the issue
API, the CLI, and the chat `create_task` tool
> - An agent can omit `parentId` when it creates a follow-up from an
ordinary execution task. The API accepts the task as a top-level task.
The task hierarchy does not show where the task came from, even though
the run provenance is recorded
> - The operator must read run history to learn why the task exists.
This is slow and it hides delegated work from the parent task
> - This pull request keeps the parent link for delegated follow-ups by
default, keeps intentional standalone tasks possible, and shows an
access-checked "Created from <task> by <agent>" note on the task page
from the recorded run provenance
> - The benefit is that each agent-created task shows its source without
log inspection, while structural parentage, provenance, and blockers
stay separate concepts

## Linked Issues or Issue Description

Refs #13241 (shows tasks created from a task on the source task page;
this PR shows the reverse direction on the created task)
Refs #9727 (open proposal for creation provenance tracking)

**What existing behavior does this improve?**

Issue creation by agent runs, and the task detail page.

**Subsystem affected**

Server (`POST /api/companies/:companyId/issues`, `GET /api/issues/:id`),
shared types, UI task detail page, CLI `issue create`, Paperclip skill
docs.

**Current behavior**

When an agent run bound to an execution task creates a task and omits
`parentId`, the task is created with no parent. The task page shows no
source. The operator must inspect the run log to find the task that the
agent was working on. The source run and the privacy parent are
recorded, but the UI does not use them.

**Proposed behavior**

- When an agent run bound to an ordinary execution task omits
`parentId`, the server sets `parentId` to that task. The `issue.created`
activity records `parentDefaultedFromRunIssue: true`.
- An explicit `parentId: null` keeps the task standalone. Conversation
runs (chat handoffs), board or user creates, watchdog follow-ups, a run
task the agent may not mutate, a default parent that would cause a
delegation cycle, and a default parent the agent may not create a child
under (for example a protected assignment policy without a grant) also
stay standalone. The server runs the same `tasks:assign` decision with
the defaulted parent in scope before it accepts the default. No request
that succeeded before now fails. An explicit `parentId` is still checked
as before.
- `GET /api/issues/:id` returns `createdFrom` with the source task, the
run, and the agent. The server resolves it from `origin_run_id`, or from
the recorded creation activity for historical rows. It returns `null`
when the viewer may not read the source task or the run, when the run is
from another company, or when provenance is missing.
- The task page shows "Created from <task> by <agent>" in the classic
breadcrumb area and in the streamlined Tasks tab. The note is not a
blocker and it is shown with or without a parent.
- The CLI `issue create` command gets `--standalone`, which sends
`parentId: null`.

**Reason and benefit**

Operators see why a task exists on the task itself. Delegated follow-ups
stay inside the parent task tree, so parent completion and child
wake-ups keep working. Private and cross-company sources stay hidden.

**Breaking changes**

None for the API schema. Agents that create tasks from a run and relied
on the omitted `parentId` to mean "top-level" must now send `parentId:
null` (CLI: `--standalone`). The skill and the spec document this.

## What Changed

- `server/src/services/issue-creation-source.ts` (new): resolves a
task's creation source from the origin run or the creation activity,
scoped to the company, and loads the run-bound task by UUID or
identifier.
- `server/src/routes/issues.ts`: `POST /companies/:companyId/issues`
defaults `parentId` to the run's task when the body omits it (with the
exclusions above) and records the default in the activity log. `GET
/issues/:id` adds the access-checked `createdFrom` field.
- `packages/shared/src/types/issue.ts` and index exports: new
`IssueCreationSource` type and the optional `createdFrom` field on
`Issue`.
- `ui/src/components/task-detail/IssueCreatedFromNote.tsx` (new),
`TaskDetailTasksPanel.tsx`, and `IssueDetail.tsx`: show the creation
source in both task page layouts and count it in the Tasks tab badge.
- `cli/src/commands/client/issue.ts`: `issue create --standalone`.
- Docs: `doc/SPEC-implementation.md`, `doc/CLI.md`,
`skills/paperclip/SKILL.md`.
- Tests: `server/src/__tests__/issue-creation-source-routes.test.ts`
(new), `TaskDetailTasksPanel.test.tsx`, `IssueDetail.test.tsx`.

## Verification

Run the new server route test (embedded Postgres):

```sh
cd server && pnpm exec vitest run src/__tests__/issue-creation-source-routes.test.ts
```

It covers: ordinary delegation gets the parent and `createdFrom`;
explicit `parentId: null` stays standalone but still shows the source;
conversation handoff and board create stay standalone; a would-be
delegation cycle falls back to standalone; a protected source task
without a grant falls back to standalone while an explicit parent still
returns 403; an explicit cycle still returns 409; historical provenance
from the creation activity; missing provenance, a private source for an
outsider, and a cross-company run all return `null`.

Related existing suites that passed locally (11 files, 313 tests):
`issue-created-from-routes`, `issue-create-deduplication-routes`,
`issue-agent-mutation-ownership-routes`, `chat-project-tools`,
`issue-assignee-invokability-routes`, `issue-watchdogs-routes`,
`issue-dependency-wakeups-routes`, `public-mcp`,
`native-runtime/task-creation-backlog`, `issue-access-grants-routes`,
`issue-list-assignee-filter-routes`.

UI:

```sh
cd ui && pnpm exec vitest run src/components/task-detail/TaskDetailTasksPanel.test.tsx src/pages/IssueDetail.test.tsx
pnpm check:token-gates
```

Manual check: as an agent run bound to a task, `POST
/api/companies/:companyId/issues` with only a title. The response has
`parentId` set to the run's task. Open the new task; the page shows
"Created from <task> by <agent>".

The related server suites pass locally (`issue-created-from`,
`issue-create-deduplication`, `issue-agent-mutation-ownership`,
`chat-project-tools`, `issue-watchdogs`, `public-mcp`,
`task-creation-backlog`, `issue-access-grants`,
`permissions-upgrade-boundary`). The full `pnpm -r typecheck`, `pnpm
test:run`, and `pnpm build` were not run locally because the sandbox has
a 4 GB memory limit and the TypeScript compiler is killed on the server
project; the CI gates on this PR cover them.

## Risks

- Behavior shift for agent clients: a run-created task without
`parentId` now becomes a child of the run's task. Clients that want a
top-level task must send `parentId: null`. The watchdog, conversation,
and board paths are excluded, and the default never adds a blocker.
- The default parent inherits the parent's execution workspace, which
run-created tasks already inherited from the run's task.
- `createdFrom` adds one run lookup, one issue lookup, one access
decision, and one agent lookup to the task detail request. It is skipped
when the task has no origin run and no creation activity with a run.
- Privacy: the field returns `null` unless the viewer can read the
source task and the run, so a private source is indistinguishable from
missing provenance.

## Model Used

Claude Fable 5.1 (`claude-fable-5-1`), Anthropic, run through Claude
Code in a Paperclip-managed execution workspace with tool use, shell
execution, and a local embedded Postgres for tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-10 00:40:08 -07:00