0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-27 11:20:42 +01:00

Files

T

DottaandPaperclip 422287eecd fix: preserve runner recovery, warm sessions, and task outcomes (#13338 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner connects task messages, provider execution, and
task outcomes.
> - First-time user tests exposed gaps in recovery, completion
permissions, message delivery, and Stop behavior.
> - These gaps left usable output hidden, completed work waiting for
bookkeeping, or safe work unable to continue.
> - This pull request fixes the shared lifecycle and receipt paths while
preserving process ownership and action checks.
> - Users can continue work with accurate task state and durable
messages.

## Linked Issues or Issue Description

**What happened?**

A stopped local Codex execution could remain blocked even after its
processes had stopped and its complete transcript proved that no
external action needed replay. Claude under Conservative permissions
could fail to call task completion tools. Recovery could reuse an
assistant item ID and overwrite prior output. A delivered comment could
remain marked uncertain after navigation. Stop could look like Pause or
a new recovery incident. Workspace contention could look like
cancellation. A direct reply reopening Done could enter a clarification
loop.

**Expected behavior**

Recover automatically only with verified termination and complete action
receipts. Preserve answers and messages. Keep task completion available
under Conservative permissions without broad tool access. Show crashes
as Blocked, actual human decisions as In Review, and ordinary workspace
contention as waiting. Stop the current response and allow a new
direction.

**Steps to reproduce**

1. Create ordinary response tasks with local Codex and Claude Code, then
send follow-up messages through the task composer.
2. Interrupt a disposable local Codex runner during text-only work.
Verify automatic continuation and retained output.
3. Stop a response, send a new request, answer a clarification, and
reopen completed work with another message.
4. Navigate or reload while a comment submission is pending. Confirm the
exact persisted request receipt settles it without removing newer draft
text.
5. Run two tasks in a shared Daytona workspace. Confirm waiting does not
appear as failure.

**Paperclip version or commit**

Initial acceptance baseline: `c9021c6721f91e2c74bd9fee9d3fd41c999d17b7`.
Current integration base: `6cef9743c`. Both operator-interruption and
workspace-waiting guards are preserved; native restart and legacy
permission rules remain documented.

**Deployment mode**

An isolated source-built test-drive instance, with real local Codex and
Claude Code providers and disposable Daytona environments.

Related work: #13314, #13316, #13327, #13344, #13239, #13254, #13163.
This PR addresses additional failures from ordinary task journeys,
including controller restart handoff and repeated warm sandbox setup.
Historical task status reconciliation is excluded.

## What Changed

- Persist runner ownership immediately at spawn and resume an explicitly
adopted runner even when the controller crashed before the first driver
checkpoint. Detach the controller safely across graceful restarts,
including session startup. Prevent an old finalizer from suspending or
signaling an adopted runner. Checkpoint idle warm sessions before
shutdown. Preserve the same run and queued follow-up messages.
- Scope saved legacy queue successor checks to the queue owner while
preserving ordinary task locks, operator identity, assignment gates, and
exactly-once delivery.
- Preserve managed Codex credential files when an old session is
detached for restart; normal owned cleanup still copies refreshed auth
back and removes the scoped copy.
- Reuse the bound warm shared sandbox and fully verify an existing
staged provider pack before using it. This avoids repeated uploads when
the pack is already valid.
- Add a narrow local Codex replacement path with stopped-process proof,
a closed transcript inventory, exact completion receipts, and
fresh-session lineage. Preserve no-replay holds when evidence is
incomplete. Recovery may clear only the same run's recorded Blocked
status version; manual re-blocking and dependency changes invalidate
that receipt, while queued comments do not. Later blocks stop scheduled,
queued, and final dispatch; queued/final checks re-read dependencies
even when the task status stays In Progress.
- Permit only task delivery and human-input tools through the isolated
Claude runner's exact task bridge.
- Scope assistant item identity to the provider turn and ignore only
authority-free Codex skill-change notifications during startup.
- Reconcile composer submissions by client request ID across response
loss, navigation, and reload. Retain text typed during delivery.
- Keep acknowledged run-only Stop neutral and show workspace contention
as waiting. Project exhausted native failures as Blocked.
- Restore the guarded task-page retry action for failed legacy runs,
including the server-supported explicit new-attempt path for stopped
conversation adapters. Preserve native/process recovery holds and avoid
promising Retry while a decision or execution gate hides it.
- Refresh delivered artifacts and handle direct user replies that reopen
completed work without a clarification loop.
- Check the embedded PostgreSQL PID, data directory, and actual port
before connecting or migrating.
- Document accepted behavior and add focused regressions at lifecycle,
route, transcript, and UI boundaries.

## Verification

- Final head `fece606ac2` passes the complete GitHub CI matrix: **34
green checks, two expected Storybook skips, no failures or pending
checks**, including `ci / verify`, `ci / e2e`, full runner verification,
typecheck, build, every server/workspace shard, and all browser shards.
[CI
run](https://github.com/paperclipai/paperclip/actions/runs/34727183287).
Greptile is **5/5 with no open findings**. The final two commits only
refine test fixtures; both affected suites pass 24/24 locally and in CI,
with server typecheck green.
- Complete local Vitest coverage uses the canonical groups/shards: all
635 general server suites, all 145 serialized suites, and all workspace
packages. The aggregate began on `0a8001c18` while the final queue fix
arrived: 23,903 passed, five failed, 87 skipped. The five
port/socket/timing failures passed unchanged in follow-ups (60 tests in
the exposure/file suites and 412 tests covering the serialized failures
and unrun tails). The final queue/operator-identity suites separately
passed 52/52. This is aggregate coverage plus explicit reruns, not a
pristine single-command final-head run.
- After integration with current master,
queue/operator-identity/continuation suites passed 162/162 and affected
UI suites passed 140/140. ACP Stop/continuation and legacy
task/Inbox/message browser suites passed 9/9, including both task
recovery Retry and thread Try again, automatic saved-message delivery,
exactly one new run, Done, and retained output after reload. The default
process Stop/Pause/Resume browser case passed (the native-provider case
is opt-in and skipped by default). The complete Board attachment/receipt
browser suite passed 11/11 on a disposable instance, covering both
composers, exact receipts after lost responses, no replay, bound
attachments, and newer drafts after reload.
- Blocking-intent regressions cover pre-existing Blocked, a mismatched
run/cause, an explicit manual re-block, changed dependencies, a queued
comment after failure, and a block arriving between scheduling and
provider dispatch. The negative cases reproduced before the fix. All 478
affected executor/recovery/dispatch tests passed; both database suites
ran separately after availability-probe skips in the first combined
command. The final late-dependency check passed all 143 affected
recovery/dispatch tests (zero skips) after two new negative cases
reproduced the bug.
- Focused runtime regressions cover awaited runner ownership
publication, authenticated adoption before the first checkpoint,
old-finalizer detachment, idle and busy warm-session shutdown, rejected
checkpoint propagation, provider-pack verification, and managed-Codex
credential preservation. Four managed credential detachment cases
reproduced the bug before the fix; normal owned cleanup still succeeds
exactly once.
- Live local Claude: SIGKILL 2.6 seconds into startup recovered the same
run automatically in 53 seconds, then a normal follow-up completed in 24
seconds. SIGTERM 2.5 seconds into startup preserved the same run (54
seconds) and its queued follow-up (21 seconds). Answers remained visible
and the task reached Done.
- Live Claude Daytona: a warm follow-up retained its sandbox and fell
from 121 seconds to 44 seconds. A separate cold turn took 127 seconds;
after controller shutdown and checkpointing, its follow-up completed in
33 seconds with the same sandbox, workspace, native session, and runner.
Both answers remained visible and the task was Done.
- Other live journeys covered task completion and follow-up with local
and Daytona Codex, local Codex crash recovery, Stop then new direction,
clarification response, live artifact refresh, and shared-workspace
waiting.
- Validation limits: the opt-in native composer Stop/Pause→subtree
Resume fixture exposes terminal/result ordering and subtree-cancellation
attribution bugs that can leave a child task blocked; that new finding
is assigned to a separate follow-up and is not claimed fixed here.
Default CI skips this optional native-provider fixture. Managed-Codex
credential handoff and the queue-agent integration use automated
regression evidence. Cold custom provider-pack uploads still add startup
latency.

## Risks

- Automatic replacement remains deliberately narrow: local Codex,
verified stopped identities, unchanged retained state, and a complete
text/completion-only turn. Unknown actions, partial history, or changed
ownership remain blocked.
- Claude completion permission handling changes an upstream package
patch. The exact isolated task bridge must remain pinned; unrelated
tools keep their existing permissions.
- New task failure projection changes user-visible status. No historical
status backfill or database migration is included.
- This is a broad lifecycle fix across server and UI. Live proof covers
graceful local Claude restart during startup and idle Claude Daytona
session recovery across controller shutdown. Live abrupt SIGKILL during
local Claude startup also recovered the same run. Unknown ownership or
missing action evidence still blocks reuse. Cold custom provider-pack
uploads still add startup latency; this change avoids unnecessary repeat
uploads.

## Model Used

OpenAI GPT-6 (Codex), with reasoning, code execution, browser
automation, and tool use. The exact hosted model ID and context window
are not exposed in this task.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-09-12 19:41:15 -05:00

8.9 KiB

Raw Permalink Blame History

Run-Log Events

Run-log events write to the heartbeat_run_events table (packages/db/src/schema/heartbeat_run_events.ts:6-20). They are not Paperclip Telemetry events, and they are not OpenTelemetry exports. A run-log event needs no operator endpoint.

Native PRP Run-Log Events

The hidden native coordinator writes each validated PRP event to the bound run's existing event stream before it acknowledges the runner. The row keeps the PRP eventType, source instance, source event ID, source sequence, protocol schema version, and a SHA-256 digest of the canonical source envelope. Its payload is { "prpEvent": <canonical PRP event> }.

PostgreSQL JSONB cannot represent NUL (U+0000), which can occur in command output such as Vite virtual-module paths. The run-event payload column uses a lossless storage codec for these events: the JSONB projection renders NUL as the literal \u0000, and the reserved $paperclipRunEventJsonV1 field contains the original serialized JSON as a doubly escaped string. Ordinary payloads retain their existing representation. Drizzle reads restore the exact original payload before replay, hash validation, redaction, or API presentation. SQL queries can still inspect ordinary routing fields in the projection; raw SQL readers of the whole payload must apply decodeRunEventPayload. The column remains JSONB and requires no schema migration.

The writer locks the native heartbeat_runs row and allocates the existing per-run seq cursor. A byte-equivalent retry reuses the first row; a changed retry or source-sequence gap is rejected. Company, issue, agent, run, session, and runner-source bindings must match the persisted native run. Bootstrap tickets, reconnect leases, authentication proofs, encryption keys, and raw credential material are never written to the run log.

These records remain run-log events. They do not create an OpenTelemetry or Paperclip Telemetry export, and legacy adapters do not use this writer.

Native Restart Recovery Run-Log Event

Paperclip writes a native.recovery.transition event for every native restart classification and for graceful restart suspension. This immutable run-log record lets operators reconstruct recovery decisions without exporting data to Paperclip Telemetry or OpenTelemetry.

The payload contains the restart kind, recovery request id when one exists, runner disposition, and the controller generation and provider attempt for a claimed recovery. Live-runner adoption also records the runner PID, process group, and process-start fingerprint. A non-claim disposition records a bounded reason instead. Graceful suspension records the signal and confirms that it did not create a retry run.

The event never includes bootstrap tickets, reconnect leases, authentication proofs, encryption keys, environment variables, provider credentials, command arguments, or an unsanitized stderr stream. Detailed failed-attempt diagnostics remain in the bounded native_run_finalizations.recovery_history ledger.

Native Local Process Stop Evidence

The server writes native.local_process_stopped in the same transaction that clears a local run's process identity, after it verifies that its PID and process group are absent. The payload contains only those process IDs. Remote process IDs are never checked against the control-plane host.

The server writes native.process_start_requested before a backend can spawn, and native.process_identity_recorded when it stores a new native process identity. Either invalidates an earlier local stop receipt, including a crash before the new PID callback. Continuation admission accepts only the latest server-authored event among these three types; provider source events cannot supply stop authority. These records stay in the local run log and do not add Telemetry or OpenTelemetry data.

Verified Local Codex Replacement Evidence

The server writes native.stopped_text_turn_verified in the same transaction that schedules a fresh successor for a stopped local Codex run. It records the runner and provider process identities, retained-state digests, provider thread and turn IDs, and IDs of exactly receipted task-completion calls. The server first checks the complete turn inventory, process-stop receipt, and execution binding. Unknown actions or changed retained state prevent this event and replacement.

The record documents why the old execution can be retired. It does not make the old session resumable, rewrite provider files, or authorize replay on its own. It remains in the local run log and adds no Telemetry or OpenTelemetry export.

Sandbox Startup Run-Log Event

Paperclip writes one run.startup.step event to the run log for each bring-up step. This event is a run-log record, not a first-party telemetry event. The generated telemetry contract does not cover it, so this section is its canonical contract.

The event payload carries only three fields.

Field	Type	Meaning
`step`	string	The bring-up step name, for example `stage.sync`.
`durationMs`	number	The wall time of the step. A skipped step reports `0`.
`outcome`	string	The step outcome (`ok`, `skipped`, or `failed`).

The event no longer carries the per-step round-trip count or the provider duration fields. It dropped roundTrips, providerExecMs, providerGetMs, createRuntimeMs, and ensureSessionMs. The startup spans in doc/observability.md carry that detail now. The sandbox.exec child spans hold the round-trip and provider durations. The acp.handshake step span holds the create-runtime and ensure-session sub-times.

To read the detailed timing, use the startup spans. The spans need an OTLP endpoint. A run with no endpoint keeps only the three run-log fields above.

Run Phase Timing Run-Log Event

Paperclip writes one run.phase.timing event to the run log for each run-lifecycle phase. This event is a run-log record, not a first-party telemetry event. The generated telemetry contract does not cover it, so this section is its canonical contract. The producer is emitRunPhaseTiming in packages/adapter-utils/src/acpx-engine/startup-timing.ts.

The event payload carries only three fields.

Field	Type	Meaning
`phase`	string	The run-lifecycle phase name from the closed allowlist below.
`durationMs`	number	The wall time of the phase. A negative or a non-finite value clamps to `0`.
`outcome`	string	The phase outcome (`ok` or `failed`).

The phase field is one member of a closed, low-cardinality allowlist. The producer drops any event whose phase name is outside this allowlist, so a free-form label never reaches the run log. The allowlist has twelve phase names.

Phase	Meaning
`place_workspace`	Place the run workspace.
`start_transport`	Start the agent transport.
`create_runtime`	Create the agent runtime.
`ensure_session`	Ensure the agent session exists.
`configure_session`	Configure the agent session.
`prepare_turn`	Prepare the turn.
`turn`	Run the turn.
`end_session`	End the agent session.
`settle_reuse`	Settle the session for reuse.
`stop_transport`	Stop the agent transport.
`sync_back`	Sync the workspace back.
`release_staging_lease`	Release the staging lease.

The payload never carries a command, an argument, a path, an environment value, or a raw identifier. The event rides the ctx.onEvent run-event bridge and is run-log-only. It needs no OTLP endpoint.

The sandbox duplex transport also writes one run-log event as one of its three sinks. See the Sandbox Duplex Transport Instrumentation section in the Observability contract.

Execution recovery

Provider identity diagnostics remain in the local run log. They record the notification method, expected and received thread/turn identifiers, and the classification (root, verified descendant, stale, unrelated informational, or invalid authoritative). They omit the original provider payload and credentials. Repeated informational notices are bounded.

Recovery lifecycle events retain the original structured failure code, retry attempt, next retry time, and predecessor/successor identifiers. Durable status delivery uses an idempotency marker; delivery grants no provider authority. Failed publication is retried without repeating provider work. These records are not first-party Telemetry.

Codex resume usage snapshot

The native runner retains a bounded local harness.diagnostic event with code codex_resume_usage_snapshot. It identifies thread/tokenUsage/updated as resume_usage_snapshot, retains the reported thread and completed-turn IDs, and records cumulative usage counters. It does not include provider credentials or message content. The event establishes the accounting baseline; it is not a new billable usage receipt or a user-facing provider warning. Other provider identity checks remain in force.

8.9 KiB Raw Permalink Blame History