mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call Paperclip tools through the Runner. The Runner records
each call and returns the controller's result to the provider (here
OpenCode)
> - In chat-eval cohort 13, a `list_workflows` result was recorded but
never reached OpenCode. OpenCode waited about 290 s, and the call stayed
pending
> - The next run's `run.attach` then failed with
`durable_pending_tool_calls`, so the person's follow-up question never
got an answer
> - This pull request fixes the result shape that caused the rejection,
makes the Runner settle every recorded result, and lets the next run
settle a call that a closed turn left pending
> - The benefit is that a tool result always reaches the agent, and a
stuck call can no longer block every later turn
## Linked Issues or Issue Description
**What happened?**
Chat-eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`). In
`cases.case-history` and `cases.create-case`, the CEO called
`list_documents` (or `list_projects`) and `list_workflows` (call id 5)
together. The server answered `GET /companies/{id}/workflows` with 200
in 584 ms. The OpenCode MCP bridge diagnostic for call 5 says `outcome:
"unanswered"`, `written: false`, `handlerMs: 289038` (290723 in
create-case). The sibling call was answered in about 1 s. The run ended
with `runner_settlement_incomplete`: `semanticTools.pending` held call 5
and its `semantic_tool.result` delivery command had status `failed`. The
next question started a run whose `run.attach` was refused: `run.attach
requires a settled Codex provider session: durable_pending_tool_calls`.
The run failed in 17 s, and the question was not answered.
Cause: `executeNativeWorkflowTool` returned the Runner API result
unchanged: `{ ok: true, status: 200, operationId: "GET
/api/companies/{companyId}/workflows", ... }`. Every other API-backed
tool renames that field to `apiOperationId`. The Rust tool bridge
(`semantic_response_value`) reads an object with a boolean `ok` and an
`operationId` as a semantic envelope that must name this tool and call.
The REST route did not match, so `apply_result` failed with `semantic
result envelope does not match its provider call`. Nothing was sent to
the provider, and the call stayed pending forever. The red test
reproduces this exact error. The parallel call is not the trigger: the
sibling call settled normally, and the OpenCode MCP bridge answered
other parallel pairs in the same runs. `list_workflows` first appears in
cohort 13, so cohorts 11 and 12 did not hit it. All four workflow tools
had this shape.
**Expected behavior**
Every recorded tool result reaches the provider. When a recorded result
cannot be delivered, the provider gets an error at once and the call
settles. A call that a closed turn left pending does not block the next
run.
**Steps to reproduce**
1. Run an OpenCode agent with Cases enabled on the durable Runner.
2. Ask a question that makes the agent call `list_workflows`.
3. See the call wait about 290 s. Then ask a follow-up question and see
`run.attach` fail with `durable_pending_tool_calls`.
**Paperclip version or commit**
`master` at `df1b12ad8e`.
## What Changed
- `server/src/services/native-runtime/native-workflow-tools.ts`:
workflow tool results name the route `apiOperationId`, like case tools
and `call_api`.
- `provider_bridge.rs`: add `settle_undeliverable_result`. When the
bridge rejects a recorded result for a call it still holds, the call
settles with a bounded, non-retryable runner error. Conflicting or
unknown results still fail as before.
- `provider_backend.rs` (`deliver_semantic_result`): use that
settlement, deliver the error to the provider, and record a
`semantic_tool_result_undeliverable` diagnostic with the call,
operation, reason and the rejected result's digest.
- `provider_bridge.rs` and `provider_backend.rs` (`attach_run`): add
`expire_closed_turn_calls`. When nothing is live (no active or ambiguous
turn, no provider request waiting), `run.attach` settles calls that a
closed turn left pending as an expired failure with an unknown outcome,
and records `semantic_tool_result_expired`. The provider stopped waiting
when its turn ended, and the new run's controller never delivers the old
run's results. A call that a live turn still waits on still blocks
attachment. This follows the existing `tool_call_interrupted_by_restart`
rule: never execute the operation again, and record an unknown outcome
when no result can ever arrive.
- Docs: `doc/run-log-events.md` and
`packages/paperclip-runner/docs/durable-recovery.md` describe both
diagnostics.
## Verification
- Red/green, Rust (`cd packages/paperclip-runner/runner && cargo test -p
paperclip-runner-core --test codex_provider --
a_rejected_recorded_result run_attach_expires`):
- Before:
`a_rejected_recorded_result_still_answers_the_provider_and_frees_the_session`
fails with `semantic result envelope does not match its provider call`
(the production error).
`run_attach_expires_a_call_left_pending_by_a_closed_turn` fails with
`run.attach requires a settled Codex provider session:
durable_pending_tool_calls` (the production refusal).
- After: both pass. The fake provider logs `tool-response:failure`, so
it got an answer at once. The next `run.attach` succeeds.
- Parallel calls (`cargo test -p paperclip-runner-core --lib --
provider_bridge`):
`a_rejected_result_in_a_parallel_pair_still_settles_its_own_call` sends
the cohort 13 pair (`list_documents` + `list_workflows`). The rejected
call settles, its sibling keeps its own result, and the next turn
starts. `only_a_closed_turn_expires_its_pending_calls` checks that an
open turn expires nothing.
- Upgrade and mid-run cutover:
`run_attach_expires_a_call_left_pending_by_a_closed_turn` saves state
with a pending call after the turn ends, stops that Runner, and attaches
with a new Runner process from the saved state.
`durable_backend_settles_pending_tools_when_recovery_finds_the_turn_ended`
(unchanged) still passes: a late result in the same run still settles as
`settled_after_turn`.
- Red/green, server (`cd server && npx vitest run
src/services/native-runtime/native-case-tools.integration.test.ts -t
workflows`): before, `expected { ok: true, status: 201, … } to not have
property "operationId"`; after, 2 passed.
- `cargo test -p paperclip-runner-core`: 643 lib tests pass. One lib
test (`fenced_receipt_output_keeps_prefix_when_stdout_never_closes`)
failed once with `EPERM` while signaling a process group under parallel
load. It passes alone 3 of 3 and does not touch this change. The
integration test binaries pass.
- Paid narrow run for `cases.case-history` and `cases.create-case`: see
the comment below.
## Risks
- Previous-release story: no persisted shape changes. State saved by the
previous Runner with a call pending after its turn closed carries
forward. The new Runner settles that call at the next `run.attach`, and
the session continues (tested from saved state with a new process). A
previous-release Runner that receives the corrected `apiOperationId`
result delivers it normally, because the result is no longer read as an
envelope.
- Behavior change: a call left by a closed turn is now settled at the
next run's attachment, not kept forever. Its outcome is recorded as
unknown, and the operation is never run again. A provider turn that is
still live keeps the old refusal (`durable_pending_tool_calls`,
`provider_side_effect_pending`), and the controller still never replaces
the session.
- A recorded result that the bridge rejects now becomes an error for the
agent, not an endless wait. The agent is told the action may have
completed and must check the current state before repeating it.
- Latency: nothing is added before turn start. The attach-time check
walks the in-memory pending calls (usually none) and adds no I/O beyond
the existing state save. The rejected-delivery path now answers in the
same command instead of after the provider's ~290 s client timeout, and
the follow-up run no longer fails after 17 s.
## Model Used
- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #21398 (first release where `list_workflows` was called in this
eval), #19733 (the attach refusal), #20886 (earlier OpenCode delivery
fix).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1264 lines
77 KiB
Markdown
1264 lines
77 KiB
Markdown
# Run-Log Events
|
||
|
||
Run-log events write to the `heartbeat_run_events` table
|
||
(`packages/db/src/schema/heartbeat_run_events.ts:6-20`). They are not
|
||
Paperclip Telemetry events, and they are not OpenTelemetry exports. A run-log
|
||
event needs no operator endpoint.
|
||
|
||
## ACPX sidecar shutdown
|
||
|
||
The ACPX sidecar emits `runtime.diagnostic` with code
|
||
`sidecar_shutdown_requested` once at the first local shutdown request, before
|
||
waiters are removed. Its JSON message records the local reason, pending tool,
|
||
input and permission counts, completion-normalization count, and at most 16
|
||
oldest pending tools. Each tool sample contains elapsed milliseconds and a
|
||
SHA-256 call ID. Hash the corresponding `runtime.tool_called` call ID to match
|
||
it. Arguments, results, operation names and raw call IDs are excluded.
|
||
The existing diagnostic path also writes this message to sidecar stderr.
|
||
A controller that has already stopped may not ingest the frame. Stdin EOF
|
||
identifies a closed controller pipe, not the upstream reason for that closure.
|
||
This is local run evidence; no first-party Telemetry event or OpenTelemetry
|
||
span is added. Retry, cancellation and authorization behavior stay unchanged.
|
||
|
||
## Startup reuse evidence
|
||
|
||
`run.startup.reuse` uses schema `paperclip.startup-reuse.v1`. Each receipt names
|
||
one layer: `environment` (the lease), `workspace`, `runner`, `provider_process`,
|
||
or `provider_session`. Its state is `reused`, `fresh`, `unknown`, or
|
||
`not_applicable`. Reasons use a closed set: `verified_reuse`, `created`,
|
||
`replacement`, `not_found`, `expired`, `identity_mismatch`, `resume_failed`,
|
||
`configuration_changed`, `tools_changed`, `workspace_changed`, `cache_miss`,
|
||
`not_applicable`, and `evidence_missing`. No paths, identifiers, configuration
|
||
values, or exception messages enter these fields.
|
||
|
||
A restart's reattachment to a remote Runner resumes the run's own retained
|
||
sandbox lease and never acquires another one. Its `environment` receipt is
|
||
`reused` / `verified_reuse`. Earlier releases reported the lease's original
|
||
acquisition there, so a reattachment to a sandbox the run created showed
|
||
`fresh` / `created` although no sandbox was created.
|
||
|
||
Provider-process receipts carry a cumulative `providerLaunchCount` for the run.
|
||
`null` means unobserved; zero requires a physical launch observer. Direct ACPX
|
||
processes count initial launches and respawns, including launches during retries.
|
||
On the remote process-session bridge, ACPX observes the host relay rather than
|
||
the sandbox provider. Provider-process reuse and launch counts stay unknown on
|
||
that transport. Native Runner session events establish session continuity, but
|
||
do not establish provider-process reuse or its launch count. Those values stay
|
||
unknown without physical evidence. The final ACPX receipt does not wait for
|
||
persistence, so a stalled diagnostic sink cannot hold the staging lease.
|
||
The existing successful `runner.process.launch` performance receipt establishes
|
||
a fresh Runner process, independently of provider-session continuity.
|
||
|
||
The Product E2E result stores `startupReuse` by run, and its dashboard displays
|
||
each layer, its reason, and the launch count. `warm` requires all applicable
|
||
layers to be reused and zero observed provider launches. `cold` requires all
|
||
applicable layers to be fresh. Some reuse with other fresh or missing evidence
|
||
is `partial_reuse`; otherwise the label is `unknown`. Fresh decisions dominate
|
||
later reuse receipts within a run. Legacy runs without receipts retain unknown
|
||
layers and a null launch count. Session IDs and follow-up turns never establish
|
||
warmth. These events stay in the local run log; they add no first-party Telemetry.
|
||
|
||
## Ordinary event persistence
|
||
|
||
Events without a native source identity or retry-exhaustion receipt use one
|
||
atomic PostgreSQL statement. The statement checks the stored run's company and
|
||
agent, allocates its next sequence, and inserts the event. An insertion failure
|
||
also rolls back the sequence allocation. The returned row uses the existing
|
||
column decoders, including the lossless payload codec. Callers still wait for
|
||
the durable write before they publish or acknowledge the event. Native-source
|
||
replay and retry-exhaustion deduplication retain their existing transaction path.
|
||
|
||
## Legacy case handoff cleanup
|
||
|
||
`legacy.case_handoff_cleanup_intent` records local cleanup ownership before
|
||
dispatch, then binds it atomically to the observed process identity on spawn.
|
||
It includes the host, controller/run generation, PID/group/start identity, and
|
||
owned scratch descriptor or explicit `null`. It does not prove termination.
|
||
`legacy.case_handoff_cleanup_verified` records the later stop proof before
|
||
scratch removal. Restart recovery rechecks process, controller, workspace and
|
||
ownership gates before completing cleanup and releasing the handoff.
|
||
These controller receipts are cleanup bookkeeping, not evidence of agent task
|
||
progress, and are excluded from run-liveness action counts.
|
||
The controller authors both events on the system stream. Adapter ingestion
|
||
rejects the reserved `legacy.case_handoff_` namespace. These events stay in the
|
||
instance run log and add no first-party Telemetry or OpenTelemetry export.
|
||
|
||
## Provider-pack preparation timing
|
||
|
||
Every `run.performance.span` payload also carries `wallStartedAtMs`, the span's own
|
||
start time in epoch milliseconds (since 2026-11-09). Span rows can be written late
|
||
through deferred diagnostics, so readers take a span's start from this field (or
|
||
from the trace start plus `startOffsetMs`), never from the row's write time.
|
||
The preparation spans of a native run (`heartbeat.queue`, `heartbeat.prepare_*`,
|
||
`environment.*`) are always written this way: in the order they ended, while the
|
||
provider starts, and before the run's trace finishes. They can appear after the
|
||
first Runner events of the run.
|
||
|
||
`run.performance.span` also records `task.time_to_first_token` once per native
|
||
dispatch at the first newly committed deliverable text: assistant or reasoning
|
||
`item.delta`/`item.completed`, a redaction tail released at completion, or an
|
||
accepted run-result summary. Its
|
||
`durationMs` starts at the earliest accepted input. The accepted comment or
|
||
answered question is the start only when its span is the earliest preparation span.
|
||
A comment coalesced into an already-queued run starts after run creation, so that
|
||
run uses queued run creation. The durable wake request's `requested_at` replaces
|
||
the start when it is earlier (deferred, coalesced and retried wakes). Without
|
||
preparation history, the start is dispatch and the anchor is `run_created`. It ends at Core commit callback. `anchor` names the start:
|
||
`comment`, `question_response`, `wake_request` or `run_created`. `startOffsetMs` is
|
||
zero, or negative when a wake request precedes the run trace. `eventKind` is
|
||
`agentMessage` or `reasoning`. The wake request time is read after the first text,
|
||
in detached diagnostic work bounded to 2 s (a
|
||
transaction-local `statement_timeout`, so PostgreSQL cancels a stalled read), so the measurement adds no wait before
|
||
`turn.start` and never delays event delivery. The span can appear after later
|
||
run-log rows; the trace settles it before `task.provider_session.measured`. It includes preparation and startup, but excludes browser
|
||
delivery/rendering. This measures text availability, not an exact model token.
|
||
Empty text, item starts, tools, usage, result proposals, and duplicate replay do
|
||
not qualify. No deliverable text means no timing sample. A recovered dispatch can
|
||
record newly committed output from the original start; it does not reproduce
|
||
earlier output timing. The event contains no text or provider identifiers.
|
||
|
||
## Speed budget breaches
|
||
|
||
`run.performance.budget_exceeded` (schema `paperclip.run-performance-budget.v1`,
|
||
level `warn`) records one native run that exceeded its p95 speed budget. The
|
||
budgets come from the Phase 0 local baseline and are listed in
|
||
`doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` ("Speed budgets").
|
||
`metric` is `time_to_first_activity` (`task.time_to_first_agent_activity`:
|
||
the first text delta, tool-call start or reasoning item; named `ttft` and
|
||
judged on `task.time_to_first_token` before 2026-11-08), `agent_boot` (end of
|
||
`heartbeat.queue` to the end of `runner.turn.submit` or `runner.session.resume`)
|
||
or `restart_recovery_control_plane` (startup recovery `ready` to the start of the
|
||
provider's saved-session load, `provider.session.load`, or to the recovered run's
|
||
first provider event when nothing reloads). That is Paperclip's own share of a
|
||
recovery: claim, prepare, runtime stage and runner launch. The full recovery
|
||
(`ready` to the first provider event) and the provider's session load grow with
|
||
session size and with how many sessions resume at once, so they are reported in
|
||
the payload as `restartTotalMs` and `providerLoadMs` and are not judged; events
|
||
written before 2026-11-09 judged the full latency as `restart_recovery`.
|
||
The payload names `adapter` (`codex`, `claude` or `opencode`),
|
||
`environment` (`local`), `startKind` (`cold`, `warm` or `unknown`; an unknown
|
||
start uses the looser budget), `measuredMs`, `budgetP50Ms`, `budgetP95Ms`,
|
||
`percentile` (`p95`), `baseline`, and, for restart recovery, `disposition`.
|
||
For `time_to_first_activity`, `measuredMs` counts from run creation: when the input waited for its
|
||
run to exist (for example a comment posted while the previous turn was still
|
||
running), that wait is excluded from the judgment and reported as `inputWaitMs`.
|
||
Each metric is checked at most once per dispatch. A restart-recovery dispatch is
|
||
judged only on `restart_recovery_control_plane`, because its first activity and boot span the outage. Time to first text (`task.time_to_first_token`) is reported without a budget. Remote runs and other adapters have
|
||
no budget yet. The event is written in detached diagnostic work after the
|
||
measurement, so it adds no wait before `turn.start`. It is a run-log row only
|
||
and never goes to Telemetry.
|
||
|
||
The existing `run.performance.span` event also records `provider_pack.select`,
|
||
`provider_pack.discover`, `provider_pack.verify`, `provider_pack.verify_preinstalled`,
|
||
`provider_pack.link_preinstalled`, `provider_pack.clear_stale`, `provider_pack.upload`,
|
||
and `provider_pack.verify_uploaded`. Selection records `stateSource` as `staged`,
|
||
`preinstalled`, or `uploaded`. Its closed reason is `verified_pack_reused` or
|
||
`cached_packs_unverified`. A verification failure describes an unusable candidate;
|
||
it does not identify a particular compatibility mismatch or a failed run.
|
||
Harness startup also records `harness_state.inspect`, `harness_state.path_check`,
|
||
and `harness_state.launch_material` for remote state inspection and fresh launch
|
||
material. These events retain the existing schema and carry no paths or exception strings.
|
||
They also become native Observability spans only with an operator OTLP endpoint.
|
||
|
||
## Local runner startup sub-steps
|
||
|
||
`run.performance.span` splits a local Runner startup (`runner.session.bootstrap` or
|
||
`runner.session.resume`) into three sequential children of `runner.session.startup`:
|
||
|
||
- `runner.runtime.stage`: controller-side launch preparation before the Runner
|
||
process exists (runtime context, provider home, launch profiles; on resume, the
|
||
durable-state checks and authority rotation as well).
|
||
- `runner.process.launch`: from spawning `paperclip-runnerd` until its first
|
||
authenticated connection to the PRP control plane. An exact adoption of a live
|
||
Runner records no launch, so this span keeps meaning a fresh Runner process
|
||
(see startup reuse above).
|
||
- `provider.session.load`: from that authentication until the provider session
|
||
identity is confirmed. It covers `run.prepare` and `session.open`, or on resume
|
||
the `run.attach` that restores the provider and the recovery probe.
|
||
|
||
Each carries `mode` (`bootstrap` or `resume`), `target: "local"`, and `outcome`
|
||
(`failed` when startup fails inside the step). The Runner transport reports each
|
||
milestone synchronously; the controller writes the row as deferred diagnostic
|
||
work, so the run-log write never delays `turn.start`. Remote launchers keep their
|
||
own `runner.runtime.stage` and `runner.process.launch` spans. These rows stay in
|
||
the instance run log; with an operator OTLP endpoint they are also Observability
|
||
spans. They are not Telemetry.
|
||
|
||
## Native PRP Run-Log Events
|
||
|
||
Provider startup command failures and failed provider turns can carry a
|
||
`providerFailure` cause with schema `paperclip.provider-failure.v1`. The cause
|
||
retains a normalized category, a bounded and redacted provider explanation, the
|
||
startup/turn phase, and available status, code, request ID, or retry time. Provider
|
||
adapters extract these fields; arbitrary exception objects, request bodies,
|
||
headers, environment variables, and stacks are excluded. Unknown categories retain
|
||
their explanation. The cause stays in company-scoped run records and the run log;
|
||
it is not added to first-party Telemetry, Observability attributes, or Sentry
|
||
failure reports. The reportable run error keeps its existing generic text, except
|
||
that an ACPX startup rejection keeps Runner's closed rejection text and its
|
||
classification. Startup receipts retain the cause across controller restart before
|
||
any provider turn exists.
|
||
|
||
When a failed ACPX control closes the session before polling, the resulting
|
||
`turn.failed` event retains that control's closed-code error. A later closed-session
|
||
check does not replace it, and a closed code does not infer a provider explanation.
|
||
The terminal still reports `recoverable: false` and no observed provider terminal;
|
||
it does not prove termination or authorize replay.
|
||
|
||
Every ACPX session ensure rejection carries a startup cause. When the provider
|
||
returned no ACP error, the cause keeps the rejection's own redacted message with
|
||
category `unknown`. A provider can type the failure with ACP's
|
||
authentication-required code, or with a known error type or `errorKind` in the ACP
|
||
error data. The cause then uses that category, and the rejection classification is
|
||
`provider_<category>`, such as `provider_quota_exhausted`, instead of
|
||
`session_ensure_failed`.
|
||
|
||
Completed OpenCode builtin `bash` parts preserve the numeric process exit from
|
||
`state.metadata.exit` in `tool.execution.completed.payload.exitCode`. The tool
|
||
status stays `completed` even when the command exits with a nonzero code. This
|
||
status describes completion of the tool call, not success of the command or the
|
||
native run. Metadata adds an exit only when it is an integer. Conflicting supplied
|
||
legacy exits stay unknown. Existing numeric `state.exit` values remain supported;
|
||
metadata on other tools cannot add process-exit proof.
|
||
|
||
An edit tool call can carry `fileChanges` on its `tool.execution.*` payload.
|
||
Each entry names one workspace-relative `path`, its `operation` (`create`,
|
||
`modify`, `delete` or `rename`), `previousPath`, `additions`, `deletions`, a
|
||
unified `diff` and `diffTruncated`. The runner drops paths outside the working
|
||
directory, redacts secrets in each patch, and bounds a patch to 32 KiB and one
|
||
call to 32 files. Counts cover the whole patch even when the patch is shortened.
|
||
Codex `fileChange` items produce these rows. Claude Edit and Write calls produce
|
||
them from the ACP `diff` content. Codex `turn/diff/updated` also records the
|
||
whole-turn diff as `workspace.change.updated` (`paperclip.workspace.diff.v1`).
|
||
The task chat shows each patch under its edit row. It ends the turn with the
|
||
change set, or with one built from the edit rows when the harness reports no
|
||
whole-turn diff.
|
||
|
||
`auth.credentials.updated` is a runner-produced PRP event for a change to a
|
||
supported credential store, including account identity changes and later token
|
||
refreshes on the new account. The bundled transport awaits a host commit callback
|
||
before acknowledging delivery, including during warm idle and provider cleanup.
|
||
The callback writes directly to the bound run through the shared run-log writer.
|
||
Its separate credential source namespace avoids collisions with driver-normalized
|
||
source sequences and deduplicates retries using the original PRP event identity.
|
||
The original envelope is retained in `prpEvent`. Its payload contains only a
|
||
provider, an opaque binding ID, a revision, and a detection method. It contains
|
||
no credentials, account identity, file path, or credential fingerprint. The
|
||
consumer can use the event to read the updated credentials through its existing
|
||
private credential channel. This run-log event stays in the instance database.
|
||
|
||
|
||
`native.recovery.execution_reclaimed` records an atomic same-run recovery claim
|
||
that restores a released task execution binding. The claim locks the task,
|
||
coordinator, and original run in that order. It preserves the assignee and checks
|
||
current execution/checkout owners, other live work, dependencies, pause holds,
|
||
and stop/successor records before restoring the binding. Attachment authorization
|
||
keeps its existing requirement for that exact binding. Remote retry claims also
|
||
require the exact saved controller authority and reject a newer task run. The
|
||
event does not prove that a remote provider survived or permit provider replay.
|
||
Skipped local retry candidates wait 30 seconds before recheck. Remote admission
|
||
holds use the existing bounded system-recovery backoff independently of provider
|
||
attempts. Permanent ineligibility or policy exhaustion records the terminal
|
||
automatic recovery disposition and board action without releasing the retained
|
||
execution binding, provider, or checkpoint. These settled holds schedule no
|
||
further retry. Repeated blocked visits record only a changed hold reason.
|
||
The same durable hold prevents background stale-lock cleanup and queue release
|
||
from clearing either task binding until the remote termination receipt contract
|
||
is satisfied. A terminal heartbeat status is not a provider-stop receipt.
|
||
Undated retries sort first. A failed candidate cannot discard earlier committed
|
||
claims. Queued waiters do not block the original execution owner from recovery;
|
||
they still wait for that owner to settle before normal queue promotion.
|
||
This is local run-log data.
|
||
|
||
Fresh remote Codex model substitution emits `runner.model_fallback` on the
|
||
system stream with `warn` severity. Its payload contains `requestedModel`,
|
||
`effectiveModel`, and `codexCliVersion`. It records a preparation choice before
|
||
provider launch, not a failed turn or a retry. The same substitution appears as
|
||
a system warning in the task conversation. It remains local run-log data.
|
||
|
||
The hidden native coordinator writes each validated PRP event to the bound
|
||
run's existing event stream before it acknowledges the runner. The row keeps
|
||
the PRP `eventType`, source instance, source event ID, source sequence, protocol
|
||
schema version, and a SHA-256 digest of the canonical stored envelope. Its
|
||
payload is `{ "prpEvent": <canonical PRP event> }`.
|
||
|
||
Native API tool events omit HTTP request and response bodies from durable history.
|
||
This includes serialized provider `contentItems`, legacy HTTP receipts and
|
||
`call_api` arguments. Saved receipts retain the operation and call IDs, success
|
||
flag, HTTP status and content type when present. The body becomes
|
||
`***REDACTED***`, with `apiPayloadRedacted: true` and `apiPayloadSha256`, a SHA-256
|
||
digest of the original payload. These digests distinguish duplicate events from
|
||
changed payloads; they cannot recover omitted bodies for debugging or replay.
|
||
The model receives the original authorized response. Credential-bearing mutation
|
||
receipts use encrypted storage separately from this run log.
|
||
|
||
Checkpoints and completion records apply the same redaction. Completion retries
|
||
compare stored and expected payloads through the control-plane port's idempotent
|
||
`normalizeReplayPayload` projection, so a saved redacted result can be confirmed
|
||
without restoring API bodies. Ports without a projection compare their original
|
||
payloads. Repeated redaction preserves the original digest and detects changed
|
||
private payloads.
|
||
|
||
The native runtime passes this projection through backend recovery to the Codex
|
||
driver. Recovery compares saved semantic results and turn fingerprints with
|
||
provider history through the same representation. A private body change retains
|
||
its distinct digest and fails recovery; an unchanged body does not require
|
||
restoring its plaintext into the checkpoint.
|
||
|
||
PostgreSQL JSONB cannot represent NUL (U+0000), which can occur in command
|
||
output such as Vite virtual-module paths. The run-event payload column uses a
|
||
lossless storage codec for these events: the JSONB projection renders NUL as
|
||
the literal `\u0000`, and the reserved `$paperclipRunEventJsonV1` field contains
|
||
the original serialized JSON as a doubly escaped string. Ordinary payloads
|
||
retain their existing representation. Drizzle reads restore the exact stored
|
||
payload before replay, hash validation, redaction, or API presentation. SQL
|
||
queries can still inspect ordinary routing fields in the projection; raw SQL
|
||
readers of the whole payload must apply `decodeRunEventPayload`. The column
|
||
remains JSONB and requires no schema migration.
|
||
|
||
The writer locks the native `heartbeat_runs` row and allocates the existing
|
||
per-run `seq` cursor. A byte-equivalent retry reuses the first row; a changed
|
||
retry or source-sequence gap is rejected. Company, issue, agent, run, session,
|
||
and runner-source bindings must match the persisted native run. Bootstrap
|
||
tickets, reconnect leases, authentication proofs, encryption keys, and raw
|
||
credential material are never written to the run log.
|
||
|
||
These records remain run-log events. They do not create an OpenTelemetry or
|
||
Paperclip Telemetry export, and legacy adapters do not use this writer.
|
||
|
||
## Semantic Settlement Diagnostics and Retry Logs
|
||
|
||
When native tool arguments fail their authorized JSON Schema, the runner records
|
||
`harness.diagnostic` with code `semantic_tool_denied` and `paperclipExecuted: false`.
|
||
Its optional `inputSchemaGuidance` retains the same safe repair guidance sent to
|
||
the provider: at most three authorized schema locations within 512 bytes.
|
||
It includes no rejected arguments, submitted values, or unknown input keys.
|
||
The rejected call does not create a semantic input event or execution receipt.
|
||
Validated runner diagnostics retain this scalar metadata even when the operation
|
||
is an API or case tool. API request and response bodies keep their existing redaction.
|
||
|
||
If a declared native completion tool is rejected and the turn ends without an
|
||
accepted replacement report, the runner emits `session.failed` with code
|
||
`semantic_completion_rejected` and `recoverable: false` before the provider's
|
||
original turn terminal. It emits no successful result proposal or human review
|
||
request. Core uses its existing permanent native failure path and does not start
|
||
a missing-result recovery turn. This failure also precedes terminal goal updates;
|
||
Core records the original turn terminal before rejecting goal success or further
|
||
goal continuation. Recoverable goal failures retain their existing behavior.
|
||
The rejection survives same-turn restoration;
|
||
a new turn or run binding clears it. A later accepted completion report keeps
|
||
its normal authority, and ordinary turns with no completion-tool attempt keep
|
||
their existing fallback.
|
||
|
||
Before interrupting a Codex turn, the runner durably records a
|
||
`harness.diagnostic` event with code `provider_interrupt_requested`, the stop
|
||
reason, provider turn ID, and pending tool call IDs. A harmless identical result
|
||
replay records `semantic_tool_result_duplicate` with warning severity, call ID,
|
||
operation ID, and result digest. It does not fail the task or create a user
|
||
attention request. A true conflict includes the call and operation IDs and both
|
||
result digests in its error; full arguments belong to the canonical semantic
|
||
input record, not the error message.
|
||
|
||
When the tool bridge rejects a recorded result for a call it still holds, for
|
||
example a result envelope that names another operation or a value over the
|
||
retained limit, the runner still settles that call. It returns a bounded,
|
||
non-retryable error to the provider and records
|
||
`semantic_tool_result_undeliverable` with warning severity, call ID, operation
|
||
ID, the rejection reason, and the rejected result's digest. The provider is not
|
||
left waiting for its own client timeout, and the call does not block later
|
||
turns or runs.
|
||
|
||
`run.attach` settles calls that a closed provider turn left pending. The
|
||
provider stopped waiting when that turn ended, and the new run's controller
|
||
never delivers the old run's results. Each such call is recorded as a
|
||
non-retryable failure with an unknown outcome, and the attachment records one
|
||
`semantic_tool_result_expired` diagnostic with the call and operation IDs. A
|
||
call that a live provider turn still waits on keeps blocking attachment as
|
||
`durable_pending_tool_calls`.
|
||
|
||
Incomplete close emits the bounded `native_session_settlement_incomplete`
|
||
runner diagnostic. Its evidence includes runner suspension, provider drain,
|
||
final provider state, pending call/operation/source-event IDs and input digests,
|
||
and incomplete result-delivery command IDs and statuses. If execution and
|
||
cleanup both fail, execution retains its original error identity and cleanup is
|
||
attached as `cleanupError`. `runnerRetained: true` means a remote Runner that
|
||
could not prove its suspension was left running for the same-run
|
||
reattachment instead of being force-stopped; the diagnostic
|
||
`remote runner left alive unsettled for same-run reattachment` records it.
|
||
|
||
Semantic settlement also includes up to 20 content-free failure records, with
|
||
the call ID, operation ID, stage (`dispatch` or `persist_result`), and cause.
|
||
Causes distinguish an oversized command, a full command journal, known storage
|
||
errors, dispatcher rejection, and other persistence failures. Exception messages
|
||
and tool results are excluded. These diagnostics do not authorize replay of an
|
||
operation whose outcome is unknown.
|
||
|
||
Instruction writes also commit an `agent.instruction_write_attempted` activity
|
||
row and a run-scoped `instructionToolAttempts` entry before permitting the
|
||
filesystem effect. They retain the call ID, operation ID, and input digest, not
|
||
instruction text. A later transaction rollback cannot erase this attempt proof.
|
||
A missing success or definite pre-write failure receipt means the outcome is
|
||
unknown, even if the current file contains the requested text. Known validation
|
||
and stale-base failures are saved under the attempt's `failure` entry and replay
|
||
their original status, message, and details without a new write. Unknown
|
||
outcomes remain blocked until reconciled; a
|
||
committed receipt with a lost acknowledgement can replay its exact result.
|
||
|
||
The NDJSON output log appends across repeated `begin` calls for one run. Each
|
||
new handle adds an `attemptId` to its lines. If the local file is absent and a
|
||
durable S3 mirror exists, `begin` restores that prefix before appending. A
|
||
failed restore must not replace the mirror with an empty or partial attempt.
|
||
Publishing a restored prefix uses an atomic create-if-absent operation, so a
|
||
concurrent restore cannot overwrite lines another attempt has already appended.
|
||
Earlier attempts therefore remain available for incident diagnosis. Existing
|
||
records without `attemptId` remain readable.
|
||
|
||
These records use the instance run log and its configured storage. They add no
|
||
Paperclip Telemetry or OpenTelemetry export.
|
||
|
||
## Omitted Unsafe Workspace Export
|
||
|
||
`workspace_export_omitted` is an informational system event in the local run log.
|
||
Its payload is `{ "reason": "restore_unsafe_archive" }`, with `"legacy": true`
|
||
when recovering an unsafe failure from an older controller. It records that native
|
||
finalization discarded an unsafe export and continued with the accepted result.
|
||
It contains no archive names, link targets, or raw error details. It does not
|
||
create a task warning, recovery action, Telemetry event, or OpenTelemetry export.
|
||
|
||
## Native Restart Recovery Run-Log Event
|
||
|
||
Paperclip writes a `native.recovery.transition` event for native restart
|
||
classifications and for graceful restart suspension. A blocked remote retry
|
||
records a changed hold reason once; repeated visits with the same hold reason,
|
||
an existing future deadline, or a live lease do not append another transition.
|
||
Exhausting the automatic admission policy records its terminal disposition even
|
||
when the underlying hold reason remains the same.
|
||
This immutable run-log
|
||
record lets operators reconstruct recovery decisions without exporting data to
|
||
Paperclip Telemetry or OpenTelemetry.
|
||
|
||
The payload contains the restart kind, recovery request id when one exists,
|
||
runner disposition, and the controller generation and provider attempt for a
|
||
claimed recovery. Live-runner adoption also records the runner PID, process
|
||
group, and process-start fingerprint. A non-claim disposition records a bounded
|
||
reason instead. Graceful suspension records the signal and confirms that it did
|
||
not create a retry run.
|
||
|
||
The event never includes bootstrap tickets, reconnect leases, authentication
|
||
proofs, encryption keys, environment variables, provider credentials, command
|
||
arguments, or an unsanitized stderr stream. Detailed failed-attempt diagnostics
|
||
remain in the bounded `native_run_finalizations.recovery_history` ledger.
|
||
|
||
`native.remote_runner_recovery_failed` records the closed `stage` and `reason`
|
||
when remote identity verification fails, including admission failures before the
|
||
execution coordinator opens. Unavailable probes are retried within a shared
|
||
30-second deadline; identity conflicts are not retried. The event distinguishes
|
||
an unreadable identity from a confirmed missing process and contains no command
|
||
output or credentials. Saved-state read failures use `state_read_failed`; malformed
|
||
saved JSON uses `state_invalid`. It does not itself authorize replacement or cleanup.
|
||
For `process_not_running`, the event also records `runnerLifecycle` and
|
||
`runnerSettled` from the exited runner's saved state. `runnerLifecycle` is a known
|
||
runner lifecycle, `other` for an unknown value, or `unavailable` when the state has
|
||
none. `runnerSettled` is true only when the outbox is empty and no terminal delivery,
|
||
provider cleanup or warm transition is pending. These fields explain how the runner
|
||
stopped. The event is not recorded when a scheduled retry resumes a runner that
|
||
exited after a settled suspension; see `doc/execution-semantics.md`.
|
||
|
||
`native.interrupted_turn_outcome_unknown` (level `warn`) is recorded once on a
|
||
failed run whose reattachment proved the Runner process gone
|
||
(`process_verification` / `process_not_running`) while its last provider turn
|
||
had no observed settlement, just before the server requests the automatic
|
||
Retry. The payload holds the provider `turnId` (or `null`), the last turn event
|
||
type, `replayed: false`, and the closed `runnerExit` fields `stage`, `reason`
|
||
and `runnerLifecycle`. It names a turn whose outcome is unknown; the Retry's
|
||
fresh session never replays it.
|
||
|
||
For a prepared remote workspace interrupted before its first executor claim,
|
||
that ledger records `remote_execution_never_claimed` with a
|
||
`bootstrap_incomplete` disposition. The provider attempt remains zero during
|
||
classification; the executor increments it before starting provider work.
|
||
A claimed remote bootstrap (the sandbox runner was launched and may have
|
||
connected) records `remote_bootstrap_proven_incomplete` with the same
|
||
disposition. Recovery admits it only after the previous controller lease has
|
||
expired and the Postgres controller checkpoint holds just `run.prepare` and an
|
||
unfinished `session.open`, with no committed or acknowledged event. The
|
||
executor restores that checkpoint and checks the same proof before it
|
||
bootstraps again. After three provider attempts it records
|
||
`execution_recovery_budget_exhausted` instead.
|
||
A claimed remote run whose previous controller is gone, with no
|
||
`native.process_start_requested`, process identity, adapter invocation,
|
||
runner-sourced event or controller checkpoint, records
|
||
`remote_runner_never_launched` with the same disposition and budget. The
|
||
executor records `native.process_start_requested` before a backend may spawn a
|
||
runner, so its absence (with no checkpoint) proves nothing reached a runner.
|
||
A remote run that a graceful drain suspended just after its runner opened the
|
||
provider session records `remote_session_opened_without_turn` with the same
|
||
disposition. Its Postgres controller record must hold only `run.prepare` and
|
||
`session.open`, with every committed event a session-open event and no event
|
||
missing. Before that same-run bootstrap relaunches in the reused sandbox, the
|
||
executor stops the interrupted runner it can verify there, so the new runner
|
||
does not fail with `runner_ingress_bind_conflict`.
|
||
The same rule covers a remote run that its executor already settled as a
|
||
same-run `bootstrap_retry` (for example when a shutdown closed the local
|
||
runtime listener during startup) and that is due when the server restarts.
|
||
|
||
For a local Codex, OpenCode, or ACPX turn lost while runnerd was stopped, the canonical `turn.failed`
|
||
event retains `error.code: "provider_turn_lost_on_restore"` and
|
||
`error.recoverable: true`. It records the interrupted turn, not a final task
|
||
outcome. There is no synthesized result for that interruption. An admitted
|
||
same-session continuation emits its own `turn.submitted`, accepted turn ID, and
|
||
terminal events; only the final outcome closes the run. Reconstructed thread
|
||
history retains the original error so reconciliation and live delivery agree.
|
||
An ACPX session records this only after the original provider lifetime loses
|
||
its fence quorum and the same conversation is restored. Otherwise its
|
||
`turn.failed` carries `error.code: "acpx_active_turn_recovery_closed"` with
|
||
`error.recoverable: false`, and the run closes with an honest restart message.
|
||
After an ACPX restore, the controller can still deliver results for semantic
|
||
calls of the lost turn. Runnerd acknowledges each one as `retired` without
|
||
sending it to the restored provider, so the calls settle and live semantic
|
||
dispatch resumes for the continuation. A restored ACPX provider is a new process
|
||
whose MCP request ids restart, so its semantic `callId` values carry a
|
||
per-process prefix (`r<hex>:<id>`). They never reuse an earlier call's id in
|
||
the same run.
|
||
|
||
## Native Local Process Stop Evidence
|
||
|
||
The server writes `native.local_process_stopped` in the same transaction that
|
||
clears a local run's process identity, after it verifies that its PID and process
|
||
group are absent. The payload contains only those process IDs. Remote process
|
||
IDs are never checked against the control-plane host.
|
||
|
||
A crash can land after `native.process_start_requested` and before the runner's
|
||
process identity is recorded. Continuation admission then writes
|
||
`native.local_process_stopped` with null process IDs and
|
||
`basis: "unidentified_runner_absent"`. It does this only on the host named in
|
||
the launch event's `hostname` payload, after it checks two things. First, no
|
||
process carries the run's `--run-id` runner argument. Second, no
|
||
saved runner control-plane state (canonical or quarantined) belongs to the run;
|
||
a provider starts only after its runner connects and writes that state.
|
||
|
||
The server writes `native.process_start_requested` before a backend can spawn,
|
||
and `native.process_identity_recorded` when it stores a new native process
|
||
identity. Either invalidates an earlier local stop receipt, including a crash
|
||
before the new PID callback. Continuation admission accepts only the latest
|
||
server-authored event among these three types; provider
|
||
source events cannot supply stop authority. These records stay in the local run
|
||
log and do not add Telemetry or OpenTelemetry data.
|
||
|
||
The server writes `native.prior_owner_fence_refused` when its runner-state
|
||
check refuses a run before any launch, because the conversation's saved runner
|
||
state belongs to an earlier run that cannot be proven finished. The payload
|
||
holds only the closed refusal `reason` (for example
|
||
`prior_owner_terminal_state_indeterminate`). Continuation admission reads it,
|
||
with no launch, identity, `adapter.invoke` or runner-sourced event, as proof
|
||
that the failed run never had a runner, so Retry and new input need no stop
|
||
record for it.
|
||
|
||
Explicit continuation writes `native.stopped_conversation_verified` after it
|
||
verifies the retained worker and provider processes stopped. Its payload binds
|
||
the requesting `runId`, retained `ownerRunId`, native session, runner instance,
|
||
process IDs, retained-state fingerprint, successor run, Retry receipt, and local
|
||
workspace identity. The owner can be a preceding terminal
|
||
run when the requesting attempt failed before launch with the same process
|
||
identity. The server checks host-bound process birth and independent provider
|
||
receipts before retiring cleanup ownership. The event does not settle uncertain
|
||
actions or authorize replay of the old session.
|
||
|
||
## Verified Local Codex Replacement Evidence
|
||
|
||
The server writes `native.stopped_text_turn_verified` in the same transaction
|
||
that schedules a fresh successor for a stopped local Codex run. It records the
|
||
runner and provider process identities, retained-state digests, provider thread
|
||
and turn IDs, and IDs of exactly receipted task-completion calls. The server first
|
||
checks the complete turn inventory, process-stop receipt, and execution binding.
|
||
Unknown actions or changed retained state prevent this event and replacement.
|
||
|
||
The record documents why the old execution can be retired. It does not make the
|
||
old session resumable, rewrite provider files, or authorize replay on its own.
|
||
It remains in the local run log and adds no Telemetry or OpenTelemetry export.
|
||
|
||
## Sandbox Startup Run-Log Event
|
||
|
||
Paperclip writes one `run.startup.step` event to the run log for each bring-up
|
||
step. This event is a run-log record, not a first-party telemetry event. The
|
||
generated telemetry contract does not cover it, so this section is its canonical
|
||
contract.
|
||
|
||
The event payload carries only three fields.
|
||
|
||
| Field | Type | Meaning |
|
||
| --- | --- | --- |
|
||
| `step` | string | The bring-up step name, for example `stage.sync`. |
|
||
| `durationMs` | number | The wall time of the step. A skipped step reports `0`. |
|
||
| `outcome` | string | The step outcome (`ok`, `skipped`, or `failed`). |
|
||
|
||
The event no longer carries the per-step round-trip count or the provider
|
||
duration fields. It dropped `roundTrips`, `providerExecMs`, `providerGetMs`,
|
||
`createRuntimeMs`, and `ensureSessionMs`. The startup spans in
|
||
[`doc/observability.md`](observability.md) carry that detail now. The
|
||
`sandbox.exec` child spans hold the round-trip and provider durations. The
|
||
`acp.handshake` step span holds the create-runtime and ensure-session
|
||
sub-times.
|
||
|
||
To read the detailed timing, use the startup spans. The spans need an OTLP
|
||
endpoint. A run with no endpoint keeps only the three run-log fields above.
|
||
|
||
## Run Phase Timing Run-Log Event
|
||
|
||
Paperclip writes one `run.phase.timing` event to the run log for each
|
||
run-lifecycle phase. This event is a run-log record, not a first-party telemetry
|
||
event. The generated telemetry contract does not cover it, so this section is its
|
||
canonical contract. The producer is `emitRunPhaseTiming` in
|
||
`packages/adapter-utils/src/acpx-engine/startup-timing.ts`.
|
||
|
||
The event payload carries only three fields.
|
||
|
||
| Field | Type | Meaning |
|
||
| --- | --- | --- |
|
||
| `phase` | string | The run-lifecycle phase name from the closed allowlist below. |
|
||
| `durationMs` | number | The wall time of the phase. A negative or a non-finite value clamps to `0`. |
|
||
| `outcome` | string | The phase outcome (`ok` or `failed`). |
|
||
|
||
The `phase` field is one member of a closed, low-cardinality allowlist. The
|
||
producer drops any event whose phase name is outside this allowlist, so a
|
||
free-form label never reaches the run log. The allowlist has twelve phase names.
|
||
|
||
| Phase | Meaning |
|
||
| --- | --- |
|
||
| `place_workspace` | Place the run workspace. |
|
||
| `start_transport` | Start the agent transport. |
|
||
| `create_runtime` | Create the agent runtime. |
|
||
| `ensure_session` | Ensure the agent session exists. |
|
||
| `configure_session` | Configure the agent session. |
|
||
| `prepare_turn` | Prepare the turn. |
|
||
| `turn` | Run the turn. |
|
||
| `end_session` | End the agent session. |
|
||
| `settle_reuse` | Settle the session for reuse. |
|
||
| `stop_transport` | Stop the agent transport. |
|
||
| `sync_back` | Sync the workspace back. |
|
||
| `release_staging_lease` | Release the staging lease. |
|
||
|
||
The payload never carries a command, an argument, a path, an environment value,
|
||
or a raw identifier. The event rides the `ctx.onEvent` run-event bridge and is
|
||
run-log-only. It needs no OTLP endpoint.
|
||
|
||
## Workspace Restore and Settled Run Measurements
|
||
|
||
The heartbeat controller writes these records with `eventType: "lifecycle"`,
|
||
`stream: "system"`, and `level: "info"`. The event name below is its `message`
|
||
and its payload's `phase`. They remain in the instance run log and add no
|
||
Paperclip Telemetry or OpenTelemetry export.
|
||
|
||
| Event | Payload fields besides `phase` | Meaning |
|
||
| --- | --- | --- |
|
||
| `workspace_checkpoint` | `mode`, `scannedEntries`, `changedFiles`, `payloadBytes` | Metrics for a validated sparse checkpoint, or a full restore fallback. |
|
||
| `workspace.restore` | `durationMs` | Wall time around native workspace restoration. |
|
||
| `task.run.measured` | `durationMs`, `outcome`, `durableSettlement` | Wall time through terminal owner cleanup, with the persisted terminal run status and `durableSettlement: true`. |
|
||
|
||
For `workspace_checkpoint`, `mode` is `"sparse"` or `"full"`. Sparse metrics
|
||
count snapshot entries in `scannedEntries`, regular files in the validated
|
||
downloaded payload in `changedFiles`, and their total bytes in `payloadBytes`.
|
||
Deleted entries and symlinks do not add to the regular-file count or byte total.
|
||
Full fallback reports all three counts as `null`: unavailable, not zero. A run
|
||
can emit multiple checkpoint records while restoring its repositories. These
|
||
payloads contain no paths, filenames, file contents, or credentials.
|
||
|
||
`workspace.restore` is emitted when the timed restoration exits, including an
|
||
exception. It has no outcome field and does not by itself prove restore success.
|
||
Its producer is the native workspace sync `onPhase` callback.
|
||
|
||
`task.run.measured` is emitted only after the controller observes a terminal run
|
||
with no remaining native runner ownership. Its duration starts at `startedAt`,
|
||
or `createdAt` if the run never started, and includes preparation, execution,
|
||
finalization, and cleanup. It is not model-only time; the creation-time fallback
|
||
also includes time spent queued. `outcome` is the actual terminal status, such as
|
||
`succeeded`, `failed`, or `cancelled`. Provider turn completion can precede this
|
||
settlement record. The record does not imply exactly-once emission across
|
||
controller recovery, and failure to write it is logged without changing the
|
||
already-settled outcome.
|
||
|
||
## Storage outage waits
|
||
|
||
A run waits out a file-storage outage on its critical path: when it starts its
|
||
run log, restores its saved workspace, or saves it. While it waits, the run's live
|
||
status says so, with phase `waiting_for_storage`, for example "Waiting for file
|
||
storage before starting the run log (45s so far)". That status lives in server
|
||
memory and on the live-event stream only. It writes nothing to the run row or
|
||
the run log during the wait.
|
||
|
||
When the wait ends, the controller writes one record with `eventType:
|
||
"lifecycle"`, `stream: "system"`, and `level: "warn"`. Its message is "Waited Ns
|
||
for file storage before <step>." or, when storage stayed unavailable, "Waited Ns
|
||
for file storage before <step>; it was still unavailable." Its payload holds
|
||
`reason: "storage_outage_wait"`, `purpose` (the step), `waitedMs`, and
|
||
`recovered`. The record stays in the instance run log.
|
||
|
||
When the outage outlasts that wait, the run fails. Only a run recorded with
|
||
`errorCode: "storage_unavailable"` gets a storage wait: a bounded retry
|
||
(`retryReason: "storage_unavailable_wait"`) that starts again after storage is
|
||
back. Its error says "Paperclip will retry automatically when storage is back"
|
||
only while that retry is scheduled. When the retry budget is spent, the error
|
||
says so and asks for Retry. When no storage wait follows, the error says
|
||
"Paperclip is not retrying it automatically. Retry or send a message to
|
||
continue." That happens when a native recovery already holds the run under
|
||
another code (for example `native_continuation_requires_reconciliation`, where
|
||
its files are kept and its unfinished turn is not replayed), or when the retry
|
||
is refused or cannot be written. A run saved by an earlier release with the
|
||
retry promise under another code reads with the same Retry text, with no
|
||
migration.
|
||
|
||
## Durable storage timing
|
||
|
||
A saved remote workspace is one encrypted archive in durable object storage.
|
||
Restoring it before a turn, and saving it after one, runs these steps as named
|
||
run phases:
|
||
|
||
| Phase | Step |
|
||
| --- | --- |
|
||
| `storage.snapshot.download` | Get the encrypted archive from storage. |
|
||
| `storage.snapshot.decrypt` | Hash the downloaded archive and decrypt it to private scratch, at the same time on worker threads. |
|
||
| `storage.snapshot.verify_plaintext` | Hash the decrypted archive against its sealed digest while copying it beside its destination, on a worker thread. |
|
||
| `storage.snapshot.publish` | Link the verified copy to its destination. |
|
||
| `workspace.snapshot.sandbox_restore` | Upload and unpack the archive in the sandbox. |
|
||
| `workspace.snapshot.sandbox_upload` | Inside `sandbox_restore`: copy the archive into the sandbox. |
|
||
| `workspace.snapshot.sandbox_extract` | Inside `sandbox_restore`: verify and unpack the archive, then publish the restored folder. |
|
||
| `storage.snapshot.prior_attempts` | Check whether an earlier upload attempt already landed. |
|
||
| `storage.snapshot.encrypt` | Hash the archive, encrypt it and hash the encrypted file, in one pass on a worker thread. |
|
||
| `storage.snapshot.digest` | Record the digest of the encrypted file. The `encrypt` pass computes it, so this step takes no time. |
|
||
| `storage.snapshot.put` | Upload the encrypted archive (multipart above 16 MiB). |
|
||
| `storage.snapshot.verify` | Confirm the stored object (its size when the backend checked every part, otherwise a full read-back). |
|
||
|
||
A restore runs inside run preparation, so each of its steps of 50 ms or more is
|
||
a `run.performance.span` row of the run. A save runs after the turn. Its step
|
||
durations are recorded on its `remote_workspace_snapshot` workspace operation,
|
||
in `metadata.saveTiming.storage`:
|
||
- `priorAttemptCheckMs`, `encryptMs`, `digestMs`, `putMs` and `verifyMs`;
|
||
- `bytes`;
|
||
- `requests`, the storage requests the save sent. These are `requests`,
|
||
`attempts` (SDK attempts across them), `failedRequests`, `retryDelayMs` (SDK
|
||
waits between its own retries), `parts`, `throttled` (part sends the backend
|
||
rate-limited with 503, 429 or SlowDown) and `backoffMs` (the uploader's waits
|
||
before resending a part).
|
||
|
||
A restore also records its step durations, `bytes` and `requests` on its
|
||
`remote_workspace_restore` workspace operation, in `metadata.restoreTiming`
|
||
(schema `paperclip.remote-workspace-restore-timing.v1`). A save that reuses an
|
||
earlier upload still reads it back; that read-back is counted in `verifyMs`.
|
||
A throttled part send counts in `throttled` even when it ends the retries,
|
||
and a multipart abort sent during cleanup counts in `requests`.
|
||
|
||
These are counts and durations only: no object keys, paths or provider text.
|
||
|
||
## System notices and final replies
|
||
|
||
Automatic comments with a stored `system_notice` presentation, including
|
||
workspace-ready notices, remain available as progress information. They do not
|
||
occupy the run's final reply slot. Final response selection still preserves
|
||
ordinary explicit agent comments and excludes receipt-bound generated progress.
|
||
|
||
## ACP terminal failure diagnostics
|
||
|
||
The shared ACP adapter engine preserves typed terminal session failures in the
|
||
run error, the `acpx.error` transcript record, and
|
||
`heartbeat_runs.result_json.terminalSessionFailure`. The structured diagnostic
|
||
contains the provider category, title, and details. It works when raw provider
|
||
tracing is disabled. The existing UI and CLI render the diagnostic as an error,
|
||
not as assistant output or an automatic task response.
|
||
Issue continuation summaries and session-compaction handoffs retain only the
|
||
generic failure category; provider diagnostic prose is not copied into prompts.
|
||
|
||
Both pinned ACPX patches pass complete title and detail strings to the in-memory
|
||
callback. The engine redacts configured environment values (including resolved
|
||
secrets with arbitrary variable names), launch environment values outside a
|
||
closed allowlist of public process settings, known boolean flags, and run identifiers,
|
||
connection URL passwords, the run API key, and common credential forms before
|
||
truncation. It removes control characters,
|
||
retains line breaks for JSON and stack traces, and preserves up to 4,096 title
|
||
characters and 24,576 detail characters. These bounds also keep the escaped
|
||
transcript JSON below the server's 64 KiB chunk limit. Longer fields end with an explicit
|
||
omission count and appear in `truncatedFields`. The error message includes the
|
||
same sanitized text. Ordinary run retrieval preserves the bounded structured
|
||
diagnostic even when multibyte text or other result fields exceed the result
|
||
byte budget. In that reduced response, title and details have 1 KiB and 8 KiB
|
||
byte budgets, including truncation markers. `retrievalTruncated` directs callers
|
||
to the full adapter-bounded text in the run error or transcript. Other provider
|
||
metadata and action payloads are not copied.
|
||
|
||
When an ACP request fails with a JSON-RPC error, the engine also keeps the
|
||
provider explanation. It reads a string `data`, or the string `data.details`, or
|
||
the string `data.message`, in that order. It adds the explanation on a new line
|
||
after the error message unless the message already contains it. The combined
|
||
text passes through the same redaction and the 4,096-character title bound
|
||
before it becomes the run error, the summary, and the `acpx.error` `message`.
|
||
A numeric JSON-RPC error code appears as the optional integer `rpcCode` in
|
||
`errorMeta` and in the `acpx.error` record. Errors without these fields keep
|
||
their existing message and omit `rpcCode`.
|
||
|
||
Recovery still uses the typed failure category and the adapter's existing
|
||
classifier. Provider warnings do not become failures, and timeouts or lost
|
||
control channels keep their authoritative failure messages. These diagnostics
|
||
stay in the instance's run records and configured run-log storage. They add no
|
||
Paperclip Telemetry or OpenTelemetry export.
|
||
|
||
## Related instrumentation
|
||
|
||
The sandbox duplex transport also writes one run-log event as one of its three
|
||
sinks. See the
|
||
[Sandbox Duplex Transport Instrumentation](observability.md#sandbox-duplex-transport-instrumentation)
|
||
section in the Observability contract.
|
||
|
||
## Execution recovery
|
||
|
||
Cancelled runs retain `resultJson.cancellation`: a closed `source` label
|
||
(`operator`, `queued_message`, `shutdown`, `provider`, `transport`,
|
||
`control_plane`, `controller_lease`, or `unknown`), whether the stop was expected,
|
||
the initiator, reason, and recording time. Recorded stop intent survives adapter
|
||
completion. `controller_lease` means the host stopped its own legacy adapter
|
||
after it lost the durable controller lease; the reason names a closed cause
|
||
(`renewal_rejected`, `renewal_deadline`, or `renewal_error`). The watchdog
|
||
retries a failed renewal until the lease deadline, so `renewal_error` means the
|
||
attempts still failed with an error when the deadline passed.
|
||
All legacy controllers in one server process renew through a single queue on
|
||
the private lease connection pool. Renewals that arrive while one is in flight
|
||
share the next statement, so a burst of runs costs one renewal statement per
|
||
round instead of one transaction per run, and a renewal waits behind at most
|
||
one other. A run whose row another transaction holds is skipped and retried
|
||
shortly, without delaying the other runs. A renewal only moves the lease
|
||
deadline later, so it does not wake live-run recovery; recovery rereads the
|
||
moved deadline when the earlier one comes due.
|
||
A lease lost during run setup (for example a rejected renewal while preparing
|
||
the workspace) settles the run `failed` with error code `controller_lease_lost`,
|
||
the same `controller_lease` cancellation evidence, and
|
||
`executionRecovery.providerWorkStarted: false`. Recovery retries it as transient
|
||
infrastructure instead of blocking on `setup_failed`. A setup-phase error after
|
||
provider dispatch began keeps `setup_failed`, so started work is never retried.
|
||
The local lifecycle event includes this evidence. Started cancellations without
|
||
an expected stop are also reported to Sentry; its cancellation diagnostics contain
|
||
only source, expectedness, initiator type, and the closed lease-loss cause, never
|
||
initiator IDs or reason text.
|
||
A lease loss also records `resultJson.controllerLeaseLoss`, including a loss
|
||
during setup that fails the run before adapter dispatch. It holds the closed
|
||
cause and the watchdog timings in milliseconds: `sinceRenewalMs` (since the
|
||
last successful renewal), `renewalPendingMs` (age of a renewal operation still
|
||
in progress, including its retries and retry waits, or `null`), `deadlineLateMs`
|
||
(how late the deadline callback ran, when the lease deadline ended the run,
|
||
otherwise `null`), and `renewalCount`. A long `renewalPendingMs` can mean a
|
||
stalled database query or repeated failed attempts with retry waits; the cause
|
||
label tells them apart (`renewal_deadline` or `renewal_error`). A long
|
||
`deadlineLateMs` points at a blocked event loop. Failure reports carry these
|
||
bounded numbers and the cause label only.
|
||
Historical ambiguous cancellations stay `unknown` and do not authorize replay.
|
||
|
||
Provider tool-definition validation failures use
|
||
`provider_tool_definition_invalid` / `configuration`. Automatic retry and
|
||
continuation recovery stop until the configuration is repaired. Classification
|
||
uses raw provider diagnostics in memory before redaction; stored diagnostics
|
||
remain redacted and bounded.
|
||
|
||
Provider identity diagnostics remain in the local run log. They record the notification method, expected and received thread/turn identifiers, and the classification (root, verified descendant, stale, unrelated informational, or invalid authoritative). They omit the original provider payload and credentials. Repeated informational notices are bounded.
|
||
|
||
Ignored unrelated Codex notifications use `harness.diagnostic` with code
|
||
`codex_unrelated_information`. The payload retains only the bounded provider
|
||
method and expected/received thread and turn identifiers. Account updates, skill
|
||
changes, and unrelated thread information do not create a provider notice in
|
||
chat. Chat also omits the matching notice stored by older runners. Real provider
|
||
warnings and errors remain visible.
|
||
|
||
Provider rate-limit windows use `provider.notice.recorded` with category
|
||
`provider.rate_limits` and scope `account`. Codex sends them as
|
||
`account/rateLimits/updated`. Claude sends them as `usage_update` metadata
|
||
(`_claude/rateLimit`) when it uses a subscription sign-in. The runner merges
|
||
sparse Codex updates and records a notice only when the windows change. The
|
||
notice has one `rateLimits` detail: a JSON snapshot with schema
|
||
`paperclip.provider.rate_limits.v1` (windows with used percent and reset time,
|
||
credits, and the reached window). Chat does not show the notice. The server
|
||
keeps the latest snapshot in the agent's runtime state under
|
||
`providerRateLimits`, and the task properties pane shows it as the assignee's
|
||
**Usage limit**. The snapshot is display data. It does not hold or start work;
|
||
the provider-limit wait reads typed run failures.
|
||
|
||
Harness status notices use `provider.notice.recorded` with a status category.
|
||
Codex `error` notifications read `error.message`. A notification with
|
||
`willRetry: true` becomes a recoverable `provider_retry` notice. A final error
|
||
keeps category `error`. The `CodexErrorInfo` kind, the HTTP status, and the
|
||
reconnect attempt count become notice details. Claude compaction, model
|
||
fallback, and SDK informational banners arrive as ACP message text without a
|
||
message identity. The ACPX layer records them as `context_compaction`,
|
||
`model_fallback`, or `provider_informational` notices instead of answer text.
|
||
Compaction start and end share one notice identity and carry a `Phase` detail.
|
||
The task chat shows these notices as inline status chips. A retry chip fades
|
||
after later provider work. A final provider error shows the real provider
|
||
message. Generic warnings stay out of chat as before.
|
||
|
||
Recovery lifecycle events retain the original structured failure code, retry attempt, next retry time, and predecessor/successor identifiers. Durable status delivery uses an idempotency marker; delivery grants no provider authority. Failed publication is retried without repeating provider work. These records are not first-party Telemetry.
|
||
|
||
An unstarted legacy conversation retry waiting for execution cleanup retains
|
||
`resultJson.executionWait` with the issue, blocking run, and
|
||
`execution_owner_active` cause. A local lifecycle event records the wait and the
|
||
unchanged scheduled retry attempt once per blocking run. Repeated scheduler
|
||
checks update the same retry row without repeating that event. Promotion removes
|
||
this ownership-wait marker after cleanup; other execution holds retain their
|
||
existing dispatch gates.
|
||
|
||
If execution-continuation setup finds that a task no longer exists, is closed,
|
||
or its owner changed, the existing cancellation settlement records
|
||
`continuation_task_ownership_changed`. The run and wake request become cancelled
|
||
before adapter dispatch, with the run-log message
|
||
`stale execution continuation cancelled before dispatch`. Immediate recovery is
|
||
suppressed. A comment the cancelled run carried that no run read is saved
|
||
first when an agent run closed the task: a person's comment, or another agent's
|
||
comment posted with `resume: true`. The release then reopens the task and
|
||
delivers that comment once. Another agent's comment without `resume: true`
|
||
stays communicative and does not reopen the task. Missing source context, authorization failures, and other setup
|
||
errors retain their failure classification. An untyped error with the same
|
||
message is also still a failure; cancellation requires the typed ownership guard.
|
||
|
||
Bounded retry exhaustion writes one lifecycle receipt per run, retry reason,
|
||
scheduled attempt, and retry limit. Repeated or concurrent recovery checks reuse
|
||
that receipt, including receipts from earlier builds, without advancing the event
|
||
sequence or publishing another live event. Attention reads select the latest
|
||
matching receipt in PostgreSQL and project only the run's issue/task identifiers
|
||
from its context, so historical duplicate receipts cannot multiply run contexts
|
||
in server memory. Existing duplicate events do not require deletion or migration.
|
||
|
||
### Workspace restore failures
|
||
|
||
Legacy adapter results can carry `workspaceRestoreFailure` with the code
|
||
`restore_permission_denied`, `restore_lock_timeout`, `restore_unsafe_archive`,
|
||
or `restore_failed`. The heartbeat records `workspace_restore_failed` and keeps
|
||
the run failed and the workspace-finalization barrier closed. Available output,
|
||
usage, session metadata, and the previous execution outcome survive settlement.
|
||
`executionBeforeRestore` retains the earlier error code, exit code, signal, and
|
||
timeout flag. The ordinary redacted error field retains an earlier error message.
|
||
|
||
The chat reports the restore phase separately from a missing final response.
|
||
A saved-plan link requires a stored document and its run-bound revision or a
|
||
matching run-bound review record. Older unclassified failures use neutral wording. Diagnostics show only
|
||
a validated relative member path, never an archive link target or host path.
|
||
|
||
An unsafe archive or an outbound confinement refusal keeps the existing execution recovery hold, including across
|
||
conversation resets. It cannot start another model turn until an operator uses
|
||
the existing recovery action to record `executionReconciliation` with
|
||
`workspaceRepairEvidence` (20–12000 characters). This evidence must describe
|
||
verified safe staging or repair for the referenced failed run. It does not grant
|
||
plan approval. Saved comments, document revisions, and confirmation IDs and
|
||
states stay unchanged. Recovery uses the existing delivery identity and links
|
||
the successor to the original failed run. Repair does not reset the automatic
|
||
retry budget. Transient failures retain the existing
|
||
bounded retry policy. Archive confinement remains required.
|
||
|
||
Sandbox restore tasks also write a `Workspace restore diagnostic` line to the
|
||
run log on failure. `phase` is `workspace` or `asset`, so a failed staged-asset
|
||
copy-back (such as credentials) can be distinguished from workspace restoration. The line
|
||
contains only an allowlisted OS/transport `errorCode` (otherwise `unknown`), an
|
||
optional numeric HTTP error status, and an optional bounded process exit code.
|
||
Up to four nested causes are inspected. Messages, URLs, filesystem paths, asset
|
||
names, credentials, and response bodies are excluded. Every failed outbound
|
||
task emits its own diagnostic; nested repository failures are logged once by
|
||
the enclosing workspace task. The original error and restore safety policy are
|
||
unchanged. These lines stay in the instance run log and its configured durable
|
||
storage, and are not new first-party telemetry events.
|
||
|
||
Native sandbox `environmentSyncOut` errors preserve allowlisted error codes and
|
||
bounded HTTP/exit statuses across worker RPC for this diagnostic line. The host
|
||
revalidates the envelope and keeps the original failure and recovery policy.
|
||
Provider messages, paths, response bodies, credentials, and arbitrary error data
|
||
are not copied into the new envelope. Older workers can still report `unknown`.
|
||
|
||
The optional `step` identifies the failed restore operation. For
|
||
`phase=workspace` and `step=git_integration`, `gitCommand` identifies one fixed
|
||
command family (`rev_parse`, `symbolic_ref`, `merge_base`, `merge_tree`,
|
||
`commit_tree`, `update_ref`, or `log`). `gitFailureKind` is `merge_conflict`,
|
||
`invalid_object`, `ref_conflict`, `permission_denied`, or `unknown`; it is a
|
||
bounded diagnostic clue, not a new recovery or retry decision. Only supported
|
||
exit/OS codes and recognized Git messages produce a specific classification.
|
||
No command arguments, stderr, filenames, repository URLs, or ref names are saved.
|
||
The same closed fields persist in `workspaceRestoreDiagnostic` and are
|
||
revalidated before projection into an enabled Sentry failure report.
|
||
|
||
## Codex resume usage snapshot
|
||
|
||
The native runner retains a bounded local `harness.diagnostic` event with code
|
||
`codex_resume_usage_snapshot`. It identifies `thread/tokenUsage/updated` as
|
||
`resume_usage_snapshot`, retains the reported thread and completed-turn IDs,
|
||
and records cumulative usage counters. It does not include provider credentials
|
||
or message content. The event establishes the accounting baseline; it is not a
|
||
new billable usage receipt or a user-facing provider warning. Other provider
|
||
identity checks remain in force.
|
||
|
||
## OpenRouter serving host
|
||
|
||
OpenRouter can route one model across many hosts, which may serve different
|
||
quantizations. OpenCode runs routed through OpenRouter record which host served
|
||
each model call. The runner's provider proxy reads the top-level `provider` and
|
||
`id` fields from the first streamed chunk, or from the whole JSON body when the
|
||
response is not streamed. It forwards every byte to OpenCode unchanged and
|
||
before it inspects them, so the first token is not delayed. It scans at most
|
||
the first 64 KiB of a stream and 1 MiB of a JSON body. It skips encoded bodies,
|
||
non-2xx responses, and any malformed or missing field. A recording failure
|
||
never affects the response or the turn.
|
||
|
||
Each `item.completed` usage event (`payload.kind: "usage"`) carries two fields
|
||
in `payload.usage`:
|
||
|
||
- `upstreamProvider`: the host that served the most recently finished model
|
||
call when the usage was reported, such as `Novita` or `DeepInfra`.
|
||
- `providerRequestId`: that call's OpenRouter generation ID, such as
|
||
`gen-1762981234-AbCdEfGh`.
|
||
|
||
The proxy records a call's host when that call's response ends, not when the
|
||
request starts. OpenCode reports a step's usage only a few milliseconds before
|
||
it sends the next model request, so a step's usage still names its own call's
|
||
host while the next call is in flight. A call whose host the proxy could not
|
||
read records no host when it ends, and a session clears the recorded host when
|
||
it attaches another run. Such a call therefore carries neither field. It never
|
||
inherits an earlier call's or an earlier run's values.
|
||
|
||
The runner's internal `usage.reported` event carries the same values, using its
|
||
existing `providerRequestId` field and an optional `upstreamProvider` field.
|
||
The run's `usageJson.providerRequestId` and its cost event name the first
|
||
generation ID the run reported. That value stays the same as later turns
|
||
report, so a late usage receipt still matches the charge. Remote Runner runs carry
|
||
the same ID in each saved usage receipt's producer event and in the receipt
|
||
itself. A receipt saved before this field existed still replays when the
|
||
Runner now names an ID. Per-call IDs and
|
||
hosts remain in the run log. Runs whose provider reports no host carry neither
|
||
field, and `usageJson.providerRequestId` stays `null` as before.
|
||
|
||
These are run-log fields. They stay in the instance database and are not sent
|
||
to Paperclip Telemetry or OpenTelemetry.
|
||
|
||
## AI subscription contention
|
||
|
||
A fresh task execution cannot enter this wait. A run that already entered this
|
||
wait writes an informational `lifecycle` event to the local run log. Its
|
||
payload contains only `retryScheduled`, a boolean that reports
|
||
whether the scheduler created a retry.
|
||
The message distinguishes an automatic retry from work that is no longer eligible.
|
||
This pre-provider wait records `ai_connection_busy` on the cancelled run and does
|
||
not consume the provider-failure retry allowance. The event contains no credentials
|
||
and creates no Telemetry or OpenTelemetry export.
|
||
|
||
## Agent directory waits
|
||
|
||
A run whose instruction preparation finds another run of the same agent using
|
||
its files writes one informational `lifecycle` event to the local run log, with
|
||
`stream: "system"`. It records no file contents, paths or credentials, and
|
||
creates no Telemetry or OpenTelemetry export.
|
||
|
||
- A resumed native run that stays on its own run writes the message "Another
|
||
run of this agent is still using its files; this run continues when it
|
||
finishes". Its payload has exactly two fields: `retryReasonCode`, always
|
||
`agent_directory_operation_busy`, and `nextAttemptAt`, the ISO time when the
|
||
same run is scheduled to try again.
|
||
- A run that has not yet started native execution is cancelled with error code
|
||
`agent_directory_busy` and writes "Another run of this agent is still using
|
||
its files. This task will continue automatically." when an
|
||
`agent_directory_busy` retry was scheduled, or "Another run of this agent is
|
||
still using its files, and this task could not be rescheduled." when it was
|
||
not. Its payload contains only `retryScheduled`, a boolean.
|
||
|
||
Neither wait consumes the provider-failure retry allowance. Each stops after 30
|
||
deferrals, after which the run follows the normal failure path.
|
||
|
||
## Task history reads
|
||
|
||
When a native run reads its task's comments through `get_task_history`, the
|
||
server writes one `task_history.read` event (`stream: "system"`, level `info`)
|
||
to that run's log. Its payload has one field, `comments`, which maps each
|
||
returned comment ID to that comment's `updatedAt` timestamp. It contains no comment bodies or
|
||
author identifiers. The event creates no Telemetry or OpenTelemetry export.
|
||
|
||
Per-comment delivery (`deliveredQueuedCommentIds`) reads this event together
|
||
with the run's `run.result.accepted` event. A read alone is not delivery: the
|
||
run may have read the comment and set it aside. A comment counts as received
|
||
only when a run of the task's assignee succeeded after reading the comment's
|
||
current revision, and that run's accepted result cites the comment as a
|
||
`comment:<id>` evidence reference (top-level `evidence[].ref` or a criterion's
|
||
`evidenceRefs`). The comment's queued wake then neither reopens the finished
|
||
task nor starts another turn. An edit after the read, a read by a run that did
|
||
not succeed, or a read the result does not cite leaves the comment owed, so it
|
||
is delivered once in a new turn. That turn's wake marks the comment
|
||
`readByEarlierTurn` when an earlier finished run read its current revision, and
|
||
the prompt asks the agent to check the current task state before acting so
|
||
already-applied work is not repeated. `get_task_history` returns
|
||
`arrivedDuringThisTurn` (the `comment:<id>` refs of comments that other authors
|
||
posted or edited after the run started) with instructions to cite each applied
|
||
comment. The check reads only the assignee's succeeded runs of that task that
|
||
finished at or after the comment's current revision, through the existing
|
||
run-binding and `(company_id, run_id)` event indexes. A saved follow-up
|
||
resumed after a remote Stop uses the same rule. `taskHistoryReadsForRun` lists
|
||
the comments one run read, with each read's time and the comment revision it
|
||
saw, whatever the run's outcome. A failure to write the event leaves the
|
||
comment owed in the same way.
|
||
|
||
When a run reads a finished child task, the server writes one
|
||
`child_task.read` event (`stream: "system"`, level `info`) to that run's log.
|
||
The reads that count are `get_task_context` on the parent task, and a
|
||
successful agent request with a run token for `GET /api/issues/:id`,
|
||
`GET /api/issues/:id/documents` or `GET /api/issues/:id/documents/:key` of a
|
||
child in `done`. The route writes the event only after the requested content
|
||
has loaded, so a 400 or 404 records nothing. The payload has `parentIssueId`
|
||
and `children`, which maps each finished child ID to the revision the run
|
||
received: its `status`, `updatedAt` and `statusDecisionId`. It contains no
|
||
titles, summaries or document bodies. The event creates no Telemetry or
|
||
OpenTelemetry export.
|
||
|
||
`hasPendingNativeChildCompletion` reads this event. A child-completion wake no
|
||
longer keeps the parent's done claim open when the finishing run received that
|
||
child in `done` at the completion the wake reports: the received
|
||
`statusDecisionId` is the wake's completion decision, or the received
|
||
`updatedAt` is at or after that decision was saved. The event's own write time
|
||
does not count, so a delayed write of an older revision cannot deliver a newer
|
||
completion. The wake is then cancelled at release instead of starting another
|
||
parent turn. A read while the child was still open, a read of an older
|
||
revision, a read by another run, or a failure to write the event leaves the
|
||
child result owed.
|
||
|
||
## Managed Agent File Save Receipts
|
||
|
||
The server writes `instruction_save` after managed file collection or a warm
|
||
turn checkpoint. The payload includes the save state, instruction entry path,
|
||
storage warning, and error code/message. Agent-directory receipts identify
|
||
`contract: "agent_files"` and the applied candidate hash. Legacy instruction
|
||
receipts instead identify the saved revision.
|
||
|
||
A validated warm checkpoint reports `saved` or `unchanged`, even though its
|
||
working directory remains owned by the live session. An unstable checkpoint
|
||
reports `pending_collection` until stopped collection produces a final receipt.
|
||
Successful checkpoints can include `checkpointStats`: `scannedEntries`,
|
||
`hashedBytes`, `copiedFiles`, and `copiedBytes`. These counts describe that
|
||
capture, not cumulative traffic or an atomic snapshot of background writers.
|
||
They contain no file contents. The receipt remains in the instance run log;
|
||
it adds no Paperclip Telemetry or OpenTelemetry export.
|
||
|
||
If instruction-copy release throws, `instruction_cleanup` records a warning
|
||
with payload `{ "state": "deferred" }`. The existing working-copy recovery sweep
|
||
retries cleanup. This event preserves the run outcome and does not claim a file
|
||
save; `instruction_save` remains authoritative for collection. The event contains
|
||
no raw exception, host path, file contents, or lock-owner metadata. Failure to
|
||
write the warning must not replace the provider outcome or stop lease release.
|
||
|
||
## Agent observability availability
|
||
|
||
`agent_observability.unavailable` is one warning for a preparation snapshot that
|
||
cannot apply company export settings. Its payload contains only a closed reason
|
||
and the settings revision. It carries no endpoint, credential, secret reference
|
||
or resolver error. Recording the diagnostic never contacts a destination.
|
||
|
||
## Runtime activity
|
||
|
||
`runtime.activity.recorded` is a P1 Runner PRP event with a closed
|
||
`{ namespace, activityKey }` payload. `namespace` groups a producer's activity
|
||
types under a stable name. `activityKey` identifies an activity within that group.
|
||
Consumers use the pair to select a response to activity in the current run.
|
||
|
||
For example, `{ namespace: "example.computer", activityKey: "desktop.used" }`
|
||
records successful desktop use. Core can map that activity to a Desktop viewer
|
||
for the task's running agent. The authenticated event envelope supplies the run,
|
||
session, source sequence and emission time. Core resolves the company, task and
|
||
agent through the run's existing authorization checks.
|
||
|
||
Both activity identifiers contain 1–160 Unicode code points and exclude C0
|
||
controls and DEL. Callers cannot supply run, task or agent identity in this payload.
|
||
|
||
The ordinary authenticated Runner event is the durable record. Normal source-event
|
||
acknowledgement and replay deduplicate delivery. Existing run-read authorization
|
||
and secret redaction apply to saved and live events. Recording reported activity
|
||
is not independent proof that it happened and grants no resource permission.
|
||
|
||
The native transport forwards runtime activity as a run-scoped fact. It retains
|
||
the authenticated run identity across warm session attachment. Provider messages
|
||
cannot impersonate this transport-owned notification.
|
||
New-run activity waits for warm attachment to finish. A failed attachment does
|
||
not retarget queued activity to the previous run.
|
||
|
||
Runner accepts local activity records only with an opaque context
|
||
captured before the operation. Warm run attachment and process recovery rotate
|
||
that authority. Runner supplies the run identity from its active authenticated
|
||
binding. Optional hint failure does not fail the underlying operation. No output
|
||
text is scanned and no synthetic receipt is written.
|
||
|
||
`GET /api/heartbeat-runs/:runId/events?view=plugin-panels` reads these events and
|
||
returns only installed agent task-panel contributions under normal run-read and
|
||
telemetry-read authorization. Ordinary PRP reads/live delivery retain run-secret
|
||
and current-user redaction. The task page uses one history read on mount/reconnect
|
||
and the existing live transcript stream, then makes a bounded plugin readiness
|
||
check. No panel discovery timer scans runs or provider state while connected.
|
||
|
||
## Post-result failure diagnostics
|
||
|
||
`native.post_result_failure` records `stage: post_result_execution`, a closed
|
||
`category` (`io_not_found`, `io_permission_denied`, `io_failure`, or
|
||
`execution_failure`), and `resultAlreadyAccepted: true`. It requires an accepted
|
||
selected result and exact current, unexpired controller authority. A busy diagnostic
|
||
row lock is skipped. Reporting is best effort under the existing 60-second control
|
||
bound; the sink can finish later and emit a late event. The `native_session_lease_lost`
|
||
control classification stays unchanged. The labels grant no recovery, result,
|
||
Stop, provider-work or payment authority and do not establish a hosted failure cause.
|
||
|
||
## Payload storage
|
||
|
||
Structured events in this guide remain in `heartbeat_run_events`. When the
|
||
instance's existing file storage uses S3, new stdout/stderr NDJSON logs use the
|
||
same stack bucket and configured prefix under `run-logs/`, with existing run
|
||
references in Postgres. Local-disk instances remain local. Raw provider
|
||
traces have their own restricted, encrypted object path. See
|
||
[run evidence storage](run-evidence-storage.md) for batching, durability,
|
||
configuration, migration, retention, and qualification boundaries.
|
||
|
||
### Event commit stall diagnostics
|
||
|
||
A Runner event commit stall records the first safe error code in the error and
|
||
up to three nested `cause` objects. This retains database and transport codes
|
||
when an adapter wraps the original error. Each code has the existing 80-character
|
||
identifier limit. If no safe code exists, the diagnostic uses the outer error's
|
||
safe name. It never copies exception messages, SQL, or query arguments. Cyclic or
|
||
long cause chains stop at the same four-object limit.
|
||
|
||
This only improves the recorded cause. Event acknowledgement, replay integrity,
|
||
the bounded stall policy, and operator recovery requirements stay unchanged.
|
||
|
||
A later successful continuation does not establish the cause of an earlier
|
||
failure that recorded only `Error`.
|
||
|
||
### Failure cause of generic run failures
|
||
|
||
A run that fails with a generic code (`adapter_failed`, `setup_failed`) keeps
|
||
that code, because callers compare against it, and records
|
||
`resultJson.failureCause`: one identifier of at most 80 characters
|
||
(`[A-Za-z0-9_.:-]`). For a thrown error it is the provider's HTTP status
|
||
(`provider_http_429`) or category, else the first safe code in the error and up
|
||
to three nested `cause` objects (`40001`, `ECONNRESET`, the same extraction as
|
||
the Runner's event commit stall diagnostics), else the error's class name. For
|
||
an adapter result without a code it is the error family, `signal_<name>`,
|
||
`process_cancellation_failed`, `exit_code_<n>` or `adapter_reported_error`. It
|
||
never copies message text. The server's failure log line carries the same
|
||
cause, and the board's run list shows it, for example "adapter failed (40001)".
|
||
Older rows have no cause and read as before.
|
||
|
||
Run error text keeps SQL statements off the board: a Drizzle `Failed query:`
|
||
statement and its parameters, or unwrapped statement text with `$n`
|
||
placeholders, is replaced with "Database query failed (<cause>)". The wrapper
|
||
text before the statement is kept.
|
||
|
||
|
||
## Chat outcome read receipts
|
||
|
||
Two run-log events record what the conversation agent saw before it replied.
|
||
`chat_handoff.completed_read` records that a run read a task it handed off, in
|
||
the done state, with the task ID and status version. `chat_interaction.declined_read`
|
||
records that a run listed its conversation's interactions and saw a declined tool
|
||
action, with the issue ID, interaction ID and status. Neither carries task text,
|
||
tool arguments or results.
|
||
|
||
A completion or decline continuation that an earlier reply already reported
|
||
creates no run. The server checks for a reply by the same agent in the same
|
||
conversation, posted after the outcome, from a run that has one of these receipts
|
||
for that exact outcome. A completion reply must also link the task. The
|
||
completion delivery is then marked delivered with that reply as its response,
|
||
and a deferred wake ends `skipped` with the reason "Already reported by an
|
||
earlier reply in this conversation".
|
||
The conversation's handoff listing (`GET /api/issues/{id}/chat-handoffs`) shows
|
||
that reply as `completionReportCommentId` once the latest completion is delivered.
|