From 171e70c11de96387c7161d660e7f6890148d3e3e Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 15 Nov 2026 10:06:05 -0600 Subject: [PATCH] refactor: drop span and run-log event catalogs from the observability docs (#21779) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `doc/observability.md` and `doc/run-log-events.md` described every OpenTelemetry span and every run-log event > - `rules/data-paths.mdc` told authors to update `doc/run-log-events.md` for each event, so both files only grew: 2,313 and 1,263 lines > - Each span and event is already defined next to the code that writes it, and its name is a literal string that a search finds in one step > - This pull request removes the per-span and per-event catalogs and keeps only what code does not say > - The benefit is two short docs that do not drift from the code, and no rule that makes them grow again ## Linked Issues or Issue Description Refs #21670, #21700, and #21743 (the same cleanup for other docs). **Issue type** Outdated or confusing documentation. **Where is the issue?** `doc/observability.md`, `doc/run-log-events.md`, and `rules/data-paths.mdc` **What's wrong?** The two docs repeated span names, span attributes, and event payloads that the code defines. Operator setup started at line 138 of `doc/observability.md`. The rule required a doc edit for every new run-log event. **Suggested fix** Keep operator setup and the Sentry privacy contract. Point to the code for span and event definitions. Remove the rule line that requires a doc entry per event. ## What Changed - `doc/observability.md` is now 619 lines. It keeps: how to enable tracing, `service.version` resolution, scope, how to enable Sentry, DSN delivery, environment attribution, privacy settings, server request data, the core of "Run failure context", browser data, fail-open behavior, the default capture set, and operator responsibilities. Kept text is unchanged except the intro and two sentences noted below. - Deleted from `doc/observability.md`: budget, Slack receipt, task detail, and browser tracing notes; controller preparation phases; native Runner spans; sandbox startup spans; sandbox duplex transport; shared skill preparation; invocation attribution; and about 500 lines of per-failure detail in "Run failure context". - Replaced that per-failure detail with one paragraph. It names the failures that `unexpected` mode keeps out of Sentry, and says the remaining diagnostic contexts are defined in `server/src/sentry.ts`. "Default capture set" now says the same. - `doc/run-log-events.md` is now 23 lines. It says what the run log is, where events are defined and appended, and keeps the payload storage note. - `rules/data-paths.mdc`: the run-log line now says to define an event where it is written, not in a doc. The span-allowlist line now says the allowlist is in the code that opens the span. The review levels and globs do not change. - Repointed links in two feature maps, two agent-access map files, `tests/runner-e2e/SUITES.md`, `doc/runner-api-tools.md`, and the Telemetry README. Updated the `honeycomb-tracing` skill and one code comment in `packages/adapter-utils/src/http2-bridge-server.ts`. ## Verification - A script built the new `doc/observability.md` from line ranges of the old file and asserts the first line of each range. - A link check on all changed Markdown files found 0 missing files and 0 missing anchors. The remaining inbound anchors (`#enabling-tracing`, `#sentry-error-monitoring`) still exist. - No test reads either doc. - `node scripts/check-maps.mjs` passes. - `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail. - `pnpm verify` was not run. This worktree has no installed dependencies. The only code change is a comment. ## Risks - The privacy contract no longer lists each run-failure diagnostic context field. `server/src/sentry.ts` is now the list. A reviewer of a Sentry change must read that file. - No browsable list of span names or run-log event names remains. A search for the quoted name finds the producer. - `.greptile/files.json` still scopes `rules/data-paths.mdc` to both docs. Both files still exist, so it does not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code desktop, with tool use. Earlier analysis in this session used Claude Fable 5.2. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge πŸ€– Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .agents/skills/honeycomb-tracing/SKILL.md | 11 +- .../references/trace-design.md | 6 +- CONTRIBUTING.md | 2 +- doc/observability.md | 1752 +---------------- doc/run-log-events.md | 1273 +----------- doc/runner-api-tools.md | 2 +- maps/agent-access-map/README.md | 2 +- maps/agent-access-map/runs-adapters.md | 3 +- maps/feature-map/chat-channels.md | 3 +- maps/feature-map/runs-adapters.md | 3 +- .../adapter-utils/src/http2-bridge-server.ts | 5 +- packages/shared/src/telemetry/README.md | 5 +- rules/data-paths.mdc | 6 +- tests/runner-e2e/SUITES.md | 2 +- .../runner-e2e/source-sdk-public-inputs.json | 2 +- 15 files changed, 80 insertions(+), 2997 deletions(-) diff --git a/.agents/skills/honeycomb-tracing/SKILL.md b/.agents/skills/honeycomb-tracing/SKILL.md index 587bfa5dbe4..35494334f46 100644 --- a/.agents/skills/honeycomb-tracing/SKILL.md +++ b/.agents/skills/honeycomb-tracing/SKILL.md @@ -22,11 +22,12 @@ replaying tasks, changing credentials, or deploying instrumentation. | What is a page's real-user load time or SLO on a hosted instance? | Browser `page.load` spans in [browser performance tracing](../../../doc/browser-performance-tracing.md) | | Which CI jobs or test files delay the whole suite? | [CI profiling](references/ci.md) | | How should a trace be organized so a person can understand it? | Human-readable waterfall guidance below and [trace design](references/trace-design.md) | -| Where should a new span go? | Instrumentation guidance below and [Observability contract](../../../doc/observability.md) | +| Where should a new span go? | Instrumentation guidance below and the code that opens the nearest existing span | -Repository-relative paths and commands assume the checkout root. The maintained -contracts are `doc/observability.md` and `doc/ci-profiling.md`; recheck them and -the owning code when a deployed version differs. +Repository-relative paths and commands assume the checkout root. Span names and +attributes are defined in the code that opens each span; `doc/observability.md` +covers only operator setup and Sentry. `doc/ci-profiling.md` covers CI traces. +Recheck the owning code when a deployed version differs. Keep the repository's three paths distinct: @@ -178,7 +179,7 @@ explain one slow scan. Observability must not retry the operation, replace its return/error, change authorization/locks, or break execution when the tracer fails. Preserve root sampling and the explicit endpoint gate even if another library registers an -OTel provider. Update `doc/observability.md` when the trace contract changes. +OTel provider. Update `doc/observability.md` only when operator setup changes. Run focused tests for endpoint-off behavior, parentage/concurrent-run isolation, failure handling, and the changed seam. Use real exported evidence to claim that new spans appear in Honeycomb; a unit-test tracer alone does not prove ingestion. diff --git a/.agents/skills/honeycomb-tracing/references/trace-design.md b/.agents/skills/honeycomb-tracing/references/trace-design.md index cfcde0bb07b..ecd0028d79b 100644 --- a/.agents/skills/honeycomb-tracing/references/trace-design.md +++ b/.agents/skills/honeycomb-tracing/references/trace-design.md @@ -2,9 +2,9 @@ Read this when adding, reorganizing, or reviewing a waterfall's hierarchy. The goal is to understand the work at a glance, then expand the slow phase to -find its cause. Follow the current -[Observability contract](../../../../doc/observability.md) for exact names, -roots, lifecycle boundaries, sampling, and allowed attributes. +find its cause. Follow the code that +opens the nearest existing span for exact names, roots, lifecycle boundaries, +sampling, and allowed attributes. ## Organize by the work being done diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e5c52b54c5d..62130263397 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -122,7 +122,7 @@ and coverage requirements. This repo has three separate data paths: Telemetry, Observability, and the run log. See [`rules/data-paths.mdc`](rules/data-paths.mdc) for the full definitions and the review level each path needs. -If your change adds, removes, or modifies emitted telemetry events, update the [Telemetry Data Contract](packages/shared/src/telemetry/README.md) in the same PR. Keep clients emitting raw dimension values and avoid documenting or relying on private delivery details. If your change adds, removes, or modifies an OpenTelemetry span or span attribute, keep the change inside the closed span-attribute allowlist in `packages/adapter-utils/src/acpx-engine/startup-timing.ts`. If your change adds or modifies a run-log event, update `doc/run-log-events.md` in the same PR. +If your change adds, removes, or modifies emitted telemetry events, update the [Telemetry Data Contract](packages/shared/src/telemetry/README.md) in the same PR. Keep clients emitting raw dimension values and avoid documenting or relying on private delivery details. If your change adds, removes, or modifies an OpenTelemetry span or span attribute, keep the change inside the closed span-attribute allowlist in `packages/adapter-utils/src/acpx-engine/startup-timing.ts`. Define a new or changed run-log event where it is written; do not list it in a doc. ### Paperclip Gates Must Pass diff --git a/doc/observability.md b/doc/observability.md index e2a8d6f8746..77a0b1d8e66 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -1,10 +1,16 @@ # Observability -This document is the Observability contract. It covers the OpenTelemetry -trace path, the opt-in Sentry error-monitoring path, and two local -instrumentation contracts; see the +This document covers how an operator turns on the two opt-in observability +paths, OpenTelemetry traces and Sentry error monitoring, and what Sentry events +may contain. See the [Telemetry Data Contract](../packages/shared/src/telemetry/README.md) for the -separate first-party event system. +separate first-party event system, and `rules/data-paths.mdc` for how the paths +differ. + +It does not list span names or attributes. Those are defined next to the code +that opens each span; search for the span name, or for `getTracer(` and +`startActiveSpan(` under `server/src` and `packages`. Add a new span in code +without adding it here. Paperclip ships with **opt-in** OpenTelemetry auto-instrumentation for the server process. When activated it produces **traces only** β€” no metrics and no @@ -42,99 +48,6 @@ the application healthy. For Honeycomb, configure the ingest key in the [Honeycomb authentication](https://docs.honeycomb.io/api/query-annotations/authorization). Keep credential values out of logs and diagnostic reports. -## Budget admission, accounting, and enforcement - -The operator-gated `paperclip.budgets` tracer records -`budget.admission.inspect`, `budget.admission.reconcile`, -`budget.admission.evaluate`, `budget.admission.reserve`, -`budget.accounting.transaction`, `budget.accounting.settlement`, `budget.accounting.lock_wait`, -`budget.exposure.evaluate`, `budget.enforcement.deliver`, -`budget.enforcement.scope`, `budget.cancellation.deliver`, and `budget.accounting.recover`. -Inspection and admission spans cover read-only eligibility. Paid reservations, -receipts and all five settlement owners avoid the company accounting mutex. -`budget.admission.reconcile` remains a historical span name; eligibility no longer -publishes budget transitions. `budget.accounting.lock_wait` still describes the -legacy low-frequency configuration/control wrapper, not paid settlement. -Transaction spans include commit time. Settlement spans cover the atomic ledger, -reservation and owner acknowledgement. Acquisition duration is not a pure -PostgreSQL lock-wait measurement: it can include network and Node scheduling. -Enforcement spans cover asynchronous evaluation/delivery; cancellation delivery -covers the external Stop attempt, separate from provider inference. - -The closed attribute list is `budget.policy_count` (applicable policy count), -`budget.enforcement.queue_delay_ms` (time since pending intent/retry), and -`budget.accounting.owner` (`run`, `decision`, `classifier`, `fast_response`, -`browser`). The owner tag identifies the shared settlement span's emitter. -These spans contain no prompts, usage receipt bodies, credentials or provider -error bodies. They remain gated by the operator OTLP endpoint and do not change -Paperclip's separate first-party Telemetry contract. - -## Slack receipt reactions - -Slack stages its eyes reaction in the same transaction as the authorized task, -comment, message link and wakeup intent. Transport starts after that commit, -independently of attachment preparation, thread subscription and scheduler -acceptance. Eyes acknowledge durable receipt; they do not certify agent startup. -The existing action worker retains retries, credential fencing and exact-source -cleanup when the final response is published. - -The Slack adapter also skips the SDK's two-second polling wait for optional link -unfurls before message dispatch. Text, formatted content, link URLs and file -download handles remain available. Paperclip does not consume that optional -preview metadata for task admission. - -With an operator OTLP endpoint, the `paperclip.chat` tracer records -`chat.slack.receipt` around each Slack reaction API attempt. Its closed attributes -are `paperclip.chat.receipt.operation` (`add` or `remove`), -`paperclip.chat.receipt.outcome` (`confirmed` or `failed`), -`paperclip.chat.receipt.queue_ms` (action creation to transport), and -`paperclip.chat.receipt.delivery_age_ms` (durable inbound receipt to provider -confirmation or failure). `already_reacted` confirms an idempotent add. -Delivery age excludes Slack-to-webhook transit and SDK parsing; the span duration -measures provider I/O. A failed attempt is not an acknowledgement latency sample. -No content, identifiers, credentials, URLs or exception text are added. -Without the endpoint this instrumentation does no tracer lookup. Tracer errors -cannot change reaction transport or settlement. - -## Task detail loading - -`GET /api/issues/:id` returns `Server-Timing` with `paperclip_issue` for the -whole handler and `issue_` for its reads. The total includes the final -execution-blocker read after recovery revalidation. Concurrent phases overlap; -do not sum them to obtain request duration. - -With the existing operator-configured OTLP endpoint, the same reads produce -`issue.read.` child spans through the `paperclip.issue-read` tracer. -There are no custom attributes, identifiers, content, or exception messages. -The closed phase names in `server/src/services/issue-read-timing.ts` are: -`lookup`, `authorization`, `project_goal`, `ancestors`, `mentions`, `documents`, -`relations`, `blockers`, `review`, `references`, `handoff`, `retry`, `recovery`, -`cases`, `inbox`, `channel`, `workspace`, `work_products`, `created_from`, -`execution_blocker`, `relation_recovery`, `revalidate_recovery`, and -`mentioned_projects`. -Without an OTLP endpoint, spans remain no-ops. This adds no first-party -Telemetry events or run-log events. - -The browser's `issue-detail:navigateβ†’content-paint` User Timing measure ends -after the redesigned conversation is revealed and painted, rather than when -the comments request completes while the conversation is still hidden. Durable -messages can paint before run history and plan enrichment. Runtime-only threads -wait for initial output; late history retains the existing scroll-anchor behavior. -Use Chrome's request waterfall and this mark together to distinguish API wait, -request dependencies, and rendering time. A hard-load trace also includes auth, -company selection, and JavaScript startup before the task navigation mark. - -## Browser performance tracing - -Hosted Paperclip Cloud instances can also trace the board UI in the browser: -a `page.load` span for every hard load and client-side navigation (named by -route pattern), Core Web Vitals, and same-origin API request spans whose W3C -trace context joins the server spans above. It is off unless the instance is -cloud managed and `PAPERCLIP_BROWSER_TRACES_ENDPOINT` is set; self-hosted -installs never download the browser SDK. Configuration, the closed attribute -allowlist, the page-load definitions, and example queries are in -[browser-performance-tracing.md](browser-performance-tracing.md). - ## Enabling tracing ### 1. Install the OTel peer dependencies @@ -230,615 +143,17 @@ for `fs`, `dns`, and `net` are disabled by default because they are too chatty for this workload; everything else from `@opentelemetry/auto-instrumentations-node` is on (HTTP, Express, PG, etc.). -This document also holds three local instrumentation contracts: native runner -traces, sandbox startup traces, and sandbox duplex transport instrumentation. -Those sections follow below. - -## Controller preparation phases - -With an operator OTLP endpoint, `paperclip.heartbeat-preparation` emits active -`heartbeat.prepare.` spans around environment selection, execution -configuration, session preparation, workspace preparation, GitHub environment -and launcher preparation, run output initialization, workspace validation, -runtime services, instruction files, and runtime selection. Database and HTTP -instrumentation can attach child spans during each awaited operation. - -Early context preparation also records `agent_read`, `chat_completion`, -`conversation`, `runtime_state`, `issue_context`, `run_identity`, `trust_preset`, -and `task_context` beneath `heartbeat.prepare`. These time the existing awaited -operations before environment selection; they do not add reads or record content. -The work between those phases is named too: `chat_retry_admission`, -`workspace_selection`, `dependency_readiness`, `issue_checkout`, -`instance_settings`, `workspace_context`, `project_context`, -`plan_wake_routing`, `routine_environment` and `responsible_user`. -`execution_start` times the run and agent status writes and the "run started" -event before run output starts. A repeated read or write has its own span. - -Execution configuration also records separate phases for mentioned skills, -GitHub identity selection, secret resolution, managed AI runtime preparation, -skill inventory, connector assignments, connector skill preparation and delivery, -connection instructions, and agent identity. Their names are -`heartbeat.prepare.configuration_`, using the closed names in -`server/src/services/execution-preparation-timing.ts`. These intervals are inside -`execution_configuration`; do not add them to its duration. They remain available -when a later setup step fails before the provider lifecycle trace starts. - -`task.execution_attempt` owns one invocation of the server executor, from its -admission reads through preparation, provider dispatch, and inline settlement. -Deferred native workspace finalization and reply persistence can finish after -this root ends. It starts a new root trace with a link to the active trigger. It -therefore follows the operator's root sampler even when the HTTP trigger was not -sampled. Async-local ownership parents same-run preparation and nested phases to -the current attempt. Concurrent runs have separate contexts. A phase invoked -without a live matching attempt still starts an independent linked root. - -`task.workspace_finalization` owns each deferred native workspace finalization -check. It starts when the background worker checks the run, including its -admission and ownership checks, and ends when that check returns or fails. -Workspace restore phases nest under this root using the same hashed run ID. -It does not extend an earlier execution attempt or include time waiting for -the background worker to run. Retries have separate roots. A check that finds -no eligible work can end without a restore child. - -`task.response_presentation` owns recovery of an already committed native reply. -It starts a separate linked root because deferred presentation can run after the -execution attempt and workspace finalization have ended. Worker discovery for -a specific run and worker checks include candidate reads under -`response.resolve`. Each discovery or check owns its root; direct recovery and -its worker check share one root when nested. `response.transaction` -includes the recovery transaction; `response.transaction.entry` measures entry -into its callback, including connection acquisition. These spans retain only the -same hashed run ID and never capture reply content. A successful task status is -separate from the time the reply is saved. - -These spans exist before provider dispatch, including setup failures. Preparation phases have -only the existing `paperclip.task.run.run_id` attribute: the first 12 hexadecimal -characters of the run UUID's SHA-256 hash. Failed operations set error status -without recording exception messages, paths, inputs, or returned content. -Search by that hash without a root-name filter to include these preparation -traces; the provider lifecycle trace is created later and can have a different -trace ID. Without an endpoint, tracing remains a no-op. These spans add no -first-party Telemetry events or database run-log writes. - -Workspace preparation also records `heartbeat.prepare.workspace_` spans -for source resolution, provisioning policy, base validation, source access, -provisioning, retained source access, workspace record creation and updates, -task binding, operation binding, configuration records, repository admission, -project repositories, and recovery-root reservation. These intervals are inside -`heartbeat.prepare.workspace`. Repeated bindings or updates have separate spans. -Do not add these intervals to the containing workspace duration. They use the -same run hash and contain no workspace paths, commands, or returned records. - -The `paperclip.native-controller` tracer records -`native.controller.` spans around GitHub access setup, wake binding, -backend construction, connector assignments, assigned MCP discovery, tool -authority, companion selection and Runner binary resolution. It also records -GitHub shutdown, coordinator release, instruction checkpoints and collection, -and warm-session release. Before the coordinator claim it also records -attachment staging, warm controller restore, persistence of the preparation -spans to the run log, and Runner state migration. These intervals can occur between the existing native -preparation, execution and settlement spans. They are controller work, not model -turn time, and carry only the existing run hash. Nested phases overlap their -containing backend or lifecycle intervals; do not add both durations. - -Before provider dispatch, `execution.usage_startup` measures durable usage-recorder -initialization. `execution.mcp_handoff` measures native MCP server resolution or -legacy managed MCP configuration. `execution.dispatch_admission` measures the -final admission gate, including its nested `execution.budget_reservation` span. -The admission span ends after the provider promise is handed off, not after the -provider turn finishes. These spans use only the existing run hash. They close -gaps between preparation and execution; their nested intervals are not additive. - -All preparation timing uses the endpoint gate, even when another library has -registered a tracer provider. Tracer lookup, callback, status or ending failures -cannot repeat work or replace its result or error. Automatic database children -are not guaranteed: an aggregate workspace trace can contain just one span. -If a process exits before an operation settles, its open span may not be exported. - -With an endpoint, `paperclip.controller-startup` records independent -`controller.startup.` root spans for tool reviews, gateway sessions, -delivery-worker readiness, accounting, plugin-worker readiness, native recovery, -hot-restart adoption, orphan reaping, scheduled retries, queued runs, and goal -recovery. These spans have no custom attributes. Each ends when its awaited -phase settles, including failures. They separate controller readiness from -provider startup; they are not agent-run durations. A disabled endpoint takes -the direct work path without a tracer lookup or additional log messages. -Tracing failures cannot repeat recovery or replace its result or error. - -Native recovery and orphan reaping each run a native finalization -reconciliation pass; the periodic recovery loop runs it on every scheduler tick. -With an endpoint, a pass that takes at least one second records, after it ends, -a `controller.native_reconciliation` root span from `paperclip.native-reconciliation` -with one child span per phase: `legacy_workspace_exports`, -`obsolete_policy_reviews`, `automatic_completion_reviews`, `candidate_scan` and -`candidates`. The pass span carries `paperclip.native_reconciliation.scoped` and, -once known, `paperclip.native_reconciliation.candidates` (a count). A failed -pass marks the pass span and the phase that was running with an error status. -Faster passes record nothing. - -Heartbeat configuration preparation overlaps independent reads. Skill preparation -runs alongside connector assignments; standing connection instructions wait for -assignment catalog repairs, but can overlap the skill branch. These spans can -finish out of order. Do not add their durations as serial elapsed time. The -historical `skills.prepare` span ends when the skill branch settles, rather than -when its concurrent branches finish. Each preparation attempt settles its started -branches before it returns an error or releases acquired credentials. - -### Detailed execution phases - -The `paperclip.execution` tracer adds these content-free spans beneath the current -attempt or owning phase: - -| Span family | Measured work | -| --- | --- | -| `workspace.transaction`, `.entry`, `.commit`, `.rollback` | Transaction lifetime, connection acquisition plus BEGIN, and settlement after the callback returns. Entry cannot distinguish pool wait from BEGIN. | -| `workspace.row_lock` | Current workspace row read with its row lock. | -| `workspace.sources.inventory`, `workspace.access.*` | Current selected-record reads, task/run identity, current task and selected-project authorization, and selected-source trust. The execution check is separate from full workspace-record disclosure. | -| `workspace.repository.*` | Repository admission binding and policy revalidation, receipt reads/writes, catalog access, filesystem preparation, directory publication lock boundaries, ownership verification, and Git inspection. See the nested admission hierarchy below. | -| `authorization.base`, `.responsible_user`, `.authorizing_user`, `.task_privacy`, `.project_privacy` | The existing base decision, responsible/authorizing-user intersections, and applicable app-record privacy decisions beneath `workspace.access.task_authorization` or `workspace.access.authorization`. These details are scoped to workspace authorization. | -| `workspace.notification`, `workspace.write` | Notification statements and record writes. | -| `workspace.materialize.*`, `workspace.checkout.*` | Private source-directory setup and identity verification, existing checkout lookup, Git credential resolution, clone, optional pristine-object cache lookup/publication (`cache_lookup`, `cache_publish`), requested-ref checkout, and atomic publication. The enclosing `heartbeat.prepare.workspace_source_materialize` includes the complete source-materialization path. | -| `runtime_selection.*` | Completion contracts, session/recovery reads, profiles, interaction materialization, backend discovery, dynamic skill package staging (`skill_assets`), bootstrap, run/coordinator writes, transaction entry/settlement, and workspace synchronization. | -| `runtime_context.instructions`, `.skills`, `.mcp` | The three concurrent runtime-context branches. Their durations overlap. | -| `runtime_context.asset_*` | Source reads, manifest hashing, cache lookup, cold materialization, and exact file verification on cold and warm paths. | -| `skills.inventory`, `.wait`, `.bundled`, `.releases`, `.folders`, `.local_sources` | Company skill reconciliation, waiting for an existing company refresh, and its bundled imports, release verification, folder, and local-source work beneath runtime skill preparation. Inventory children overlap their parent. | -| `skills.cache.acquire`, `skills.rows`, `skills.version.read`, `skills.repository.read`, `skills.version.materialize`, `skills.version.verify` | Dedicated connection/transaction startup plus advisory-lock acquisition, authorized skill rows, frozen version and repository reads, and exact runtime materialization/validation. Lock acquisition ends before the protected work; version verification is inside materialization. Only bounded file counts are added. | -| `skills.consume` | The caller's delivery and durable version-pinning callback while the cache lock remains held. | -| `workspace_sync.prepare_host_current`, `.prepare_durable_seed`, `.prepare_adopt_remote` | Runtime preparation under the selected inbound mode, including snapshot and staging work. A warm mismatch may produce a second preparation; it is real repeated work. | -| `workspace_sync.source_root`, `.repositories`, `.ignore_rules`, `.snapshot_git`, `.snapshot_baseline` | Physical source capture, managed repositories, ignore rules, Git metadata and baseline content snapshot. | -| `workspace_sync.pack`, `.git_clone`, `.git_archive`, `.overlay_copy`, `.overlay_archive`, `.deleted_manifest` | Temporary shallow Git clone, its archive callback, selected working-tree overlay copy/archive, and the deleted-file manifest. The archive is inside the clone's lifetime; do not sum parent and child. | -| `workspace_sync.seed_generation`, `.seed_lookup`, `.seed_replay`, `.seed_persist`, `.seed_publish`, `.seed_verify`, `.seed_extract`, `.seed_snapshot`, `.seed_cleanup` | Seed identity, cache lookup, replay, durable copies, publication, extraction/re-snapshot verification and temporary cleanup. Cache hits omit cold clone and verification work. | -| `workspace_sync.seed_cache_copy`, `.seed_cache_sync`, `.seed_cache_hash` | Archive linking/copying, disk flush, and checksum work inside cache publication. Recovery-backed archives retain their existing flush, including when a cache link shares the recovery inode. Only a separate cache-only Git seed omits the flush; its bytes are rebuilt on a cache miss. Every reader verifies checksums. These are children of seed publication, not additional elapsed time to add to that parent. | -| `workspace_sync.seed_cache_capacity`, `.seed_cache_evict`, `.seed_cache_locked_publish` | Quota inventory, eviction planning/application, and the publication body after the company lock is acquired. Capacity may run before and inside the lock. A gap after seed verification and before locked publication can include lock acquisition; locked publication itself excludes that wait. Copy/sync/hash and the locked quota/eviction work are nested children. | -| `workspace_sync.seed_read_copy`, `.seed_read_hash`, `.seed_read_touch` | Private cache materialization, verification of the exact reused bytes, and last-used bookkeeping including company-lock acquisition. All are nested in seed lookup. A long lookup alone does not identify copy, hashing, or lock contention as its cause. | -| `workspace_sync.git_index_refresh` | Fully awaited stat-metadata refresh after restoring a fresh OpenCode sandbox checkout. It shares the inbound barrier and command timeout; dirty working-tree content stays unstaged. Recovery and retained sandboxes skip it. | -| `workspace_sync.inbound_barrier`, `.stage_workspace`, `.transfer` | The inbound scheduler barrier, whole workspace task, and provider `syncIn` call (including provider guards, upload and post-upload extraction). Parallel tasks overlap; provider transfer/exec children give the finer transport split when available. | -| `workspace_sync.progress` | Awaited progress/status reporting, including the existing lifecycle run-log writes. These are not file transfer work. | -| `workspace_sync.run_read`, `.descriptor_directory`, `.descriptor_read`, `.archive_hash`, `.warm_stamp`, `.durable_seed_verify`, `.manifests_persist`, `.descriptor_write`, `.reference_write` | Saved run state, recovery descriptors, archive hashing, remote warm-stamp check, manifests and final descriptor/reference persistence. | -| `connectors.browser_use.resolve`, `connectors.speko.resolve`, `connectors.slack.resolve`, `connectors.agentmail.resolve` | Each sequential connector resource resolver beneath configuration or native-controller assignment resolution. Empty/denied assignments still have measurable work. | -| `connectors.browser_use.connection`, `.catalog`, `.grant` | Browser Use candidate reads, the full authorized catalog listing currently used to discover Browser Use resources, and grant checks. The catalog can contain work for other connectors. | -| `connectors.discovery.scheduled`, `.queue`, `.build`, `.agent`, `.connections`, `.github`, `.slack`, `.speko`, `.policy`, `.on_demand_policy` | Discovery scheduler lifetime and wait, catalog construction, agent validation, descriptor sources, and policy decisions. Queued work retains the initiating run's trace context even when another run releases the scheduler slot. | -| `connectors.github.catalog`, `.connections`, `.grants`, `.grant_members`, `.membership`, `.refresh`, `.credentials`, `.repositories`, `.grant_lock`, `.grant_write`, `.webhooks` | Authorized GitHub catalog construction, current connection/grant reads, credential and metadata refresh, repository pagination, grant transaction row lock/write, and parallel webhook binding calls. | -| `connectors.github.refresh.transaction`, `.entry`, `.commit`, `.rollback` | The existing managed-grant refresh transaction and settlement. Entry combines connection acquisition and BEGIN. | -| `connectors.slack.candidates`, `.authority`, `.profile`, `.settings`, `.identity`, `.credentials`, `.revision` | Slack endpoint enumeration, authority revalidation, enabled profile, settings, accepted requester provenance, credential reads and authorization revision. | -| `connectors.slack.authority.transaction*`, `connectors.slack.catalog.transaction*`, `connectors.slack.catalog.lock`, `.entries`, `.write`, `.profile` | Slack authority and catalog transaction entry/settlement, advisory lock, catalog reads/updates and existing-profile lookup. Entry combines pool acquisition and BEGIN; it does not isolate pool wait. | -| `execution.runtime` | Runtime dispatch, usage, workspace finalization and runtime cleanup; this includes work after provider execution. | -| `execution.workspace_finalize` | Best-effort native workspace finalization, including restore recovery, copyback and commit. Its nested restore and commit spans measure distinct steps. | -| `execution.workspace_ownership`, `.workspace_ownership_wait`, `.workspace_restore`, `.workspace_commit`, `.native_finalize` | Ownership coordination including its action, acquisition before the action, copyback, final snapshot/stamp/descriptor persistence, and native settlement. | -| `completion.*` | Completion handler, run-log finalization, terminal persistence, accounting, liveness, and continuation summaries. | -| `response.resolve`, `.authorize`, `.add_comment`, `.transaction*`, `.row_lock`, `.write`, `.publish` | Response selection, existing authorization, comment transaction including commit or rollback, row wait, insert, and external publication enqueue. | -| `execution.realtime` | Synchronous live-event and plugin publication dispatch. It does not measure browser receipt or paint. | - -For bounded native OpenCode diagnosis, set -`PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` as an agent/environment variable. -The existing configured-environment projection carries it into the native Runner. -It defaults off. The Runner creates an authenticated loopback OTLP sink and enables -the pinned OpenCode runtime's internal tracing. The sink discards logs, resource -attributes, span attributes, events, and unknown span names. It retains only closed -phase names, wall timestamps, and success/failure, with a 1 MiB request limit, -128 retained measurements, and 16 measurements per phase. No operator OTLP endpoint -or ingestion credential is passed to OpenCode. - -For fresh remote diagnostic runs, the selected pack must contain the current -build's OpenCode proxy. An otherwise compatible image pack can predate the -collector. Such a pack cannot satisfy the diagnostic request. The existing -verified pack preparation path selects or stages the current proxy; it can add -startup time or fail admission when matching runtime assets are unavailable. -Ordinary runs retain compatible image-pack reuse. Recovery retains the pack -identity already admitted for that session. - -Available phases cover context/instruction loading, skill discovery, tool and model -resolution, model streaming (`ai.streamText`, including stream consumption), Git -discovery/default branch, snapshots, dependency installation, and plugin startup. -These intervals overlap; do not sum them as elapsed time. At each terminal turn, -already-received measurements travel in one `harness.diagnostic` event through the -existing durable PRP channel. Late exports may be absent; collection never waits -for a provider flush or delays terminal publication to complete the diagnostic. -Late spans from a prior completed turn are discarded at the next boundary. - -The server validates the closed fields again and records them under a foldable -`provider.runtime` group inside `native.session.execute`. The normal operator -endpoint gate controls Honeycomb export. The single diagnostic batch stays in the instance run log. Its OTel projection -does not write duplicate phase rows or await additional database work before -terminal consumption. Without an endpoint, no provider child spans are exported. Raw OpenCode OTLP must -never be sent directly to the operator endpoint: it can contain prompts and -authorization attributes before the loopback sink removes them. - -Detail phases emit at most 16 spans of each name per attempt. Larger receipt and -asset inventories continue under the containing group without unbounded per-row -or per-file spans. Adding detail spans does not itself change work, query order, -authorization, or locks. -Manual transaction and dispatch boundaries retain their actual lifetime. -Asset writes and verification may add only non-negative integer -`paperclip.execution.fileCount` and `paperclip.execution.bytes` attributes. -The closed keys are declared in `EXECUTION_PREPARATION_SPAN_ATTRS` in -`packages/adapter-utils/src/acpx-engine/startup-timing.ts`. -Repository checkout cache phases also carry the closed -`paperclip.execution.repositoryCache.outcome` value: `disabled`, `hit`, `miss`, -`published`, `already_present`, `contended`, `capacity`, `budget`, `invalid`, or `error`. -The clone span carries the boolean -`paperclip.execution.repositoryCache.reference_supplied`, including `false` for -a fallback clone. A lookup hit means a validated cache was found, and supplying -a reference does not prove Git avoided a download. These attributes carry no -cache keys, remote URLs, paths, refs, credentials, or error text. - -Pristine repository-object cache version 2 mirrors the fresh checkout's refs, -including remote-tracking refs. Version 1 copied their objects into a bare -repository but omitted the remote-tracking tips. Git could then download those -objects again during negotiation. The version is part of the cache key, so the -first checkout creates a new generation and later checkouts can reuse its tips. -Existing generations remain under the same storage limits. Cached refs stay -private to the company and credential scope. Task checkouts still authenticate -online, use the current remote refs, and dissociate their objects from the cache. - -Other execution phases, including manual boundaries, carry only the existing -hashed run ID. Exceptions set status without exporting their messages. - -#### Repository admission - -Repository preparation keeps repeated current-identity and permission checks -visible. An existing ready checkout still validates its receipt, origin, pin, -current binding and acquisition authority. A ready checkout can omit clone work -while retaining catalog, database, filesystem and Git inspection work. - -The hierarchy folds those operations beneath the admission that initiated them: - -```text -heartbeat.prepare.workspace_repository_admission - workspace.repository.binding (repeated at publication boundaries) - workspace.repository.db.issue / db.workspace / db.run - workspace.repository.paths (when the destination must be checked) - workspace.repository.current_policy - workspace.repository.db.agent / db.project / db.run - workspace.access.execution - workspace.access.task_read / agent_read / run_read - workspace.access.task_authorization - authorization.base / responsible_user / authorizing_user / task_privacy - workspace.access.workspace_read / source_read / projects_read - workspace.access.authorization (selected project) - authorization.base / responsible_user / authorizing_user / project_privacy - workspace.access.trust - workspace.access.trust.project_read - workspace.repository.receipts - workspace.repository.paths - workspace.repository.catalog - connectors.github.catalog (when GitHub catalog access is needed) - connectors.github.connections / grants / grant_members / membership - connectors.github.refresh (managed grant, including existing retry) - connectors.github.credentials - connectors.github.repositories (metadata and repository pagination) - connectors.github.refresh.transaction - connectors.github.refresh.transaction.entry - connectors.github.grant_lock / grant_write - connectors.github.refresh.transaction.commit / rollback - connectors.github.webhooks (existing parallel binding calls) - connectors.github.credentials / repositories (legacy or unmanaged connection) - workspace.repository.fs_prepare - workspace.repository.ignore_rules - workspace.repository.publication (one per concrete receipt) - workspace.repository.lock_wait - workspace.repository.critical_section - workspace.repository.checkout_identity - workspace.repository.state_write - workspace.repository.ownership_read / ownership_verify - workspace.repository.acquisition (repeated current acquisition check) - workspace.repository.db.issue / db.workspace - workspace.repository.current_policy - workspace.repository.materialize - workspace.checkout.lookup - workspace.checkout.git_auth / git_clone / git_ref / publish (cold path only) - workspace.repository.git_origin / git_branch / git_commit - workspace.repository.binding (same current checks as above) - workspace.repository.ownership_write / state_write - workspace.repository.lock_release -``` - -Optional branches appear only when the corresponding existing operation runs. -The first binding check precedes the receipt lookup; an empty receipt list returns -without catalog, path, staging or publication work. Later binding and acquisition -checks revalidate current authority rather than authorizing historical readers. -The retired filesystem-history inventory, root-policy reservations and reader -authorization phases are no longer part of admission. - -Catalog groups retain the existing sequential connection processing and any -credential refresh or pagination it requires. The initial four catalog reads -can overlap, as can webhook-binding requests. Automatic HTTP spans, when enabled, -sit beneath the owning credential, repository-metadata or webhook group. They -do not require a separate execution-attempt root. `credentials` includes its -existing secret/credential lookup and any OAuth refresh; `repositories` includes -the whole metadata or token-repository listing, rather than one invented span -per repository. No repository list or credential value is attached to a custom -span. - -Database read/write spans measure the complete awaited client operation. Their -duration can include pool acquisition, driver and network time, database work, -and database lock waits. They do not by themselves identify a slow SQL plan or -prove a pool wait. `lock_wait` instead measures the directory publication helper -up to entry into its callback, including canonicalization and lock acquisition. -`critical_section` measures the callback after acquisition. `lock_release` starts -after that callback settles and ends when the helper finishes releasing its -lock. A callback failure marks the critical section and publication as failed; -a successful release remains successful. These directory boundaries are not -PostgreSQL pool or row-lock measurements. - -Parents include their children, so do not add binding, catalog or publication -durations to their descendants. Repeated guards and concurrent HTTP requests -must be read in timeline order. Detail names remain capped at 16 occurrences per -attempt; later operations still execute under their enclosing group. A missing -leaf after that cap, or absent automatic driver instrumentation, does not prove -that no work occurred. - -The `authorization.*` details appear only inside a workspace task/project -authorization group. Unrelated startup decisions do not emit those details or -consume their per-attempt budget before repository admission reaches its current -task checks. - -These additions use the existing opt-in OpenTelemetry path. With -`OTEL_EXPORTER_OTLP_ENDPOINT` unset or blank they perform the original work -without consulting a tracer, even if a provider is globally registered. They add -no Telemetry events, run-log writes or span-attribute keys. The closed attribute -allowlist above remains unchanged: no raw SQL, paths, company/task/workspace or -receipt IDs, repository URLs, refs, credentials, request bodies or error text. -Tracer failures must preserve the original result or error and must not replay -authorization, writes, Git commands or lock callbacks. - -A committed comment transaction establishes durable server storage. A publication -enqueue establishes pending delivery, and realtime dispatch establishes a server -notification call. None establishes browser paint or external delivery. The attempt -can continue through later bookkeeping and resource cleanup. A later recovery -invocation has its own attempt; this does not manufacture one continuous trace -across queueing, process restarts, or an unavailable controller. - -### Connector preparation work - -Run configuration resolves the four connector families concurrently and preserves -registration order. Browser Use assignment reads only active Browser Use -connections and their eligible catalog entries. It applies the same tool policy, -grant, company, and restricted GitHub guest checks without joining the full tool -discovery queue or resolving unrelated Slack, GitHub, and Speko tools. - -Native startup registers direct connector tool definitions from the run's pinned -skills. It does not repeat assignment discovery. These definitions do not grant -access: each invocation still resolves current assignments and enters the existing -permission and approval gateway. A retained session may advertise a tool whose -access was revoked; the call fails under the current authorization state. - -Slack assignment checks stored catalog hashes against the bundled definitions. -A current catalog and existing profile need no catalog transaction or advisory -lock. Current profile bindings and entries still determine availability, followed -by one live task authority check. Missing or changed catalog definitions use the -existing locked reconciliation path, with authority checks before and after the -update. Disabled catalog entries and operator-removed bindings remain disabled. -No authorization result is cached across calls. - -### Reading a slow startup in Honeycomb - -Filter `paperclip.task.run.run_id` to the first 12 hex characters of -`SHA256(run UUID)`, then inspect the attempt's waterfall with its -`service.version`. An agent/model setting or the repository's checked-out commit -does not establish the server revision. The detailed spans are available only -after this instrumentation is deployed; an earlier aggregate cannot establish -which child was slow. - -Start beneath `heartbeat.prepare.workspace` with source materialization versus -provisioning, repository admission, and environment acquisition/realization. -Source materialization now separates filesystem setup, credential resolution, -Git cloning and source identity checks. Under -`runtime_selection.workspace_sync`, compare snapshot work, the seed-cache path, -archive construction/verification, provider transfer and descriptor persistence. -Use span start/end times to identify serial work and uncovered gaps; summing -nested or concurrent durations overstates elapsed time. - -For connector delay, compare the four `connectors.*.resolve` spans beneath each -assignment-resolution parent. A long Browser Use `.catalog` can be discovery -queueing, connection descriptors, another connector's authority checks, or policy -evaluation. A long Slack transaction entry, lock or credential span calls for a -different investigation. Configuration and native backend setup currently resolve -assignments separately; the trace records both without caching authorization or -changing access decisions. - -New workspace detail callbacks have fixed names and remain trace-only. They do -not create additional lifecycle events or awaited run-log writes. Existing coarse -progress writes remain intact and appear under `workspace_sync.progress`, so -reporting overhead is visible alongside useful work. No per-file spans, paths, -repository URLs, connector resource IDs, tool schemas, credentials or exception -messages are added to the export. The endpoint gate and per-phase span limit -above still apply. - - -## Native Runner Trace Spans - -Provider-pack fallback staging separates `provider_pack.archive.verify` -(controller archive identity), `provider_pack.archive.probe` (remote helper support), -`provider_pack.archive.reuse` (copy and hash image files), -`provider_pack.archive.verify_missing` (verify only layers needing upload), -`provider_pack.archive.transfer` (compressed file -transfer), and `provider_pack.archive.extract` (received hash check and extraction). -They are children of the existing pack selection/upload path and also appear in -the local run timing log. Their fixed names add no paths, credentials, or content -attributes. The operator endpoint gate remains unchanged. Source installs without -prebuilt archives retain the existing directory upload span. -`provider_pack.directory.verify` measures expanded source verification when -directory staging is required. - -Archive installation verifies the archive bytes and then the installed pack -against the selected manifest. It does not hash the unused expanded controller -tree. Directory installation still verifies that tree before upload, including -when the remote cannot extract an available archive. The final installed-pack -verification applies to both paths. A slow controller tree scan can block the -server event loop and inflate overlapping asynchronous span durations; a copy -span alone does not establish that the disk spent all that time copying. - -The native provider lifecycle emits one foldable OpenTelemetry trace, linked to -the server attempt described above. This is -the native-run trace schema version `3`. `task.provider_session` is the native -provider lifecycle root; it includes preparation and provider execution, but -ends before outer workspace persistence. The heartbeat-owned local run-log -`task.run.measured` records terminal elapsed time including durable copyback; -every other native span carries a real OpenTelemetry parent context rather than -only a descriptive `parentName` field. - -The native lifecycle root uses the operator's root sampling policy. It links -to the matching live server attempt (or a valid active trigger) without inheriting -its sampling decision. This is a link because historical ingress and queue spans -can predate this invocation. It does not reparent earlier work beneath a later span. -An unsampled HTTP request therefore cannot suppress an otherwise sampled -background run. An operator who disables root sampling still gets no spans. -Without an OTLP endpoint, tracing remains silent and does not export data. - -The run link in Honeycomb matches `task.provider_session` for native runs, `task.run` for legacy adapters, -`task.execution_attempt` for server preparation and settlement, and -`task.workspace_finalization` for deferred native copyback and `task.response_presentation` -for committed native reply recovery. It retains the exact hashed run-ID and service filters and groups by trace ID. The URL builder uses Honeycomb’s supported `in` operator for the five root names; see the [query specification](https://github.com/honeycombio/honeycomb-mcp#query-specification-for-run_query). This changes only the query link. The operator endpoint gate and span attribute allowlist remain unchanged. - -The canonical lifecycle is: - -```text -task.provider_session -β”œβ”€β”€ heartbeat.queue -β”œβ”€β”€ task.prepare -β”‚ β”œβ”€β”€ environment.startup -β”‚ β”‚ β”œβ”€β”€ environment.acquire -β”‚ β”‚ └── environment.workspace.realize -β”‚ β”œβ”€β”€ skills.prepare -β”‚ β”œβ”€β”€ heartbeat.prepare_before_environment -β”‚ β”œβ”€β”€ heartbeat.prepare_after_environment -β”‚ └── native.coordinator.claim -β”œβ”€β”€ native.session.execute -β”‚ β”œβ”€β”€ runner.session.startup -β”‚ β”‚ β”œβ”€β”€ runner.transport.connect -β”‚ β”‚ β”‚ └── runner.artifact.prepare -β”‚ β”‚ β”œβ”€β”€ runner.transport.activation -β”‚ β”‚ β”œβ”€β”€ runner.transport.ready -β”‚ β”‚ β”œβ”€β”€ runner.runtime.stage -β”‚ β”‚ β”‚ └── stage.sync -β”‚ β”‚ β”‚ β”œβ”€β”€ stage.asset.home -β”‚ β”‚ β”‚ β”‚ └── session.checkpoint.restore -β”‚ β”‚ β”‚ β”œβ”€β”€ stage.asset.runtime_context -β”‚ β”‚ β”‚ └── stage.asset.ca_bundle -β”‚ β”‚ β”œβ”€β”€ runner.session.bootstrap | runner.session.resume -β”‚ β”‚ └── runner.turn.submit -β”‚ └── agent.turn -β”‚ β”œβ”€β”€ runner.turn.submitted_to_started -β”‚ β”œβ”€β”€ provider.time_to_first_agent_event -β”‚ β”œβ”€β”€ tool.request.input_stream -β”‚ └── tool.execute -└── task.settle - β”œβ”€β”€ native.result.finalize - └── session.checkpoint.persist -``` - -The tree shows stable semantic groups, not an exhaustive leaf list. Existing -artifact discovery and verification, process launch, PRP/websocket, ingress, -sandbox lease, harness-state, provider, duplex, and `sandbox.exec` spans remain -under the closest group. This keeps detailed diagnosis available while letting -a trace UI collapse the run into preparation, runner startup, agent work, and -settlement. A repeated operation creates another span with the same semantic -name; attempts are not encoded into span names. - -`runner.session.startup` ends at the first durable `turn.submitted` event. A -fresh provider session records `runner.session.bootstrap`; an exact recovered -session records `runner.session.resume`. `agent.turn` begins at -`turn.submitted` and ends at the provider terminal event. `task.settle` begins -at that terminal event and remains open through finalization and checkpoint -persistence, so settlement work does not appear to outlive its parent. - -`provider.time_to_first_agent_event` ends at the first meaningful provider -activity: a nonempty assistant/reasoning delta, a tool announcement with an -execution identity, or an existing assistant/reasoning/tool item start or -completion. Usage updates and empty deltas do not count. This is a first -activity metric, not time to a finished answer. - -`task.time_to_first_token` measures the wait until text is deliverable to the -task transcript for each native dispatch. The first qualifying event is a -nonempty assistant or reasoning text delta, a completed assistant or reasoning -item with text, or an accepted run result with a nonempty summary. Text released -by redaction at item/turn/session completion also qualifies. Item identities and -text fields follow the task transcript projection; item starts, tools, usage, -empty text, result proposals, and duplicate PRP replay do not end the wait. - -The first-text milestone occurs after `task.prepare` and Runner startup. The -timing span is a cumulative interval, not a separate preparation step: - -```text -accepted ingress β†’ queue β†’ task.prepare β†’ Runner startup β†’ provider work β†’ first text -└────────────────────── task.time_to_first_token β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - -The span starts at the earliest accepted input. The accepted comment or attested -question response is the start only when its span is the earliest preparation span. -A comment coalesced into an already-queued run starts after run creation, so that -run starts at run creation from the queue span and reports `run_created`. The -durable wake request's acceptance time replaces the start when it is earlier -(deferred, coalesced and retried wakes). The `anchor` attribute (`comment`, `question_response`, -`wake_request`, `run_created`) names which one. The wake request time is read -after the first text in detached diagnostic work bounded to 2 s (PostgreSQL -cancels a stalled read through a transaction-local `statement_timeout`), so it never adds a -wait before `turn.start` and never delays event delivery. Callers without -preparation history start at native dispatch and report `run_created`. The interval includes queueing, Core preparation, workspace preparation, -Runner startup, provider latency, transport, and durable event ingestion. It ends -in Core's commit callback, when the text is readable through the run-events API, -using the Core clock rather than the provider event timestamp. Browser delivery -and rendering are excluded. The name is retained for existing trace queries; -this measures the first deliverable text, not an exact model token boundary. -Only the first qualifying event in a dispatch emits a span. Its `eventKind` is -`reasoning` for thinking text and `agentMessage` for assistant text or a result. -A dispatch without deliverable text emits no sample, including an early failure. -Same-run recovery retains the original historical start and can emit a new sample -only for newly committed output; it does not reconstruct prior first-token timing. -`task.time_to_first_token` is a child of `task.provider_session` that overlaps -queueing, preparation and execution; do not add its duration to those phases. -No text or provider event identifiers are exported. The existing duration attribute -is `paperclip.native.span.duration_ms`. - -`task.time_to_first_agent_activity` uses the same start and `anchor` as -`task.time_to_first_token`, and ends when Core commits the first event that -`provider.time_to_first_agent_event` counts: a text delta, a tool-call start, or -a reasoning item, whichever comes first. That is when the user sees the agent -respond, so the native speed budgets judge this span (metric -`time_to_first_activity`, measured from run creation). Its `eventKind` is -`agentMessage`, `reasoning`, `toolCall` or `dynamicToolCall`. Time to first text -stays a reported metric without a budget: tool-first adapters (for example -Claude) often call tools before they write any text, so first text includes -model tool work rather than control-plane latency. Like TTFT, it is a cumulative -child of `task.provider_session`; do not add it to the phases it overlaps. - -`tool.request.input_stream` measures the observed request-input window: from -an ACP tool announcement to its last input update before the provider reports -the tool complete. It can include provider buffering and transport. It is not -a claim about the exact end of model generation or the start of API execution. -The span is omitted when no subsequent input update is observed. - -For native control-plane tools, `tool.execute` measures authorization and -execution inside the server authority, including time spent in HTTP requests. -Provider tool IDs and MCP request IDs are separate namespaces; the trace does -not guess a pairing by tool name or arrival order. Both spans belong to the -current run's `agent.turn`, including when a warm provider session is reused. -Their `operation` attribute is a known semantic tool name or `other`. Only a -boolean input-update marker crosses the provider-event boundary; no arguments, -argument hashes, or raw call IDs are exported by these spans. Concurrent tools -have separate durations; overlapping spans must not be summed as wall time. - -The active native scope is also published through the existing asynchronous -runtime-parent seam. Provider execution, plugin, websocket/duplex, daemon, and -sandbox spans therefore inherit the correct branch even when their callbacks -run in another service layer. With no active native scope, those existing seams -retain their documented fallback behavior. - -The root carries only a hashed run id, runtime label, schema version, wall time, -and outcome. Native child attributes use the bounded -`paperclip.native.span.` prefix and a closed key allowlist; values are limited -to finite numbers, booleans, or short strings. Commands, arguments, environment -values, paths, output, credentials, and raw identifiers are discarded by the -trace helper. `task.provider_session.measured` records that same interval in the -local run log, without a duplicate full-width OTel span. The separate heartbeat -`task.run.measured` event is the authoritative end-to-end duration. -Persisted `run.performance.span` events retain the v1 run-log schema and include -`traceSchemaVersion: 3` so local tooling can distinguish the hierarchy. - -Like every span in this document, native-run spans are opt-in. When -`OTEL_EXPORTER_OTLP_ENDPOINT` is unset, the tracer remains a no-op; the local -run-log copy is unaffected. +## Other diagnostic switches + +- **OpenCode performance diagnostics.** Set + `PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` as an agent or environment + variable. It defaults off, and setting the OTLP endpoint alone does not turn it + on. The Runner then records closed OpenCode phase timings through a local, + authenticated sink. No operator OTLP endpoint or ingestion credential is + passed to OpenCode. +- **Browser page-load tracing** on hosted instances needs + `PAPERCLIP_BROWSER_TRACES_ENDPOINT`. See + [browser performance tracing](browser-performance-tracing.md). ## Sentry Error Monitoring @@ -1016,7 +331,7 @@ backend Sentry DSN and the audited SDK. Set `PAPERCLIP_SENTRY_RUN_FAILURES` to: | Value | Reports | | --- | --- | | `off` (default) | No run failure events. Ordinary server exception reporting still uses the backend DSN. | -| `unexpected` | Failed and timed-out runs, plus unexpected started cancellations, with the existing configuration, quota, and external connection filters described below. | +| `unexpected` | Failed and timed-out runs, plus unexpected started cancellations, with the configuration, quota, and external connection filters described below. | | `all` | Every failed or timed-out run, including the known configuration, quota, and external connection failures. Unexpected started cancellations also report. | Missing, blank, or unrecognized values keep run reporting off. Expected operator @@ -1116,509 +431,17 @@ Sentry, so a crash or network outage can lose events. `all` selects failures; it does not promise delivery or export complete transcripts. Detailed output stays in the instance behind the existing authenticated APIs, with its normal retention. -Known missing-secret configuration blockers are kept in the task's run log, -blocked state, and owner recovery action, without a Sentry run-failure event. -The exclusions in this section apply to `unexpected` mode only. -This requires a failed `configuration_incomplete` run in the preparing stage, -a setup-phase report, explicit proof that provider work did not start, and a -nonempty list of recognized missing or inactive secret bindings. Process exit -evidence, unknown binding reasons, secret-provider failures, ambiguous missing -secret-definition lookups remain reportable. This filter does not change task -recovery, credentials, or execution policy. - -Native `native_provider_usage_limit` failures also stay on the task when their -typed provider failure is `quota_exhausted` or `rate_limit`. The task keeps its -blocked state and quota cooldown. Setup failures, interrupted processes, missing -or malformed provider evidence, and other failure categories still report. -`all` mode also reports typed native usage limits. - -AI account selection also stays local when the selection service proves a -missing responsible user, personal default, or selected connection; an -incompatible selection; an account that needs reconnection; or an active -responsible user who lacks the selected credential's human sharing permission. -The sharing rejection keeps its existing HTTP 403 and does not grant access. -Other permission denials, including absent identity and inactive membership, -remain outside this sharing classification. The service marks the original error, -and setup preserves its closed `selectionFailure` reason. -The same failed/preparing/setup/bootstrap checks apply, and resumed native runs -with persisted provider input remain reportable. The generic -`ai_connection_unavailable` wrapper alone is insufficient: database, credential -provider, controller, and unclassified errors still report. Failed runs, blocked -tasks, and account-repair actions remain unchanged. - -A workspace policy conflict also stays local when the resolver proves that an -explicit `local_path` or `non_git_path` project workspace has no repository URL, -Git confirms the selected directory is not a repository, and the task requests -a Git worktree. The run must fail during setup before provider work starts. -Its existing validation error, blocked task and board recovery action remain. -Local Git command failures, permissions, corrupt repositories, fallback paths, -and generic missing or unrestorable workspaces remain reportable. The error text -or `workspace_validation_failed` code alone never suppresses them. - -Recognized external connection failures also stay in the local run result instead -of producing application-error events in Sentry. Their producer records a closed -`connectionFailure` descriptor: schema version, provider, operation, and reason. -It contains no URL, host, path, credential, or remote response. This classification -does not change the run's failed state, task recovery action, retry policy, or -whether remote work may have been accepted. - -- Managed remote Git clones classify known authentication, repository access, - invalid URL, DNS (including machine-specific SSH aliases), connection, TLS, - rate-limit, and upstream failures from the actual clone subprocess. Setup must - fail before provider work starts. Every failed workspace candidate must have - proven connection evidence; a mixed or unknown failure remains reportable. - Local/file clones, credential-provider exceptions, local checkout or copy - failures, process signals, and repository corruption remain reportable. Remote - fetch failures already return workspace warnings; their classification is not - transferred to a later failure resolving a local base ref. -- Hermes gateway configuration validation and the initial create-run request use - exact config, HTTP, or structured transport evidence. Missing configuration, - rejected credentials, unavailable endpoints, rate limits, and upstream outages - remain visible to the run owner. Unknown protocol/parser errors, missing run - identifiers, untyped terminal failures, and timeouts after accepted work remain - reportable. Stream/poll errors retain their existing retry and logging behavior. - -A Codex ACP model capacity failure also stays local. The Codex ACP classifier -must match a typed `service` terminal failure that reports the model at capacity -or the server overloaded. It records `codex_transient_upstream` with the -`transient_upstream` family, and the bounded transient retry owns recovery. The -report must come after setup and carry no signal. Other `service` failures, -untyped capacity text, and Codex CLI transient upstream failures remain reportable. -The run keeps its failed state and its terminal diagnostic. - -A native Codex model capacity failure stays local only while its automatic -retry owns recovery. The run must carry `native_provider_overloaded`. The proof -of the retry is the `native_provider_overloaded` successor run for the same -task, which only the committed capacity status decision creates, and only for -the typed `codex`/`server_overloaded` diagnostic. The run's own `resultJson` is -not required: the live heartbeat's failed-status write replaces it with the -adapter's copy, which lacks the finalizer's diagnostic and finalization fields. -Exhausted or held capacity failures, other retry lanes, setup failures, signals, -and unreadable retry evidence remain reportable. A failed retry reports its own -outcome. - -Expected pre-provider waits for a shared project workspace, a credential rotation -lock, or an exhausted AI connection pool carry a server-owned expected -cancellation receipt. They retain existing retry and lock behavior without a -Sentry cancellation event. A later failed retry still reports its actual error; -the wait code alone is not sufficient to suppress an event. - -MCP tool discovery records recognized outbound HTTP/transport failures through -server-only provenance. These failures preserve the response status, stored -connection health, audit record, local context, and first-party telemetry while -skipping only Sentry. A proven missing OAuth scope returns an actionable 422 -response with its reconnect guidance. Unknown responses, parsing errors, local -callbacks, database errors, and unproven network-looking messages still report. - -The `run_failure` context also includes the recorded process `exitCode` and -`signal`, so a generic adapter error can still distinguish a nonzero exit from -a signal termination. Exit codes must fit the database's signed 32-bit integer; -missing or malformed values become `null`. Signals must match the reporting -host's Node signal constants; missing values become `null` and unrecognized -values become `unknown`. A signal such as `SIGKILL` does not establish who sent -it or prove an out-of-memory kill. These fields do not change error grouping -or run outcomes, and do not include process output or adapter result payloads. - -For a native Codex turn that ends with a structured HTTP 400 model/ChatGPT -compatibility rejection, the run records `native_provider_model_rejected` -and a fixed message that asks the user to select a supported model or connection. -This requires a committed runner terminal for the selected model and the exact -failed turn. Agent text, tool output, unrelated turns, and unknown provider -errors do not create this classification. The provider diagnostic contains only -`provider=codex`, `category=model_auth_incompatible`, `status=400`, and -`authMode=chatgpt`. It excludes the model name and provider response. This does -not change credentials, select a fallback model, replay a turn, or alter the -accepted semantic result or cleanup. The existing configuration-blocker recovery -path requires a configuration change instead of automatically retrying the same -model and connection. Native finalization also derives this evidence from the -committed event and the run's pinned provider configuration before it can queue -a failed-result retry. This survives a controller restart before the adapter -diagnostic is saved. Invalid bindings, missing evidence, or a mismatched event -hash do not grant this classification. A current worker's task is blocked for a -configuration change. A pending review instead keeps its original decision and -status version, releases execution, and creates no automatic recovery action. -After repairing the configuration, explicitly retry the reviewer to resolve the -same pending review. The Board's exact failed-run Retry accepts this classified -native failure only while its saved review is still current and pending; caller -payloads cannot select a different review. Dispatch checks that assignment again. -Superseded or completed reviews retain their authority. - -Cloud portfolio proxy failures have an event-local `cloud_portfolio` context. -It contains only the phase (`fetch`, `http_response`, `response_body`, or -`response_write`), upstream HTTP status when available, elapsed milliseconds, -and an allowlisted network error code. Unknown values become `unknown` or -`null`; elapsed time outside 0–60,000 ms becomes `null`. Codes come from at -most four error/cause objects' own data properties, never error messages. -`DEADLINE_EXCEEDED` means the request's own ten-second signal was aborted. -The route's warning uses the same safe fields. Neither record includes URLs, -headers, cookies, bodies, user/stack IDs, or the original exception. Sentry -receives a plain error with the existing generic message and default grouping. -The context does not change the HTTP response, authentication, cookies, cache, -deadline, or single-fetch behavior, and does not carry into unrelated events. - -A file storage provider failure has an event-local `storage_failure` context -and `storage_operation` and `storage_error_code` tags. The S3 provider records -the failed call (`put_object`, `get_object`, `head_object`, `delete_object`, -`create_multipart_upload`, `upload_part`, or `complete_multipart_upload`), -the provider's error code, the HTTP status, the provider-issued request ID, -and the SDK attempt count. Values come from the SDK error's own data -properties. A value that does not match its bounded format becomes `unknown` -or `null`. The record never includes the bucket, object key, endpoint, -credentials, error message, or provider response body (such as an S3 -`StringToSign`). Sentry receives the original error with default grouping, -so existing issues keep collecting events. The record does not change -storage behavior, not-found handling, or HTTP responses. It is attached only -to the error that the failed call threw. - -An unconfirmed adapter Stop timeout has an event-local `adapter_stop` context: -the run UUID, built-in adapter type, native/legacy runtime mode, configured -wait duration, and whether abort was requested. Invalid identities become -`null`, and unknown adapter/runtime values become `unknown`. The report does -not include stop reasons, prompts, process output, provider responses, or -credentials. It preserves default error grouping and does not acknowledge -termination, remove the live execution control, or change the timeout. The -context is sent only through the existing opt-in Sentry gate and never leaks -into unrelated captures. - -The context also samples the pending execution `phase` and `phaseElapsedMs` -when the Stop timer expires. An in-memory tracker belongs to the exact live -execution control; a replaced or missing owner reports `unknown` with a null -elapsed time. Labels come from a closed list and elapsed time uses a monotonic -clock, capped at one day. Nested scopes remain visible while their awaits are -pending, including ACP session close, transport stop, instruction collection, -workspace restore, and host instruction or lease cleanup. `phase_reporting` -means a step is waiting to write timing or teardown error diagnostics. `adapter_execution` and -`host_execution` are coarse labels for work outside those narrower scopes. -The tracker clears when its executor finishes. A phase is diagnostic context, -not evidence that a provider stopped or that files were recovered. No new -run-log or Telemetry event is emitted by this tracker. - -The shared reporter also attaches bounded diagnostic contexts for both legacy -and native runs: - -- `run_execution`: runtime mode, execution stage/native phase, driver and version, - duration, ACP activity, failure phase, stop reason, error family, - and timeout settings when available. -- `adapter_failure`: selected adapter error fields such as phase, category, - protocol code, retryability, cause message, stack preview, HTTP status, and request ID. -- `provider_failure`: the saved provider failure category, title, and details. -- `run_exception_0` through `run_exception_3`: exception names, codes, HTTP - statuses, and request IDs for a caught exception and up to three causes. - -When a Daytona acquisition fails with pending allocation cleanup, the immediate -setup-failure report can include `environmentAcquisitionPhase`, -`environmentAcquisitionElapsedMs`, and `environmentAcquisitionBudgetMs` in -`run_execution`. The phase is one of `create`, `workspace`, `shell`, `expiry`, -or `sentinel`. It records the first acquisition failure observation, before -inline cleanup, not the cause of the failure. Elapsed time measures acquisition -start to that observation; if cleanup later consumes the deadline, this time -still describes the earlier failure. The overall error message is unchanged. -Elapsed time is bounded to seven days, and the budget to one day. Unknown or -malformed optional values are omitted without invalidating cleanup ownership. - -When the deadline expires while Daytona creation is still pending, the plugin -makes one read-only lookup (bounded to five seconds, inside the host's 30-second -RPC margin) of its provisional creation name and reports -`environmentAcquisitionProviderState`. The value is one of `absent`, `creating`, -`pending_build`, `building_snapshot`, `pulling_snapshot`, `starting`, `started`, -`error`, `build_failed`, `other`, or `unavailable` (lookup failed, timed out, or -ownership labels did not match). It records where the provider left the -allocation, not why. The lookup does not delete, retry, or change the cleanup -envelope. - -These fields cross the existing creation-cleanup RPC envelope only for lease -acquisition. The host accepts them only from an actual RPC error with matching -company, environment, and run ownership. A private receipt binds the report to -that run; arbitrary error fields and saved result JSON do not supply evidence. -Raw provider causes, output, paths, URLs, and cleanup ownership identifiers are -not copied into these diagnostic fields. This does not change timeout budgets, -commands, cleanup, retries, task status, or Sentry filtering. The receipt is -process-local: post-restart reports without the original error cannot recover -these observations. Existing saved runs and leases are not backfilled. - -The orphan reaper records a bounded `processLossDiagnostic` before status writes -or cleanup. For `process_lost` failures, `run_execution` includes -`processLossPidRecorded`, `processLossGroupRecorded`, and `processLossLocalCheck` -(`not_observed_alive`, `not_checked`, or `no_identifiers`). It also includes -`processLossRunPredatesObserver`, `processLossObserverUptimeMs`, and -`processLossLastOutputAgeMs` when the timestamps are available and valid. Ages -above seven days are omitted. These are observations, not proof of an OOM, -provider failure, or deployment. The last-output age can include system output. -`processLossRetryEligible` records eligibility for the existing process-loss retry -at detection; it does not claim that a retry was queued or succeeded. This data -contains no process IDs, task text, paths, or credentials and changes no recovery -or ownership decisions. - -When a native session fails because close could not prove durable runner -suspension (`provider_transport_failed: runner did not durably suspend before -checkpoint`), `run_execution` records which barrier was unproven: -`closeRunnerSuspended`, `closeProviderDrained`, `closeSuspendCommandStatus` -(a runner command status, or `not_queued`), `closeRunnerLifecycle` (`ready`, -`suspended`, `closed`, `recoverable_failure`, or `unobserved`), -`closeRunnerIdentityMatches`, `closeProviderState` (`settled`, `unsettled`, -`unreadable`, or `not_local`), `closeProviderPendingEventCount`, -`closeProviderTurnActive`, `closeSemanticResultPersistenceFailed`, and the counts -`closeSemanticToolsPending`, `closeSemanticToolDeliveriesOpen`, and -`closeSemanticResultFailures`. Fields absent from the runner's settlement are -omitted. Tool call, operation, command and event identities, input digests and -provider turn IDs stay in the instance run log. The report reads the caught error -only for these labels; its text and stack are not copied, because they bypass the -executor's identity redaction. These observations do not change close, quarantine -or recovery behavior. - -Remote runner reattachment failures also record `remoteRunnerRecoveryStage` -and `remoteRunnerRecoveryReason` from the saved run. A confirmed missing process -also records the closed `remoteRunnerLifecycle` and boolean `remoteRunnerSettled` -from the exited runner's saved state. When reattachment fails -before the coordinator saves its result, the report reads the same labels from -the caught recovery error instead. These fields contain -closed labels for marker reads and process verification. They distinguish -missing or unsafe markers, invalid or missing fingerprints, changed or -unavailable process identity, and failed or timed-out probes. Paths, PIDs, -nonces, command lines and transport output are excluded. The existing -`runner_remote_process_identity_unavailable` error message and native failure -classification remain unchanged. The coordinator also saves these labels in -its failure detail and recovery evidence. A later failure clears stale labels. - -ACP turns record `acpLastEventAgeMs`, `acpObservedEventCount`, -`acpPendingToolCount`, and `acpToolInventoryComplete` at finalization, before -usage reads, error logging, and cleanup. The age measures time since the last -runtime event and is omitted if no event was observed or the clock is invalid. -Timeout and cleanup log messages do not reset this age. The pending count uses -known tool statuses; an incomplete inventory cannot establish that no work is -pending. Recent events do not prove useful progress. These fields contain only -numbers and a boolean, never tool names, IDs, arguments, or event content, and -do not change the execution timeout, cancellation, or recovery policy. - -A stopped turn (wall-clock timeout or operator stop) also records -`acpLastEventAgeAtStopMs` and `acpOldestPendingToolAgeAtStopMs`, measured at the -moment the stop fired. A cancel can make the agent emit closing events, so the -finalization age alone cannot separate a turn that went silent long before its -deadline from one that was still working. For a stopped turn, pending tools -means the tools pending when the stop fired, so a tool the cancel closes still -counts; `acpPendingToolCount` keeps its finalization-time meaning. When tools are -pending, `acpPendingToolKinds` lists their ACP tool kinds as a sorted, comma-separated -string drawn only from the closed ACP `ToolKind` enum plus `unknown`; the -server drops the field if any other value appears. - -Claude ACP turns also run a stalled-tool watchdog. A pending tool call that gets -no ACP update for 30 minutes (`stalledToolTimeoutSec`; kept at least 15 minutes -above `MCP_TOOL_TIMEOUT` and `BASH_MAX_TIMEOUT_MS`) ends the turn through the -same cancel and fail-fast deadline as a duplex channel loss. The run fails with -`acpx_tool_stalled` instead of waiting for the wall-clock timeout, and records -`acpStalledToolKind` (one closed `ToolKind` label or `unknown`) and -`acpStalledToolSilentMs`. The stop-time fields above are measured when the -watchdog fired. A live subagent's Task call keeps receiving heartbeats, so only -a call that itself went silent trips the watchdog. A background subagent's Task -call completes when the subagent starts, so the watchdog watches that subagent's -own tool calls instead. Other ACP agents are watched only when -`stalledToolTimeoutSec` is set. - -The engine also observes ACP permission requests without deciding them; ACPX's -permission mode still decides each one. A stalled run records -`acpStalledToolPermissionRequested` (whether the stalled call's permission -request reached Paperclip) and `acpPendingToolPermissionRequestedCount` (how many -pending calls had one). These fields record only that a request arrived, not -that the reply reached the agent. A call whose request never arrived stalled -before the request reached Paperclip. A call whose request arrived stalled -later: on the reply's return trip, or after the agent received the decision. -Both fields hold only a boolean and a count. - -A missing request is informative only if calls like it normally send one. So a -stalled run also records `acpSettledSameKindToolCount` (how many finished calls -of the stalled call's kind the run saw) and -`acpSettledSameKindToolPermissionRequestedCount` (how many of those sent a -permission request). These counts are supporting evidence, not proof of where -the call stalled. The ACP kind is coarse, so peers of the same kind can be -different tools with different permission rules. Read them this way: - -- All or nearly all finished peers sent a request, and the stalled call did not: - this supports a stall before the request reached Paperclip. -- No finished peer sent a request: calls of that kind may skip the request, so - its absence on the stalled call says nothing about where it stalled. -- Some finished peers sent a request and others did not: the counts do not show - whether the stalled call should have sent one. Compare the specific tools in - the run log before drawing a conclusion. -- No finished peers (a count of zero): there is no comparison. - -Both fields hold only counts. - -A sandbox ACP run also records how its agent process session ended, when the -host bridge saw it end before finalization. Startup failures (`create_runtime`, -`ensure_session`, `configure_session`) carry the same fields, so an agent process -that dies during the handshake is reported too. `acpProcessTerminalSource` is one of -`remote_event`, `input_delivery`, `output_poll`, `output_stream`, or -`proxy_close`, and `acpProcessTerminalOutcome` is `exit` or `error`. -`acpProcessTerminalExitCode` (0–255) and `acpProcessTerminalSignal` (a closed -POSIX signal name) are present only when known. These are the same values as the -run log's `ACP process session terminal:` line. They separate a killed or exited -agent process from a host proxy close after a generic ACP connection failure. The -server drops all of them if the source is not a closed label. - -ACP and legacy OpenCode results retain the adapter's resolved wall-clock timeout as -`adapterExecutionTimeout` in the instance run result. Finalization uses it for -`effectiveTimeoutSec`, `timeoutSource`, and `timeoutConfigured`. Sources are -`configured`, `sandbox_default`, or `unlimited`; `timeoutConfigured` identifies an -explicit override, including a negative value that disables the timer. An untouched -sandbox value of zero reports the four-hour default, while a local zero reports -unlimited. Older adapters without a valid resolution retain the config-based -metadata fallback. This does not change timers, Stop acknowledgement, or recovery. -Legacy OpenCode reports a generic timeout message because a provider or output -observation timeout can occur before the configured execution deadline. The -resolved policy describes the configured limit, not proof that its timer fired; -`timeoutFired` retains the existing terminal timeout classification. Provider -stderr remains in the instance run log and is not used as the timeout exception -message. An earlier provider timeout does not establish its upstream cause. - -Legacy OpenCode requests Daytona's persistent process session for -its main CLI command. Daytona targets with the effective persistent-session capability -use asynchronous dispatch and log observation instead of holding one synchronous -command HTTP request open for the whole run. The command's resolved timeout is -forwarded separately from individual status/log observation timeouts. One timer -bounds dispatch and observation, fences late log callbacks, and clears at settlement. -A timeout does not prove command termination; the existing stop/lease cleanup -path still owns that evidence. Existing session callers without this deadline -keep their previous lifetime. Probes, preparation and other providers -keep their existing paths. Local and SSH execution do not use this option. -The resolved execution budget and lease cleanup remain unchanged; the session -request does not prove that a previously observed gateway timeout had this cause. - -When settlement records a workspace restore failure, `run_execution` also -includes `workspaceRestoreFailure` with one of the shared, path-free codes: -`restore_permission_denied`, `restore_lock_timeout`, `restore_unsafe_archive`, -or `restore_failed`. Unknown values are omitted. Workspace paths and arbitrary -pre-restore result data are not included. A later successful run does not, by -itself, establish that an earlier failed restore recovered the workspace files. - -For `configuration_incomplete` with `workspace_base_ref_unresolved`, -`run_execution` includes `workspaceBaseRefRemoteLookup`, `workspaceBaseRefAuthLookup`, -`workspaceBaseRefFetch`, and `workspaceBaseRefRefResolution` when producer evidence -is available. These closed outcomes distinguish skipped operations, auth lookup -failure, and failed or completed Git commands. Optional `workspaceBaseRefFetchExitCode` -and `workspaceBaseRefRefExitCode` are integers from 0 through 255. -`workspaceBaseRefFetchFailureKind` is a coarse, bounded Git diagnostic category; -unknown, mixed, truncated, or oversized command output remains `unknown`. -A failed fetch that prints β€œremote ref not found” is an observation, not -authoritative proof of user error. These failures remain reportable. The fields -add no Git calls and do not change authentication, retries, task blocking, -recovery actions, or Sentry filtering. They never contain repository URLs, -configured refs, command output, paths, or credentials. - -For `workspace_validation_failed` with `git_worktree_not_reusable`, `run_execution` -includes `workspaceValidationReason` and an allowlisted `workspaceValidationReasonCode`: -`missing_worktree`, `not_a_git_checkout`, `not_registered`, `wrong_repository_root`, -`branch_mismatch`, or `git_inspection_failed`. The last code distinguishes an -unsuccessful or truncated Git registration inspection from confirmed absence in -a complete list. Its optional `workspaceValidationInspection*` fields identify -the fixed command `worktree_list`, failure (`spawn_failed`, `nonzero_exit`, or -`output_truncated`), an allowlisted OS error code (or `unknown`), and an exit code -from 1 through 255. No paths, repository or branch names, IDs, command output, or -arbitrary messages enter these fields. Validation still blocks reuse; it does -not repair Git metadata, change the selected repository, or retry the task. - -For explicit reuse of a retained Git workspace, `persisted_workspace_source_conflict` -uses the same two fields with the fixed reason codes `source_scope_mismatch`, -`explicit_project_workspace_conflict`, `source_path_unproven`, -`source_registration_unproven`, `source_repository_mismatch`, or -`source_repository_unavailable`. These distinguish an unverified original source -from a missing Git registration. Paths, repository URLs, and workspace IDs stay -out of the diagnostic fields. The task remains blocked until its original source -is available or the owner intentionally selects a different workspace. A failed -Git registration probe retains `git_inspection_failed` and the same bounded -inspection fields, including when it occurs during original-source selection. - -When available, the saved `workspaceRestoreDiagnostic` adds the bounded fields -`workspaceRestorePhase`, `workspaceRestoreStep`, `workspaceRestoreErrorCode`, -`workspaceRestoreHttpStatus`, and `workspaceRestoreExitCode` to `run_execution`. -The phase is `workspace` or `asset`. The failing step is one of `git_export`, -`git_import`, `workspace_transfer`, `workspace_extract`, `directory_merge`, -`git_integration`, `index_reset`, `git_ref_cleanup`, or `asset_restore`. -Nested steps retain the most specific failing operation. Error codes come from -the restore diagnostic allowlist, with unrecognized codes reported as `unknown`; -HTTP statuses are integers from 400 through 599 and process exit codes are -integers from 1 through 255. These fields accompany a known restore failure code -only. They omit error messages, raw command lines, paths, process output, and arbitrary -cause data. Git error wrappers preserve only these safe codes and numbers for -diagnostics, without adding the original error as a cause. -Native sandbox `environmentSyncOut` failures carry the same allowlisted codes -and bounded HTTP/exit statuses across the plugin worker RPC boundary. The host -revalidates that optional envelope and retains it only for restore diagnostics. -The envelope excludes provider messages, response bodies, names, paths, and -credentials. The existing RPC error message and code remain unchanged, as do -restore classification, retries, and source retention. Older workers without the -envelope still report `unknown` when no structured cause is available. -For native transfers, optional `workspaceRestoreTransferStep` identifies -`sandbox_access`, `sandbox_guard`, `file_download`, `file_finalize`, -`archive_create`, `archive_download`, `archive_validate`, or `archive_extract`. -`workspaceRestoreTransferFailureKind` distinguishes a producer-confirmed -`command_failed`, `download_failed`, `download_missing`, `unsafe_archive`, -`listing_timeout`, `listing_entry_limit`, `listing_byte_limit`, -`listing_line_limit`, or `listing_stderr_limit`. These labels are fixed values, -not classifications inferred from provider message text. Missing labels remain -absent. A listing deadline records `ETIMEDOUT`; sandbox and listing commands -retain bounded nonzero exit codes even when their error text is wrapped. -`workspaceRestoreRpcCode` retains only known numeric plugin RPC error codes from -a typed error at the host's sync-out boundary. In particular, RPC `TIMEOUT` -(`-32003`) is separate from the model execution's `timedOut` value; successful -model execution does not exclude a later transfer timeout. Transfer evidence is -isolated per outbound request and per restore settlement, then independently -revalidated before persistence and Sentry projection. These fields only accompany -`workspace_transfer` or `asset_restore` diagnostics and do not authorize cleanup -or retries. Older workers remain compatible. -For `git_integration`, optional `workspaceRestoreGitCommand` identifies the fixed -command family: `rev_parse`, `symbolic_ref`, `merge_base`, `merge_tree`, -`commit_tree`, `update_ref`, or `log`. `workspaceRestoreGitFailureKind` is -`merge_conflict`, `invalid_object`, `ref_conflict`, `permission_denied`, or -`unknown`. A merge conflict requires an uninterrupted `merge-tree --write-tree` -exit of 1 with a completed tree ID in stdout; exit 1 alone is ambiguous. -Object and ref classifications require recognized Git diagnostics; -permission denial requires an OS `EACCES` or `EPERM` code. Unrecognized or -localized messages remain `unknown`. Up to 16 KiB of stderr is inspected only -in memory; no arguments, stderr, paths, repository URLs, filenames, or ref names -enter these fields. Handled probes and successful retries emit no diagnostic. -This does not change -restore behavior, retries, timeouts, or recovery policy. - -A caught directory-merge lock timeout also records `restoreLockOwnerState` -(`alive`, `dead`, `unknown`, `missing`, or `invalid`), `restoreLockKnownLocalHolder`, -and, when available, `restoreLockOwnerSameProcess`, `restoreLockOwnerPredatesProcess`, -`restoreLockOwnerAgeMs`, and `restoreLockWaitMs` in `run_execution`. Ages are capped -at seven days. These fields omit paths, PIDs, owner records, and absolute timestamps. -The local-holder flag covers this module's active acquisitions only. Process-age -comparison uses the wall clock and a one-second margin; it is a clue to PID reuse, -not proof of ownership or permission to remove a lock. Diagnostic reads can race -with release. The extra diagnostic owner read has a 100 ms budget; a stalled or -unreadable read leaves the owner state `unknown`, while malformed JSON is `invalid`. -These fields do not change lock acquisition, reclamation, or retries. -Agent-directory callers also supply `restoreLockOperation`: `agent_directory_release`, -`agent_directory_collect`, `agent_directory_checkpoint`, or `agent_directory_handoff`. -This identifies the operation waiting for the lock, including cleanup after a -completed adapter turn when the recorded execution stage has not advanced. - -The execution and setup catch paths pass the original exception to the reporter. -It snapshots and rebuilds only these selected fields and the original message and -stack. Sentry receives the sanitized cause chain rather than a stack created at -the reporting call. Saved adapter results use an available adapter stack preview; -without one, the report has no synthetic reporting stack. - -Credential patterns, the current user's home path, registered run-secret values, -and known host/runtime environment credentials are removed before truncation. -Environment values stay private unless their names and values identify an ordinary -process setting. Flags and counts require both a setting-name match and a boolean -or numeric value. Mode labels use a closed value list. Paths, regions, and key -selectors such as `SORT_KEY` use explicit ordinary-setting names. Unknown opaque -values remain private, including credentials under arbitrary names. Declared -secret bindings and recognizable credential contents override ordinary-setting -exemptions, including short values. Runtime values are used only for redaction -and are never copied into an event. Stack fields are limited to 8,192 characters, -exception messages to 2,048, provider titles to 4,096, and provider details to -12,288. Other diagnostic strings are limited to 200 characters (adapter cause -messages: 2,048). Truncation is marked in the text and in -`run_execution.truncatedFields`; cyclic and deeper cause chains are marked too. -Request/response objects, headers, environment/configuration, prompts, stdout, -stderr, and arbitrary adapter result fields are not copied. The existing DSN -opt-in gate and error-code/adapter fingerprint remain unchanged. These reports -cannot recover diagnostic data that a provider or adapter discarded upstream. +In `unexpected` mode, some failures that the task already shows to its owner +stay in the run log and the task's blocked state instead of reporting. Examples +include known missing-secret configuration blockers, some provider quota and +rate-limit failures, recognized external connection failures, and a scheduled +Codex capacity retry. Most exclusions have conditions, so read +`server/src/services/run-failure-report.ts` for the full policy. `all` mode +reports these failures. +Other run failures attach bounded diagnostic contexts. Their fields are defined +in `server/src/sentry.ts` and pass through the existing secret redaction and size +limits. They can include selected provider error titles and details, but not raw +request or response objects, prompts, message bodies, or credentials. ### Browser data @@ -1661,7 +484,9 @@ log line and keep running with no error monitoring. ### Default capture set The lists below name every event and every context field this feature -sends, so an operator can read what the feature does before turning it on. +sends, so an operator can read what the feature does before turning it on. The +diagnostic contexts on run failure events are the exception: they are defined in +`server/src/sentry.ts`, as described under "Run failure context". Each Sentry integration name below is verified against the default integration list of `@sentry/node@10.71.0` and `@sentry/browser@10.71.0`. @@ -1806,508 +631,3 @@ Two controls belong to the operator. This feature ships neither one. events from the operator's network, not from the server's network, so an internal-only host name fails silently for the browser even when it works for the server. - -## Sandbox Startup Trace Spans - -Paperclip opens OpenTelemetry spans on the sandbox start path. These spans are -an Observability surface. They are not Paperclip Telemetry events. The -generated telemetry contract does not cover them, so this section is their -canonical contract. - -The spans are opt-in. Paperclip exports them only when an OTLP endpoint is -configured. With no endpoint the whole span path is a no-op. Paperclip opens the -spans only for a run that targets a remote sandbox. A local run and an SSH run -stay out of these spans. - -Every span attribute uses the closed `paperclip.sandbox.startup.` prefix and -rides a fixed allowlist. A command line, an argument, an environment value, a -file path, program output, or a raw identifier never rides a span. It rides -neither as an attribute nor as an event. The producer bounds each free-form -value: - -- A command basename maps to a small known set. Any other value maps to `other`. -- A region maps to a small known set. Any other value maps to `unknown`. -- An image id, a sandbox id, and a lease id ride only as a non-reversible short - hash. - -Each numeric attribute is finite. Paperclip omits an attribute when its value is -absent, never a misleading `0`. - -### Spans - -| Span | Scope | Parent | -| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | -| `sandbox.startup` | The one root span for a sandbox bring-up. | none (root) | -| `workspace.resolve` | Workspace resolution step. | `sandbox.startup` | -| `codex-home.seed` | Managed-home seed step. | `sandbox.startup` | -| `skills.reconcile` | Skills reconcile step. | `sandbox.startup` | -| `stage.sync` | Workspace stage-sync step. | `sandbox.startup` | -| `snapshot.git` | Host-side git workspace enumeration inside `stage.sync` (`git status --ignored`, the HEAD diffs, `ls-files`). | `stage.sync` | -| `snapshot.baseline` | Host-side baseline workspace content-hash walk inside `stage.sync`, kept for restore. | `stage.sync` | -| `stage.workspace` | One inbound workspace stage task inside `stage.sync`. It packs and uploads the workspace. | `stage.sync` | -| `stage.asset.` | One inbound asset stage task inside `stage.sync`. It packs and uploads one managed-home asset. The `` segment is the asset key. | `stage.sync` | -| `stage.project.` | One inbound referenced-project stage task inside `stage.sync`. It uploads one referenced project. The `` segment is the project id. | `stage.sync` | -| `pack` | Host-side workspace tarball build inside the `stage.workspace` task. | `stage.workspace` | -| `bridge.paperclip` | Paperclip bridge start step. | `sandbox.startup` | -| `bridge.process-session` | Process-session bridge start step. | `sandbox.startup` | -| `acp.handshake` | ACP session handshake step. | `sandbox.startup` | -| `sandbox.syncBack` | The settlement sync-back that restores the managed home at teardown. | the active run span | -| `restore.workspace` | One outbound workspace restore task at teardown. It reads the sandbox workspace back and merges it into the host workspace. | `sandbox.syncBack` | -| `restore.asset.` | One outbound asset restore task at teardown. It reads one asset back to its host store. The `` segment is the asset key. | `sandbox.syncBack` | -| `sandbox.agentSession.sendInput` | One outbound ACP message to the agent β€” the socket handler's one `writeTextFile` exec. | the active run span | -| `sandbox.agentSession.pollOutput` | One 100 ms poll tick β€” `list`, then `read`+`remove` per file found (`1 + 2n` execs). | the active run span | -| `sandbox.callbackBridge.relayRequest` | One Paperclip-API callback request β€” read the request, write the response, remove it. | the active run span | -| `sandbox.agentProcess` | The persistent streamed agent process the process-session bridge launches; open until the process settles or the bridge tears down, whichever comes first. | the active run span | -| `sandbox.exec` | One host-to-sandbox execution. | the active step or wrapper span | - -A step span name is the step name. The `sandbox.exec` span parents to the step -span that runs the execution, so each execution nests under its step. Within -`stage.sync`, the host-side sub-steps `snapshot.git` and `snapshot.baseline` open -as child spans of the step, so the host work at the head of the step is -attributed rather than showing as a gap. Each inbound sync operation also opens -its own task span under `stage.sync`: `stage.workspace`, one `stage.asset.` -per asset, and one `stage.project.` per referenced project. The `pack` span -nests under `stage.workspace`, because the host builds the tarball inside that -task. Two concurrent tasks produce overlapping spans. - -The settlement `sandbox.syncBack` span runs at teardown and parents to the run -span. It wraps the managed-home restore. Each outbound restore operation opens -its own task span under `sandbox.syncBack`: `restore.workspace` and one -`restore.asset.` per asset. Two concurrent restore tasks produce overlapping -spans. A run-time -`sandbox.exec` span parents instead to the run-time wrapper span that runs it -(`sandbox.agentSession.sendInput`, `sandbox.agentSession.pollOutput`, -`sandbox.callbackBridge.relayRequest`, or `sandbox.agentProcess`). Each run-time -wrapper span parents to the live run span (`agent.turn` during the turn, -`task.provider_session` otherwise). With no active trace context the exec span opens -unparented. - -`sandbox.agentProcess` wraps the persistent streamed agent process. The -process-session bridge launches it during `bridge.process-session`, so it opens -under `task.provider_session` β€” no turn has started yet. It therefore overlaps the sibling -`agent.turn` rather than nesting under it or dangling off the short-lived bring-up -step. The span ends when the process settles or when the bridge tears down, -whichever comes first. The bridge tears down before the run root span ends, so -the span never outlives `task.provider_session` even when the process lingers past teardown -(the sandbox `execute` has no cancel, so a lingering process cannot be forced to -resolve). - -The root span sets the error status when the bring-up fails. Each step span sets -the error status when its step fails. The `sandbox.exec` span sets the error -status when the exit code is non-zero or the execution throws. - -### Outcome values - -The `paperclip.sandbox.startup.outcome` attribute uses a closed value set: - -- `ok` β€” the step or the execution settled with a success result. -- `skipped` β€” a warm cache skipped the step; the step ran no work. -- `failed` β€” the step or the execution threw, or the exit code was non-zero. - -### Root span attributes - -The `sandbox.startup` root span uses this closed attribute allowlist. - -| Attribute | Type | Optional | Meaning | -| ---------------------------------------- | ------- | -------- | ---------------------------------------------------------- | -| `paperclip.sandbox.startup.root.wall_ms` | number | no | The root-span wall time of the whole bring-up. | -| `paperclip.sandbox.startup.root.work_ms` | number | no | The sum of the step wall times. | -| `paperclip.sandbox.startup.root.diff_ms` | number | no | `work_ms βˆ’ wall_ms`; the overlap the parallel steps saved. | -| `paperclip.sandbox.startup.provider` | string | yes | The normalized provider family. | -| `paperclip.sandbox.startup.cold_start` | boolean | yes | Legacy value, set only when receipts establish cold or warm startup. | -| `paperclip.sandbox.startup.warmth` | string | yes | Receipt-derived `cold`, `warm`, `partial_reuse`, or `unknown`. | -| `paperclip.sandbox.startup.region` | string | yes | The clamped region label. | -| `paperclip.sandbox.startup.image_id` | string | yes | The hashed image id. | -| `paperclip.sandbox.startup.sandbox_id` | string | yes | The hashed sandbox id. | -| `paperclip.sandbox.startup.lease_id` | string | yes | The hashed lease id. | - -### Step span attributes - -The root's warmth uses the per-layer receipts documented in -[run-log-events.md](./run-log-events.md#startup-reuse-evidence). Partial and -unknown evidence omit the legacy `cold_start` boolean. A reused session ID or -runtime object alone cannot establish warmth. These attributes retain the -operator OTLP endpoint gate and are no-operations without that endpoint. - -Each bring-up step span uses this closed attribute allowlist. The step name -rides the span name, so no `step` attribute repeats it. - -| Attribute | Type | Optional | Meaning | -| ------------------------------------------------------------ | ------ | -------- | -------------------------------------------------------- | -| `paperclip.sandbox.startup.step.wall_ms` | number | no | The wall time of the step. | -| `paperclip.sandbox.startup.outcome` | string | no | The step outcome (`ok`, `skipped`, or `failed`). | -| `paperclip.sandbox.startup.provider` | string | yes | The normalized provider family. | -| `paperclip.sandbox.startup.batch` | string | yes | A shared tag that marks two parallel steps as one batch. | -| `paperclip.sandbox.startup.handshake.create_runtime.wall_ms` | number | yes | The create-runtime sub-time of the `acp.handshake` step. | -| `paperclip.sandbox.startup.handshake.ensure_session.wall_ms` | number | yes | The ensure-session sub-time of the `acp.handshake` step. | - -The round-trip count and the provider durations no longer ride a step span. The -per-execution `sandbox.exec` child spans carry that detail. - -### `sandbox.exec` span attributes - -The `sandbox.exec` span uses this closed attribute allowlist. Paperclip omits a -numeric attribute when the provider does not report the value. - -| Attribute | Type | Optional | Meaning | -| ----------------------------------------------- | ------- | -------- | -------------------------------------------------------------------------- | -| `paperclip.sandbox.startup.provider` | string | no | The normalized provider family. | -| `paperclip.sandbox.startup.exec.command` | string | no | The clamped `argv[0]` command label. | -| `paperclip.sandbox.startup.exec.exit_code` | number | yes | The numeric process exit code. | -| `paperclip.sandbox.startup.exec.wall_ms` | number | no | The host-measured wall time of the execution. | -| `paperclip.sandbox.startup.exec.wait_before_ms` | number | yes | The provider handle-fetch wait before the execution ran. | -| `paperclip.sandbox.startup.exec.sandbox_ms` | number | yes | The in-sandbox run time of the execution. | -| `paperclip.sandbox.startup.exec.network_ms` | number | yes | The transport time the host adds; `wall_ms βˆ’ wait_before_ms βˆ’ sandbox_ms`. | -| `paperclip.sandbox.startup.exec.critical_path` | boolean | no | Whether the execution sits on the startup critical path. | -| `paperclip.sandbox.startup.exec.cache_hit` | boolean | yes | Whether the provider served the sandbox handle from its warm cache. | -| `paperclip.sandbox.startup.outcome` | string | no | The execution outcome (`ok` or `failed`). | - -The plugin decides the cache hit at the sandbox-handle lookup. The span no -longer infers a cache hit from `wait_before_ms == 0`. Paperclip omits the -`cache_hit` attribute when the provider does not report the value. - -To add a span attribute, extend the `SANDBOX_STARTUP_SPAN_ATTRS` allowlist in -the code first. Keep the attribute low-cardinality and free of user content. - -### Provider spans - -A sandbox provider plugin also opens spans for its own sync steps. These spans -use the `sandbox.daytona.` name prefix. They share the -`paperclip.sandbox.startup.` attribute prefix and obey the same opt-in and -no-user-content rules as the startup spans above. - -The plugin worker runs in a separate process from the host. So the host treats -every field of a worker-sent span as untrusted input. The host re-clamps the -span name and every attribute at one boundary, the `span.record` host handler, -before it records the span. - -| Span | Scope | Parent | -| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | -| `sandbox.daytona.pack` | The host-local pack step that builds the upload tarball. It makes no sandbox round trip. | the active startup step span | -| `sandbox.daytona.transfer` | The transfer step: an upload to the sandbox (inbound) or a download from the sandbox (outbound). The `paperclip.sandbox.startup.transfer.direction` attribute records the direction. | the active sync task span (`stage.*` inbound, `restore.*` under `sandbox.syncBack` outbound) | -| `sandbox.daytona.ensureDirectory` | The `mkdir -p` step that ensures a directory exists before a write. | the active startup step span | -| `sandbox.daytona.checkSymlinkEscape` | The re-check step that a path resolves inside the workspace root before use. | the active startup step span | -| `sandbox.daytona.promote` | The atomic move of a staged temp onto its target via a pinned dir handle. | the active startup step span | -| `sandbox.daytona.extractTarball` | The one round trip that re-checks the path, runs `tar -xf`, and removes the scratch tarball. | the active startup step span | -| `sandbox.daytona.postUploadCommand` | One caller-supplied post-upload command. | the active startup step span | -| `sandbox.daytona.session.open` | The create of the one persistent session for a lease, on the first in-run command. | the active run span | -| `sandbox.daytona.session.close` | The delete of that persistent session on lease release. | the active run span | -| `sandbox.daytona.other` | Any span name outside the known set. | the active startup step span | - -The host clamps the span name to the closed set of leaf names above (`pack`, -`transfer`, `ensureDirectory`, `checkSymlinkEscape`, `promote`, `extractTarball`, -`postUploadCommand`, `session.open`, and `session.close`). The host maps a known -name to `sandbox.daytona.`. The host maps any other value to -`sandbox.daytona.other`, so a span name never carries free-form data. Only the -daytona provider emits these spans today, so the segment is the literal -`daytona`. - -### Daytona lease acquisition and resume - -The worker invocation carries nested, monotonic timing receipts for lease work. -The host also admits the following fixed `lease.*` names and prefixes them with -`sandbox.daytona.`. These use the existing opt-in endpoint and host-owned parent -context; they do not create an exporter or first-party Telemetry events. - -```text -execution.plugin_dispatch -└── plugin.worker - β”œβ”€β”€ sandbox.daytona.lease.wait_for_teardown (only when teardown blocks resume) - β”œβ”€β”€ sandbox.daytona.lease.lookup - β”œβ”€β”€ sandbox.daytona.lease.refresh (missing-container confirmation) - β”œβ”€β”€ sandbox.daytona.lease.channel_cleanup (before restart) - β”œβ”€β”€ sandbox.daytona.lease.wait_transition (only while the sandbox is changing state) - β”‚ └── sandbox.daytona.lease.force_stop (only when `stopping` outlasts 30 s) - β”œβ”€β”€ sandbox.daytona.lease.start (or lease.recover) - └── sandbox.daytona.lease.verify - β”œβ”€β”€ sandbox.daytona.lease.workspace - β”œβ”€β”€ sandbox.daytona.lease.sentinel - └── sandbox.daytona.lease.shell -``` - -`lease.lookup` includes cached-handle lookup and any required provider refresh. -`lease.start` and `lease.recover` measure the actual SDK operation, including its -internal readiness wait. They also cover starts triggered by execution or sync. -They do not separate SDK-internal requests or provider-side queue time. -Their `paperclip.sandbox.startup.state` attribute records the state before the -call: `started`, `stopped`, `archived`, `error`, a transition (`starting`, -`stopping`, `restoring`, `archiving`, `resizing`), or `other`. Both the plugin and -host restrict this value; no provider error text or resource identity is sent. -`lease.wait_transition` covers polling a sandbox that is changing state before a -start; its state attribute is the state it found. A start that finds the sandbox -`stopping` for more than 30 s forces the stop (`lease.force_stop`) and keeps -waiting within the same start deadline if Daytona refuses. The forced stop belongs -to the sandbox: parallel callers send one, and none of them starts the sandbox -until that request settles. A confirmed start ends that stuck stop, so a later -one is forced again. - -Fresh acquisition emits `lease.create`, `lease.workspace`, `lease.shell`, -`lease.expiry`, and `lease.sentinel`. `lease.create` covers the existing creation -helper, including qualification-only snapshot resolution and creation-failure -cleanup. Setup-failure deletion is a separate `lease.cleanup` sibling, so it -does not inherit a failed workspace or verification phase. Existing deadlines, -ownership, cleanup and retry behavior are unchanged. Failed awaited steps close -with error status; the receipt contains no exception message. - -Children nest beneath the current `plugin.worker` receipt and verification -children nest beneath `lease.verify`. Never add parent and child durations. -Receipt budgets and clock validation still apply. Older workers without -`tracer.withSpan` keep the original behavior without these detail spans. Tracing -failures preserve the operation's result or error and never repeat it. - -The `sandbox.daytona.*` spans use this closed attribute allowlist. The host -drops every other key, so a command, an argument, a path, an id, a standard -output, or a standard error never rides a provider span. The host records only -the attributes that the producer sends for one span. - -| Attribute | Type | Optional | Meaning | -| ------------------------------------------------ | ------ | -------- | --------------------------------------------------------------------------------------------------------- | -| `paperclip.sandbox.startup.provider` | string | no | The normalized provider family. | -| `paperclip.sandbox.startup.state` | string | yes | State before start/recovery: `started`, `stopped`, `archived`, `error`, or `other`. | -| `paperclip.sandbox.startup.outcome` | string | yes | The step outcome (`ok`, `skipped`, or `failed`). | -| `paperclip.sandbox.startup.pack.wall_ms` | number | yes | The host-local wall time of the pack step. It rides the `sandbox.daytona.pack` span. | -| `paperclip.sandbox.startup.transfer.wall_ms` | number | yes | The wall time of the transfer step. It rides the `sandbox.daytona.transfer` span. | -| `paperclip.sandbox.startup.transfer.guard.count` | number | yes | The number of serial guard round trips before one transfer. It rides the `sandbox.daytona.transfer` span. | -| `paperclip.sandbox.startup.transfer.direction` | string | yes | The transfer direction (`inbound` or `outbound`). It rides the `sandbox.daytona.transfer` span. | - -The `span.record` host handler enforces the allowlist. It re-maps `provider` -through the provider-family normalizer. It keeps `outcome` only when the value -is `ok`, `skipped`, or `failed`. It keeps `transfer.direction` only when the -value is `inbound` or `outbound`. It keeps a numeric attribute only when the -value is a finite number. It drops a status message and keeps only the numeric -status code. The handler never throws, because observability must not change the -sync control flow. - -The `span.record` host method needs the `environment.drivers.register` -capability. So only a plugin that registers an environment driver may emit a -provider span. The capability gate rejects a provider span from any other -plugin. - -The host parents each provider span to the active sync task span. An inbound -transfer runs inside a `stage.*` task span, so its provider spans parent there. -An outbound transfer runs inside a `restore.*` task span under `sandbox.syncBack` -at teardown, so its provider spans parent there. The host mints a W3C -`traceparent` from the active task span and passes it to the plugin worker on the -per-call invocation channel. The teardown restore runs inside the run-parented -`sandbox.syncBack` span, so the host mints a `traceparent` for an outbound -provider span the same way it does for an inbound one. The worker tags its span with the -`traceparent` and treats the value as opaque. The worker never derives the -parent from it. The host recovers the `traceparent` from its own invocation -record, so a worker can never forge a parent. The host validates the -`traceparent` and rejects a missing or malformed value. With no active host -trace context the worker sends no span, so the whole provider-span path is a -no-op. - -## Sandbox Duplex Transport Instrumentation - -This section documents one duplex transport with three sinks: an -OpenTelemetry span, a counter in the `tool_runtime_metric_counters` table, and -one run-log event. - -Paperclip opens a fixed observability surface for the sandbox duplex transport. -This instrumentation is separate from Paperclip Telemetry events and from the -sandbox startup trace spans above. The generated telemetry contract does not -cover it, so this section is its canonical contract. The code owner is -`packages/adapter-utils/src/duplex-observability.ts`. That module holds each name and -each enum value as a literal constant, so the surface never drifts. - -The surface is opt-in. The host injects a recorder that binds the span to the -OTel tracer, the counter to the guarded counter store in -`server/src/services/tool-runtime-metrics.ts`, and the event to the run-events -bridge. The default recorder is a no-op, so the whole surface stays inert until -the host binds a real recorder. Every recorder call sits inside an error swallow, -so a telemetry failure never breaks the request path. - -The surface carries no user content. No route, no query, no request body, no -token, and no raw identifier rides a span, a counter, or an event. Each record -carries only the closed dimension keys below and, for the request span, a -latency. The `provider` dimension carries only the allowlisted public value -`daytona`. Any other plugin key maps to `other` before the record reaches a sink, -so a raw plugin key never reaches a span attribute, a counter label, or an event -field. - -### Spans - -| Span | Scope | Latency | -| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------ | -| `sandbox.duplex.channel_open` | One duplex channel-open attempt. The `outcome` dimension is `ok` when the channel opened and readiness passed, or `error` when the open or readiness failed. | none | -| `sandbox.duplex.request` | One duplex request the broker forwarded to the host. | The request latency in milliseconds. | - -### Event - -| Event | Scope | -| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `sandbox.duplex.transport` | The host emits it at each transport boundary: a ready duplex channel, a fallback to the file bridge, and a terminal channel loss. Its dimensions record the boundary. | - -### Counters - -| Counter | Scope | -| ----------------------------------- | ----------------------------------------------------------------------------------- | -| `sandbox_duplex_channel_open_total` | One successful duplex channel open. | -| `sandbox_duplex_fallback_total` | One fallback to the file bridge. The `fallback_reason` dimension records the cause. | -| `sandbox_duplex_loss_total` | One terminal duplex channel loss. The `loss_class` dimension records the phase. | -| `sandbox_duplex_session_leak_total` | One leaked provider session at teardown. | - -### Dimension keys - -Counters carry no dimension labels. The guarded counter store keys each counter -on `(companyId, metric)` with no label column, so the `fallback_reason` and -`loss_class` values fold into the counter metric name instead. The full closed -dimension set below rides only the spans and the `sandbox.duplex.transport` -event, which use only these closed keys. A test asserts the exact set, so a new -key never reaches a sink by accident. - -| Key | Type | Optional | Value set | -| --- | --- | --- | --- | -| `provider` | string | no | `daytona`, or `other` for any other plugin key. | -| `transport` | string | no | `duplex`, `http2`, or `file`. `duplex` names the retired bespoke frame protocol; `http2` names the Node HTTP/2 session over the sandbox channel; a fallback record uses `file`. | -| `outcome` | string | yes | `ok` or `error`. | -| `fallback_reason` | string | yes | `gate_off`, `capability_absent`, `route_busy`, `entrypoint_sync_failed`, `broker_construction_failed`, `channel_open_failed`, `ready_invalid`, `ready_nonce_mismatch`, `ready_timeout`, `contaminated`, or `preface_missing`. It rides only a fallback record. `route_busy` marks the process-scoped route ceiling full. `entrypoint_sync_failed` and `broker_construction_failed` mark the named build step. `channel_open_failed` marks a failed channel open. `preface_missing` marks a missing or an invalid HTTP/2 client connection preface inside the bounded readiness buffer: the host found no valid preface after the accepted READY line, aborted the `http2` open, and moved the run to the file bridge (`queue_v1`) one time. | -| `loss_class` | string | yes | `pre_dispatch` or `post_dispatch`, relative to the first request dispatch. It rides only a loss record. | -| `loss_reason` | string | yes | `stdin_eof`, `provider_exit`, `heartbeat_timeout`, `rpc_failure`, `write_error`, `transport_closed`, or `other`. The host maps every loss cause to one of these values, so no raw provider text reaches a sink. `write_error` marks a rejected host-to-sandbox write. `transport_closed` marks a reason-less provider transport close with no exit data. It rides only a loss record. | - -To add a name or an enum value, extend the literal constant in -`duplex-observability.ts` first, then update the test that asserts the closed set. - -### Known behavior: aggregate retained body bytes - -Each HTTP/2 bridge route holds up to 168,820,736 bytes (161 MiB) at its own -peak (see `HTTP2_BRIDGE_MAX_CONCURRENT_STREAMS` in `http2-bridge-server.ts`). -The host process admits up to 128 concurrent routes (see -`DEFAULT_MAX_CONCURRENT_DUPLEX_ROUTES` in `plugin-worker-manager.ts`). Those -two figures alone would let the process retain up to 21,609,054,208 bytes -(about 20.1 GiB) of body data across every route at the same time. - -The process does not reach that figure, on two levels. -`HTTP2_BRIDGE_MAX_PROCESS_BODY_BYTES` (`http2-bridge-server.ts`) enforces a -real, live ledger: 1,073,741,824 bytes (1 GiB) across every route, not merely -an accepted paper ceiling. Every HTTP/2 stream creates one `BridgeBodyReservation` owner over -its lifetime, and every source-level full-body buffer that stream retains β€” -its request-body chunk array, the concatenated request body, the -response-body chunk array, and the concatenated response body β€” reserves -against that one owner before it allocates. A reservation that would pass the -process total is denied before it copies anything, and the host answers 503 -instead of accepting the body. The reservation stays live for the response -body until the HTTP/2 write actually finishes flowing to the peer or the -stream closes, not merely until the write call returns, so a slow or -backpressured peer cannot hold response bytes in memory the ledger no longer -counts. - -`HTTP2_BRIDGE_MAX_ROUTE_BODY_BYTES` adds a second, per-route ledger on top of -that process-wide one: each route's own reservations also check a ceiling -scoped to that one route (its own 168,820,736-byte peak from above), so one -busy or malicious route can pass its own ceiling and get denied with a 503, -but it can never spend the whole process-wide total and deny every sibling -route admission. This accounting covers source-level full-body buffers only: -internal Node.js and Undici copies (socket buffers, HTTP/2 frame buffers, -decompression buffers) stay outside it. - -The generated gateway process inside the sandbox (`getSandboxCallbackBridgeServerSource` -in `sandbox-callback-bridge.ts`) enforces its own separate ledger, independent -of the two host-side ledgers above: each side bounds only the memory in its -own process. `readBodyBytes` reserves a request body's chunk bytes as they -arrive, then reserves the concatenated buffer's own byte count before -`Buffer.concat` allocates it, against a ceiling of `maxBodyBytes * 8` (4 -concurrent bodies, each counted twice for its two live copies). A denied -reservation answers 503 with no forward call. Each request handler releases -its own reservation once the whole request settles: a completed response, a -thrown error, a client abort, or a deadline timeout all reach the same -release call. - -Keep every dimension low-cardinality and free of user content. - -### Shared skill preparation - -`skills.prepare` measures the shared inventory listing and runtime materialization -inside `task.prepare`. It is also contained in the broader -`heartbeat.prepare_before_environment` interval; do not add those two durations. -Preparation failures emit a failed span even when no native session starts. -It carries no skill contents, identifiers, locations, or credentials. It uses the -existing run performance events and operator-configured OpenTelemetry endpoint; -no first-party Telemetry event is added. - -The `skills.*` detail spans above distinguish inventory reconciliation and lock -acquisition from repository transfer and file validation. A repository-backed -skill's authored `SKILL.md` is not its complete runtime payload: the generated -discovery entrypoint includes the pinned repository under `.paperclip-repository`. -Measure the delivered bundle's file count and bytes when reproducing its cost. - -Runtime preparation refreshes the company inventory once per listing. Local and -catalog directories remain direct sources, so edits are visible on the next -preparation. Explicit version selections still use their stored snapshots. - -Reconstructed skills use `__runtime_cache_v1__///files` -beneath company skill storage, with a sibling manifest of paths, sizes, and SHA-256 -content digests. Every warm hit validates the manifest and exact file contents; -it does not fetch upstream, rewrite files, or remove directories. The fingerprint -includes installed source identity, revision, file inventory, and stored Markdown, -and excludes display names, stars, and general update timestamps. Manifests stay -outside the directory delivered to agents. - -GitHub and skills.sh imports are cached only when pinned to a full commit SHA. -Remote freshness is explicit: update or reimport selects a new revision, including -supporting-file-only changes. A branch advancing upstream does not change an -installed revision. Legacy mutable refs retain uncached behavior until updated. -URL-only skills use stored Markdown. An unavailable new revision reports missing; -it never silently reuses an older revision. Stored `SKILL.md` remains a fallback, -but missing supporting files prevent publication of a reusable partial cache. - -Builds publish read-only files and directories from unique staging directories. -A skill-scoped lock serializes builds and cleanup across processes. Cold builders -recheck that the skill still exists under its original key before reading files -and before atomic publication. Existing valid -revisions stay readable during updates. Invalid entries are quarantined in the -same skill cache root for inspection; rename/removal cleans up that skill's cache. -Read-only listings validate caches without downloading or repairing them. A -publication lock left by an abruptly terminated process is reported for operator -cleanup; remove it only after confirming its recorded PID is no longer running. - -Run `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-skill-preparation.ts` for an isolated embedded -PostgreSQL benchmark with 114 mixed skills and at least 400 remote files. It -reports one cold sample and ten warm samples (one in a new process), refresh and -fetch counts, rebuilds, missing entries, and content checks. Upstream responses are -deterministic fixtures; use real deployed run spans for user-facing latency. - -## Invocation and native turn attribution (v1) - -The existing opt-in OTLP endpoint gate also controls invocation timing. No new -exporter is installed. Generic plugin timing uses `span.record`, still gated by -`environment.drivers.register` or `workspaces.manage`, and the host-owned invocation ID. The host clamps -names, attributes, status and timestamps again. A worker cannot choose a trace -parent. Existing provider `startSpan` calls are retained as children of the -worker receipt. `ctx.tracer.withSpan` supplies async-local nesting; it is optional -for compatibility with older SDK fixtures. The host generates each child span's -identity. Receipts share one invocation-wide budget of 256 nodes and 12 levels. The worker root supports up to 24 hours; short children keep their widths on long execute calls. Nested timestamps share one monotonic origin. Missing, invalid, -expired and cross-invocation context cannot select another call's parent. - -| Boundary | Coverage | Owner and clock | Meaning / limit | -| --- | --- | --- | --- | -| Dispatch / receipt / return | `execution.plugin_dispatch` β†’ `plugin.worker` | Host invocation; worker monotonic durations anchored to its epoch start | Outer minus worker time is an unattributed dispatch/scheduling remainder. It is not network latency. | -| Repeated host reads | `plugin.host.identity.read`, `config.read`, `progress.read`, `progress.write` | Worker invocation, one span per awaited call, bounded by receipt budget | Round trip and host service together. Span count is call count; overlapping reads are not additive. | -| Plugin coordination | `plugin.storage.queue`, `storage.execute`, `runtime.queue`, `runtime.prepare`, `runtime.reconcile`, `runtime.verify`, `workspace.prepare`, `workspace.failure` | Invoking plugin, monotonic durations | The plugin keeps its locking and authorization owners. `plugin.reused` is boolean. `plugin.failure_reason` on `workspace.failure` is a closed workspace operation failure reason or `unknown`. | -| Completed remote lifecycle | `plugin.remote.lifecycle`, linked to the host-owned observer | Bounded prior monotonic durations, up to ten minutes old | Independent lifecycle trace; never a child of a later polling request. The worker cannot supply link IDs. | -| Remote operation receipt | `plugin.remote.request` β†’ `plugin.remote.operation` | Remote monotonic offsets, checked against local round trip and parent bounds | Closed operation classes cover mount refresh, guest setup, process execution, scripts and cleanup. No remote wall-clock subtraction. | -| Retained/cold transport | `runner.transport.attach`, `resume`, `cold_start`, `prepare`, `registration`, `provider_identity` | Controller process, current active context on every operation | A retained transport must not capture the preceding run's trace. These parents overlap their children. | -| Command and provider admission | `runner.transport.turn_start`, `command_ack`, `started_event`; optional ACPX `runner.remote.turn_start` β†’ `runner.sidecar.turn_start` β†’ `status_read`, `prompt_started` | Local monotonic spans; remote numeric durations anchored at receipt | Command ACK proves durable command completion. ACPX `promptStarted` proves the runtime prompt-start barrier, not a model token or provider queue exit. SDK-internal work stays opaque. | -| Durable events | `runner.event.persist` with bounded event kind | Host-local monotonic interval | Receipt through event transaction commit, before live publication. Duplicate receipts are not new provider starts. | -| First event / first text | Existing `provider.time_to_first_agent_event`, `task.time_to_first_token`; `response.first_nonempty_text` | Durable event observations | First event can be a tool; text must be nonempty and pass the existing deliverable-text observer. | -| Completion | `completion.request.first`, existing `tool.request.input_stream`, `completion.tool_accepted`, `completion.result_committed` | Same turn; provider tool identity retained only in memory | Input-stream end is the last observed argument update. `run.result.proposed` is the validated tool receipt; `completion.result_committed` follows the successful result transaction; the earlier `run.result.accepted` event alone does not prove persistence. Neither certifies response publication. | -| Reply commit / publication | Existing response transaction/write/publish phases | Response owner, transaction and local publication | Keep separate from tool acceptance and first text. | -| Input/context | `inputBytes`, `toolSchemaBytes`, `toolCount`, `contextReused`, `tokenDataAvailable` | Controller numeric/boolean metadata | Counts only. No prompts, instructions, tool arguments or credentials. Token/cache counters remain unavailable unless supplied by a provider with known semantics. Model/effort stay in existing effective run settings. | - -Remote receipt spans use their measured duration and causal bounds. Their absolute -placement inside a round trip remains uncertain by transport/scheduling time. -Never add parent and child widths, sum concurrent host reads, or label this -remainder inference or network delay. Failure receipts carry status codes without -exception text. Incomplete or malformed receipts are discarded without affecting -execution. A receipt cap can omit further detail; it does not truncate work. - -The linked private implementation owns its remote stage names and protocol. -Core contains generic operation semantics only. Before using this coverage to -rank optimizations, deploy a compatible source set and inspect actual collector -exports for parentage, concurrent ownership, widths and closed failure spans. -Unit tests do not establish ingestion or a latency improvement. - -The shared ACPX instrumentation does not change any provider profile identity: -Copilot, Grok and Pi digests attest their upstream releases, not Runner source. -Provider packages, models and permission defaults are unchanged. diff --git a/doc/run-log-events.md b/doc/run-log-events.md index b1d45b17acd..76b3f84ff11 100644 --- a/doc/run-log-events.md +++ b/doc/run-log-events.md @@ -1,1263 +1,24 @@ # Run-Log Events -Run-log events write to the `heartbeat_run_events` table -(`packages/db/src/schema/heartbeat_run_events.ts:6-20`). They are not -Paperclip Telemetry events, and they are not OpenTelemetry exports. A run-log -event needs no operator endpoint. - -## ACPX sidecar shutdown - -The ACPX sidecar emits `runtime.diagnostic` with code -`sidecar_shutdown_requested` once at the first local shutdown request, before -waiters are removed. Its JSON message records the local reason, pending tool, -input and permission counts, completion-normalization count, and at most 16 -oldest pending tools. Each tool sample contains elapsed milliseconds and a -SHA-256 call ID. Hash the corresponding `runtime.tool_called` call ID to match -it. Arguments, results, operation names and raw call IDs are excluded. -The existing diagnostic path also writes this message to sidecar stderr. -A controller that has already stopped may not ingest the frame. Stdin EOF -identifies a closed controller pipe, not the upstream reason for that closure. -This is local run evidence; no first-party Telemetry event or OpenTelemetry -span is added. Retry, cancellation and authorization behavior stay unchanged. - -## Startup reuse evidence - -`run.startup.reuse` uses schema `paperclip.startup-reuse.v1`. Each receipt names -one layer: `environment` (the lease), `workspace`, `runner`, `provider_process`, -or `provider_session`. Its state is `reused`, `fresh`, `unknown`, or -`not_applicable`. Reasons use a closed set: `verified_reuse`, `created`, -`replacement`, `not_found`, `expired`, `identity_mismatch`, `resume_failed`, -`configuration_changed`, `tools_changed`, `workspace_changed`, `cache_miss`, -`not_applicable`, and `evidence_missing`. No paths, identifiers, configuration -values, or exception messages enter these fields. - -A restart's reattachment to a remote Runner resumes the run's own retained -sandbox lease and never acquires another one. Its `environment` receipt is -`reused` / `verified_reuse`. Earlier releases reported the lease's original -acquisition there, so a reattachment to a sandbox the run created showed -`fresh` / `created` although no sandbox was created. - -Provider-process receipts carry a cumulative `providerLaunchCount` for the run. -`null` means unobserved; zero requires a physical launch observer. Direct ACPX -processes count initial launches and respawns, including launches during retries. -On the remote process-session bridge, ACPX observes the host relay rather than -the sandbox provider. Provider-process reuse and launch counts stay unknown on -that transport. Native Runner session events establish session continuity, but -do not establish provider-process reuse or its launch count. Those values stay -unknown without physical evidence. The final ACPX receipt does not wait for -persistence, so a stalled diagnostic sink cannot hold the staging lease. -The existing successful `runner.process.launch` performance receipt establishes -a fresh Runner process, independently of provider-session continuity. - -The Product E2E result stores `startupReuse` by run, and its dashboard displays -each layer, its reason, and the launch count. `warm` requires all applicable -layers to be reused and zero observed provider launches. `cold` requires all -applicable layers to be fresh. Some reuse with other fresh or missing evidence -is `partial_reuse`; otherwise the label is `unknown`. Fresh decisions dominate -later reuse receipts within a run. Legacy runs without receipts retain unknown -layers and a null launch count. Session IDs and follow-up turns never establish -warmth. These events stay in the local run log; they add no first-party Telemetry. - -## Ordinary event persistence - -Events without a native source identity or retry-exhaustion receipt use one -atomic PostgreSQL statement. The statement checks the stored run's company and -agent, allocates its next sequence, and inserts the event. An insertion failure -also rolls back the sequence allocation. The returned row uses the existing -column decoders, including the lossless payload codec. Callers still wait for -the durable write before they publish or acknowledge the event. Native-source -replay and retry-exhaustion deduplication retain their existing transaction path. - -## Legacy case handoff cleanup - -`legacy.case_handoff_cleanup_intent` records local cleanup ownership before -dispatch, then binds it atomically to the observed process identity on spawn. -It includes the host, controller/run generation, PID/group/start identity, and -owned scratch descriptor or explicit `null`. It does not prove termination. -`legacy.case_handoff_cleanup_verified` records the later stop proof before -scratch removal. Restart recovery rechecks process, controller, workspace and -ownership gates before completing cleanup and releasing the handoff. -These controller receipts are cleanup bookkeeping, not evidence of agent task -progress, and are excluded from run-liveness action counts. -The controller authors both events on the system stream. Adapter ingestion -rejects the reserved `legacy.case_handoff_` namespace. These events stay in the -instance run log and add no first-party Telemetry or OpenTelemetry export. - -## Provider-pack preparation timing - -Every `run.performance.span` payload also carries `wallStartedAtMs`, the span's own -start time in epoch milliseconds (since 2026-11-09). Span rows can be written late -through deferred diagnostics, so readers take a span's start from this field (or -from the trace start plus `startOffsetMs`), never from the row's write time. -The preparation spans of a native run (`heartbeat.queue`, `heartbeat.prepare_*`, -`environment.*`) are always written this way: in the order they ended, while the -provider starts, and before the run's trace finishes. They can appear after the -first Runner events of the run. - -`run.performance.span` also records `task.time_to_first_token` once per native -dispatch at the first newly committed deliverable text: assistant or reasoning -`item.delta`/`item.completed`, a redaction tail released at completion, or an -accepted run-result summary. Its -`durationMs` starts at the earliest accepted input. The accepted comment or -answered question is the start only when its span is the earliest preparation span. -A comment coalesced into an already-queued run starts after run creation, so that -run uses queued run creation. The durable wake request's `requested_at` replaces -the start when it is earlier (deferred, coalesced and retried wakes). Without -preparation history, the start is dispatch and the anchor is `run_created`. It ends at Core commit callback. `anchor` names the start: -`comment`, `question_response`, `wake_request` or `run_created`. `startOffsetMs` is -zero, or negative when a wake request precedes the run trace. `eventKind` is -`agentMessage` or `reasoning`. The wake request time is read after the first text, -in detached diagnostic work bounded to 2 s (a -transaction-local `statement_timeout`, so PostgreSQL cancels a stalled read), so the measurement adds no wait before -`turn.start` and never delays event delivery. The span can appear after later -run-log rows; the trace settles it before `task.provider_session.measured`. It includes preparation and startup, but excludes browser -delivery/rendering. This measures text availability, not an exact model token. -Empty text, item starts, tools, usage, result proposals, and duplicate replay do -not qualify. No deliverable text means no timing sample. A recovered dispatch can -record newly committed output from the original start; it does not reproduce -earlier output timing. The event contains no text or provider identifiers. - -## Speed budget breaches - -`run.performance.budget_exceeded` (schema `paperclip.run-performance-budget.v1`, -level `warn`) records one native run that exceeded its p95 speed budget. The -budgets come from the Phase 0 local baseline and are listed in -`doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` ("Speed budgets"). -`metric` is `time_to_first_activity` (`task.time_to_first_agent_activity`: -the first text delta, tool-call start or reasoning item; named `ttft` and -judged on `task.time_to_first_token` before 2026-11-08), `agent_boot` (end of -`heartbeat.queue` to the end of `runner.turn.submit` or `runner.session.resume`) -or `restart_recovery_control_plane` (startup recovery `ready` to the start of the -provider's saved-session load, `provider.session.load`, or to the recovered run's -first provider event when nothing reloads). That is Paperclip's own share of a -recovery: claim, prepare, runtime stage and runner launch. The full recovery -(`ready` to the first provider event) and the provider's session load grow with -session size and with how many sessions resume at once, so they are reported in -the payload as `restartTotalMs` and `providerLoadMs` and are not judged; events -written before 2026-11-09 judged the full latency as `restart_recovery`. -The payload names `adapter` (`codex`, `claude` or `opencode`), -`environment` (`local`), `startKind` (`cold`, `warm` or `unknown`; an unknown -start uses the looser budget), `measuredMs`, `budgetP50Ms`, `budgetP95Ms`, -`percentile` (`p95`), `baseline`, and, for restart recovery, `disposition`. -For `time_to_first_activity`, `measuredMs` counts from run creation: when the input waited for its -run to exist (for example a comment posted while the previous turn was still -running), that wait is excluded from the judgment and reported as `inputWaitMs`. -Each metric is checked at most once per dispatch. A restart-recovery dispatch is -judged only on `restart_recovery_control_plane`, because its first activity and boot span the outage. Time to first text (`task.time_to_first_token`) is reported without a budget. Remote runs and other adapters have -no budget yet. The event is written in detached diagnostic work after the -measurement, so it adds no wait before `turn.start`. It is a run-log row only -and never goes to Telemetry. - -The existing `run.performance.span` event also records `provider_pack.select`, -`provider_pack.discover`, `provider_pack.verify`, `provider_pack.verify_preinstalled`, -`provider_pack.link_preinstalled`, `provider_pack.clear_stale`, `provider_pack.upload`, -and `provider_pack.verify_uploaded`. Selection records `stateSource` as `staged`, -`preinstalled`, or `uploaded`. Its closed reason is `verified_pack_reused` or -`cached_packs_unverified`. A verification failure describes an unusable candidate; -it does not identify a particular compatibility mismatch or a failed run. -Harness startup also records `harness_state.inspect`, `harness_state.path_check`, -and `harness_state.launch_material` for remote state inspection and fresh launch -material. These events retain the existing schema and carry no paths or exception strings. -They also become native Observability spans only with an operator OTLP endpoint. - -## Local runner startup sub-steps - -`run.performance.span` splits a local Runner startup (`runner.session.bootstrap` or -`runner.session.resume`) into three sequential children of `runner.session.startup`: - -- `runner.runtime.stage`: controller-side launch preparation before the Runner - process exists (runtime context, provider home, launch profiles; on resume, the - durable-state checks and authority rotation as well). -- `runner.process.launch`: from spawning `paperclip-runnerd` until its first - authenticated connection to the PRP control plane. An exact adoption of a live - Runner records no launch, so this span keeps meaning a fresh Runner process - (see startup reuse above). -- `provider.session.load`: from that authentication until the provider session - identity is confirmed. It covers `run.prepare` and `session.open`, or on resume - the `run.attach` that restores the provider and the recovery probe. - -Each carries `mode` (`bootstrap` or `resume`), `target: "local"`, and `outcome` -(`failed` when startup fails inside the step). The Runner transport reports each -milestone synchronously; the controller writes the row as deferred diagnostic -work, so the run-log write never delays `turn.start`. Remote launchers keep their -own `runner.runtime.stage` and `runner.process.launch` spans. These rows stay in -the instance run log; with an operator OTLP endpoint they are also Observability -spans. They are not Telemetry. - -## Native PRP Run-Log Events - -Provider startup command failures and failed provider turns can carry a -`providerFailure` cause with schema `paperclip.provider-failure.v1`. The cause -retains a normalized category, a bounded and redacted provider explanation, the -startup/turn phase, and available status, code, request ID, or retry time. Provider -adapters extract these fields; arbitrary exception objects, request bodies, -headers, environment variables, and stacks are excluded. Unknown categories retain -their explanation. The cause stays in company-scoped run records and the run log; -it is not added to first-party Telemetry, Observability attributes, or Sentry -failure reports. The reportable run error keeps its existing generic text, except -that an ACPX startup rejection keeps Runner's closed rejection text and its -classification. Startup receipts retain the cause across controller restart before -any provider turn exists. - -When a failed ACPX control closes the session before polling, the resulting -`turn.failed` event retains that control's closed-code error. A later closed-session -check does not replace it, and a closed code does not infer a provider explanation. -The terminal still reports `recoverable: false` and no observed provider terminal; -it does not prove termination or authorize replay. - -Every ACPX session ensure rejection carries a startup cause. When the provider -returned no ACP error, the cause keeps the rejection's own redacted message with -category `unknown`. A provider can type the failure with ACP's -authentication-required code, or with a known error type or `errorKind` in the ACP -error data. The cause then uses that category, and the rejection classification is -`provider_`, such as `provider_quota_exhausted`, instead of -`session_ensure_failed`. - -Completed OpenCode builtin `bash` parts preserve the numeric process exit from -`state.metadata.exit` in `tool.execution.completed.payload.exitCode`. The tool -status stays `completed` even when the command exits with a nonzero code. This -status describes completion of the tool call, not success of the command or the -native run. Metadata adds an exit only when it is an integer. Conflicting supplied -legacy exits stay unknown. Existing numeric `state.exit` values remain supported; -metadata on other tools cannot add process-exit proof. - -An edit tool call can carry `fileChanges` on its `tool.execution.*` payload. -Each entry names one workspace-relative `path`, its `operation` (`create`, -`modify`, `delete` or `rename`), `previousPath`, `additions`, `deletions`, a -unified `diff` and `diffTruncated`. The runner drops paths outside the working -directory, redacts secrets in each patch, and bounds a patch to 32 KiB and one -call to 32 files. Counts cover the whole patch even when the patch is shortened. -Codex `fileChange` items produce these rows. Claude Edit and Write calls produce -them from the ACP `diff` content. Codex `turn/diff/updated` also records the -whole-turn diff as `workspace.change.updated` (`paperclip.workspace.diff.v1`). -The task chat shows each patch under its edit row. It ends the turn with the -change set, or with one built from the edit rows when the harness reports no -whole-turn diff. - -`auth.credentials.updated` is a runner-produced PRP event for a change to a -supported credential store, including account identity changes and later token -refreshes on the new account. The bundled transport awaits a host commit callback -before acknowledging delivery, including during warm idle and provider cleanup. -The callback writes directly to the bound run through the shared run-log writer. -Its separate credential source namespace avoids collisions with driver-normalized -source sequences and deduplicates retries using the original PRP event identity. -The original envelope is retained in `prpEvent`. Its payload contains only a -provider, an opaque binding ID, a revision, and a detection method. It contains -no credentials, account identity, file path, or credential fingerprint. The -consumer can use the event to read the updated credentials through its existing -private credential channel. This run-log event stays in the instance database. - - -`native.recovery.execution_reclaimed` records an atomic same-run recovery claim -that restores a released task execution binding. The claim locks the task, -coordinator, and original run in that order. It preserves the assignee and checks -current execution/checkout owners, other live work, dependencies, pause holds, -and stop/successor records before restoring the binding. Attachment authorization -keeps its existing requirement for that exact binding. Remote retry claims also -require the exact saved controller authority and reject a newer task run. The -event does not prove that a remote provider survived or permit provider replay. -Skipped local retry candidates wait 30 seconds before recheck. Remote admission -holds use the existing bounded system-recovery backoff independently of provider -attempts. Permanent ineligibility or policy exhaustion records the terminal -automatic recovery disposition and board action without releasing the retained -execution binding, provider, or checkpoint. These settled holds schedule no -further retry. Repeated blocked visits record only a changed hold reason. -The same durable hold prevents background stale-lock cleanup and queue release -from clearing either task binding until the remote termination receipt contract -is satisfied. A terminal heartbeat status is not a provider-stop receipt. -Undated retries sort first. A failed candidate cannot discard earlier committed -claims. Queued waiters do not block the original execution owner from recovery; -they still wait for that owner to settle before normal queue promotion. -This is local run-log data. - -Fresh remote Codex model substitution emits `runner.model_fallback` on the -system stream with `warn` severity. Its payload contains `requestedModel`, -`effectiveModel`, and `codexCliVersion`. It records a preparation choice before -provider launch, not a failed turn or a retry. The same substitution appears as -a system warning in the task conversation. It remains local run-log data. - -The hidden native coordinator writes each validated PRP event to the bound -run's existing event stream before it acknowledges the runner. The row keeps -the PRP `eventType`, source instance, source event ID, source sequence, protocol -schema version, and a SHA-256 digest of the canonical stored envelope. Its -payload is `{ "prpEvent": }`. - -Native API tool events omit HTTP request and response bodies from durable history. -This includes serialized provider `contentItems`, legacy HTTP receipts and -`call_api` arguments. Saved receipts retain the operation and call IDs, success -flag, HTTP status and content type when present. The body becomes -`***REDACTED***`, with `apiPayloadRedacted: true` and `apiPayloadSha256`, a SHA-256 -digest of the original payload. These digests distinguish duplicate events from -changed payloads; they cannot recover omitted bodies for debugging or replay. -The model receives the original authorized response. Credential-bearing mutation -receipts use encrypted storage separately from this run log. - -Checkpoints and completion records apply the same redaction. Completion retries -compare stored and expected payloads through the control-plane port's idempotent -`normalizeReplayPayload` projection, so a saved redacted result can be confirmed -without restoring API bodies. Ports without a projection compare their original -payloads. Repeated redaction preserves the original digest and detects changed -private payloads. - -The native runtime passes this projection through backend recovery to the Codex -driver. Recovery compares saved semantic results and turn fingerprints with -provider history through the same representation. A private body change retains -its distinct digest and fails recovery; an unchanged body does not require -restoring its plaintext into the checkpoint. - -PostgreSQL JSONB cannot represent NUL (U+0000), which can occur in command -output such as Vite virtual-module paths. The run-event payload column uses a -lossless storage codec for these events: the JSONB projection renders NUL as -the literal `\u0000`, and the reserved `$paperclipRunEventJsonV1` field contains -the original serialized JSON as a doubly escaped string. Ordinary payloads -retain their existing representation. Drizzle reads restore the exact stored -payload before replay, hash validation, redaction, or API presentation. SQL -queries can still inspect ordinary routing fields in the projection; raw SQL -readers of the whole payload must apply `decodeRunEventPayload`. The column -remains JSONB and requires no schema migration. - -The writer locks the native `heartbeat_runs` row and allocates the existing -per-run `seq` cursor. A byte-equivalent retry reuses the first row; a changed -retry or source-sequence gap is rejected. Company, issue, agent, run, session, -and runner-source bindings must match the persisted native run. Bootstrap -tickets, reconnect leases, authentication proofs, encryption keys, and raw -credential material are never written to the run log. - -These records remain run-log events. They do not create an OpenTelemetry or -Paperclip Telemetry export, and legacy adapters do not use this writer. - -## Semantic Settlement Diagnostics and Retry Logs - -When native tool arguments fail their authorized JSON Schema, the runner records -`harness.diagnostic` with code `semantic_tool_denied` and `paperclipExecuted: false`. -Its optional `inputSchemaGuidance` retains the same safe repair guidance sent to -the provider: at most three authorized schema locations within 512 bytes. -It includes no rejected arguments, submitted values, or unknown input keys. -The rejected call does not create a semantic input event or execution receipt. -Validated runner diagnostics retain this scalar metadata even when the operation -is an API or case tool. API request and response bodies keep their existing redaction. - -If a declared native completion tool is rejected and the turn ends without an -accepted replacement report, the runner emits `session.failed` with code -`semantic_completion_rejected` and `recoverable: false` before the provider's -original turn terminal. It emits no successful result proposal or human review -request. Core uses its existing permanent native failure path and does not start -a missing-result recovery turn. This failure also precedes terminal goal updates; -Core records the original turn terminal before rejecting goal success or further -goal continuation. Recoverable goal failures retain their existing behavior. -The rejection survives same-turn restoration; -a new turn or run binding clears it. A later accepted completion report keeps -its normal authority, and ordinary turns with no completion-tool attempt keep -their existing fallback. - -Before interrupting a Codex turn, the runner durably records a -`harness.diagnostic` event with code `provider_interrupt_requested`, the stop -reason, provider turn ID, and pending tool call IDs. A harmless identical result -replay records `semantic_tool_result_duplicate` with warning severity, call ID, -operation ID, and result digest. It does not fail the task or create a user -attention request. A true conflict includes the call and operation IDs and both -result digests in its error; full arguments belong to the canonical semantic -input record, not the error message. - -When the tool bridge rejects a recorded result for a call it still holds, for -example a result envelope that names another operation or a value over the -retained limit, the runner still settles that call. It returns a bounded, -non-retryable error to the provider and records -`semantic_tool_result_undeliverable` with warning severity, call ID, operation -ID, the rejection reason, and the rejected result's digest. The provider is not -left waiting for its own client timeout, and the call does not block later -turns or runs. - -`run.attach` settles calls that a closed provider turn left pending. The -provider stopped waiting when that turn ended, and the new run's controller -never delivers the old run's results. Each such call is recorded as a -non-retryable failure with an unknown outcome, and the attachment records one -`semantic_tool_result_expired` diagnostic with the call and operation IDs. A -call that a live provider turn still waits on keeps blocking attachment as -`durable_pending_tool_calls`. - -Incomplete close emits the bounded `native_session_settlement_incomplete` -runner diagnostic. Its evidence includes runner suspension, provider drain, -final provider state, pending call/operation/source-event IDs and input digests, -and incomplete result-delivery command IDs and statuses. If execution and -cleanup both fail, execution retains its original error identity and cleanup is -attached as `cleanupError`. `runnerRetained: true` means a remote Runner that -could not prove its suspension was left running for the same-run -reattachment instead of being force-stopped; the diagnostic -`remote runner left alive unsettled for same-run reattachment` records it. - -Semantic settlement also includes up to 20 content-free failure records, with -the call ID, operation ID, stage (`dispatch` or `persist_result`), and cause. -Causes distinguish an oversized command, a full command journal, known storage -errors, dispatcher rejection, and other persistence failures. Exception messages -and tool results are excluded. These diagnostics do not authorize replay of an -operation whose outcome is unknown. - -Instruction writes also commit an `agent.instruction_write_attempted` activity -row and a run-scoped `instructionToolAttempts` entry before permitting the -filesystem effect. They retain the call ID, operation ID, and input digest, not -instruction text. A later transaction rollback cannot erase this attempt proof. -A missing success or definite pre-write failure receipt means the outcome is -unknown, even if the current file contains the requested text. Known validation -and stale-base failures are saved under the attempt's `failure` entry and replay -their original status, message, and details without a new write. Unknown -outcomes remain blocked until reconciled; a -committed receipt with a lost acknowledgement can replay its exact result. - -The NDJSON output log appends across repeated `begin` calls for one run. Each -new handle adds an `attemptId` to its lines. If the local file is absent and a -durable S3 mirror exists, `begin` restores that prefix before appending. A -failed restore must not replace the mirror with an empty or partial attempt. -Publishing a restored prefix uses an atomic create-if-absent operation, so a -concurrent restore cannot overwrite lines another attempt has already appended. -Earlier attempts therefore remain available for incident diagnosis. Existing -records without `attemptId` remain readable. - -These records use the instance run log and its configured storage. They add no -Paperclip Telemetry or OpenTelemetry export. - -## Omitted Unsafe Workspace Export - -`workspace_export_omitted` is an informational system event in the local run log. -Its payload is `{ "reason": "restore_unsafe_archive" }`, with `"legacy": true` -when recovering an unsafe failure from an older controller. It records that native -finalization discarded an unsafe export and continued with the accepted result. -It contains no archive names, link targets, or raw error details. It does not -create a task warning, recovery action, Telemetry event, or OpenTelemetry export. - -## Native Restart Recovery Run-Log Event - -Paperclip writes a `native.recovery.transition` event for native restart -classifications and for graceful restart suspension. A blocked remote retry -records a changed hold reason once; repeated visits with the same hold reason, -an existing future deadline, or a live lease do not append another transition. -Exhausting the automatic admission policy records its terminal disposition even -when the underlying hold reason remains the same. -This immutable run-log -record lets operators reconstruct recovery decisions without exporting data to -Paperclip Telemetry or OpenTelemetry. - -The payload contains the restart kind, recovery request id when one exists, -runner disposition, and the controller generation and provider attempt for a -claimed recovery. Live-runner adoption also records the runner PID, process -group, and process-start fingerprint. A non-claim disposition records a bounded -reason instead. Graceful suspension records the signal and confirms that it did -not create a retry run. - -The event never includes bootstrap tickets, reconnect leases, authentication -proofs, encryption keys, environment variables, provider credentials, command -arguments, or an unsanitized stderr stream. Detailed failed-attempt diagnostics -remain in the bounded `native_run_finalizations.recovery_history` ledger. - -`native.remote_runner_recovery_failed` records the closed `stage` and `reason` -when remote identity verification fails, including admission failures before the -execution coordinator opens. Unavailable probes are retried within a shared -30-second deadline; identity conflicts are not retried. The event distinguishes -an unreadable identity from a confirmed missing process and contains no command -output or credentials. Saved-state read failures use `state_read_failed`; malformed -saved JSON uses `state_invalid`. It does not itself authorize replacement or cleanup. -For `process_not_running`, the event also records `runnerLifecycle` and -`runnerSettled` from the exited runner's saved state. `runnerLifecycle` is a known -runner lifecycle, `other` for an unknown value, or `unavailable` when the state has -none. `runnerSettled` is true only when the outbox is empty and no terminal delivery, -provider cleanup or warm transition is pending. These fields explain how the runner -stopped. The event is not recorded when a scheduled retry resumes a runner that -exited after a settled suspension. - -`native.interrupted_turn_outcome_unknown` (level `warn`) is recorded once on a -failed run whose reattachment proved the Runner process gone -(`process_verification` / `process_not_running`) while its last provider turn -had no observed settlement, just before the server requests the automatic -Retry. The payload holds the provider `turnId` (or `null`), the last turn event -type, `replayed: false`, and the closed `runnerExit` fields `stage`, `reason` -and `runnerLifecycle`. It names a turn whose outcome is unknown; the Retry's -fresh session never replays it. - -For a prepared remote workspace interrupted before its first executor claim, -that ledger records `remote_execution_never_claimed` with a -`bootstrap_incomplete` disposition. The provider attempt remains zero during -classification; the executor increments it before starting provider work. -A claimed remote bootstrap (the sandbox runner was launched and may have -connected) records `remote_bootstrap_proven_incomplete` with the same -disposition. Recovery admits it only after the previous controller lease has -expired and the Postgres controller checkpoint holds just `run.prepare` and an -unfinished `session.open`, with no committed or acknowledged event. The -executor restores that checkpoint and checks the same proof before it -bootstraps again. After three provider attempts it records -`execution_recovery_budget_exhausted` instead. -A claimed remote run whose previous controller is gone, with no -`native.process_start_requested`, process identity, adapter invocation, -runner-sourced event or controller checkpoint, records -`remote_runner_never_launched` with the same disposition and budget. The -executor records `native.process_start_requested` before a backend may spawn a -runner, so its absence (with no checkpoint) proves nothing reached a runner. -A remote run that a graceful drain suspended just after its runner opened the -provider session records `remote_session_opened_without_turn` with the same -disposition. Its Postgres controller record must hold only `run.prepare` and -`session.open`, with every committed event a session-open event and no event -missing. Before that same-run bootstrap relaunches in the reused sandbox, the -executor stops the interrupted runner it can verify there, so the new runner -does not fail with `runner_ingress_bind_conflict`. -The same rule covers a remote run that its executor already settled as a -same-run `bootstrap_retry` (for example when a shutdown closed the local -runtime listener during startup) and that is due when the server restarts. - -For a local Codex, OpenCode, or ACPX turn lost while runnerd was stopped, the canonical `turn.failed` -event retains `error.code: "provider_turn_lost_on_restore"` and -`error.recoverable: true`. It records the interrupted turn, not a final task -outcome. There is no synthesized result for that interruption. An admitted -same-session continuation emits its own `turn.submitted`, accepted turn ID, and -terminal events; only the final outcome closes the run. Reconstructed thread -history retains the original error so reconciliation and live delivery agree. -An ACPX session records this only after the original provider lifetime loses -its fence quorum and the same conversation is restored. Otherwise its -`turn.failed` carries `error.code: "acpx_active_turn_recovery_closed"` with -`error.recoverable: false`, and the run closes with an honest restart message. -After an ACPX restore, the controller can still deliver results for semantic -calls of the lost turn. Runnerd acknowledges each one as `retired` without -sending it to the restored provider, so the calls settle and live semantic -dispatch resumes for the continuation. A restored ACPX provider is a new process -whose MCP request ids restart, so its semantic `callId` values carry a -per-process prefix (`r:`). They never reuse an earlier call's id in -the same run. - -## Native Local Process Stop Evidence - -The server writes `native.local_process_stopped` in the same transaction that -clears a local run's process identity, after it verifies that its PID and process -group are absent. The payload contains only those process IDs. Remote process -IDs are never checked against the control-plane host. - -A crash can land after `native.process_start_requested` and before the runner's -process identity is recorded. Continuation admission then writes -`native.local_process_stopped` with null process IDs and -`basis: "unidentified_runner_absent"`. It does this only on the host named in -the launch event's `hostname` payload, after it checks two things. First, no -process carries the run's `--run-id` runner argument. Second, no -saved runner control-plane state (canonical or quarantined) belongs to the run; -a provider starts only after its runner connects and writes that state. - -The server writes `native.process_start_requested` before a backend can spawn, -and `native.process_identity_recorded` when it stores a new native process -identity. Either invalidates an earlier local stop receipt, including a crash -before the new PID callback. Continuation admission accepts only the latest -server-authored event among these three types; provider -source events cannot supply stop authority. These records stay in the local run -log and do not add Telemetry or OpenTelemetry data. - -The server writes `native.prior_owner_fence_refused` when its runner-state -check refuses a run before any launch, because the conversation's saved runner -state belongs to an earlier run that cannot be proven finished. The payload -holds only the closed refusal `reason` (for example -`prior_owner_terminal_state_indeterminate`). Continuation admission reads it, -with no launch, identity, `adapter.invoke` or runner-sourced event, as proof -that the failed run never had a runner, so Retry and new input need no stop -record for it. - -Explicit continuation writes `native.stopped_conversation_verified` after it -verifies the retained worker and provider processes stopped. Its payload binds -the requesting `runId`, retained `ownerRunId`, native session, runner instance, -process IDs, retained-state fingerprint, successor run, Retry receipt, and local -workspace identity. The owner can be a preceding terminal -run when the requesting attempt failed before launch with the same process -identity. The server checks host-bound process birth and independent provider -receipts before retiring cleanup ownership. The event does not settle uncertain -actions or authorize replay of the old session. - -## Verified Local Codex Replacement Evidence - -The server writes `native.stopped_text_turn_verified` in the same transaction -that schedules a fresh successor for a stopped local Codex run. It records the -runner and provider process identities, retained-state digests, provider thread -and turn IDs, and IDs of exactly receipted task-completion calls. The server first -checks the complete turn inventory, process-stop receipt, and execution binding. -Unknown actions or changed retained state prevent this event and replacement. - -The record documents why the old execution can be retired. It does not make the -old session resumable, rewrite provider files, or authorize replay on its own. -It remains in the local run log and adds no Telemetry or OpenTelemetry export. - -## Sandbox Startup Run-Log Event - -Paperclip writes one `run.startup.step` event to the run log for each bring-up -step. This event is a run-log record, not a first-party telemetry event. The -generated telemetry contract does not cover it, so this section is its canonical -contract. - -The event payload carries only three fields. - -| Field | Type | Meaning | -| --- | --- | --- | -| `step` | string | The bring-up step name, for example `stage.sync`. | -| `durationMs` | number | The wall time of the step. A skipped step reports `0`. | -| `outcome` | string | The step outcome (`ok`, `skipped`, or `failed`). | - -The event no longer carries the per-step round-trip count or the provider -duration fields. It dropped `roundTrips`, `providerExecMs`, `providerGetMs`, -`createRuntimeMs`, and `ensureSessionMs`. The startup spans in -[`doc/observability.md`](observability.md) carry that detail now. The -`sandbox.exec` child spans hold the round-trip and provider durations. The -`acp.handshake` step span holds the create-runtime and ensure-session -sub-times. - -To read the detailed timing, use the startup spans. The spans need an OTLP -endpoint. A run with no endpoint keeps only the three run-log fields above. - -## Run Phase Timing Run-Log Event - -Paperclip writes one `run.phase.timing` event to the run log for each -run-lifecycle phase. This event is a run-log record, not a first-party telemetry -event. The generated telemetry contract does not cover it, so this section is its -canonical contract. The producer is `emitRunPhaseTiming` in -`packages/adapter-utils/src/acpx-engine/startup-timing.ts`. - -The event payload carries only three fields. - -| Field | Type | Meaning | -| --- | --- | --- | -| `phase` | string | The run-lifecycle phase name from the closed allowlist below. | -| `durationMs` | number | The wall time of the phase. A negative or a non-finite value clamps to `0`. | -| `outcome` | string | The phase outcome (`ok` or `failed`). | - -The `phase` field is one member of a closed, low-cardinality allowlist. The -producer drops any event whose phase name is outside this allowlist, so a -free-form label never reaches the run log. The allowlist has twelve phase names. - -| Phase | Meaning | -| --- | --- | -| `place_workspace` | Place the run workspace. | -| `start_transport` | Start the agent transport. | -| `create_runtime` | Create the agent runtime. | -| `ensure_session` | Ensure the agent session exists. | -| `configure_session` | Configure the agent session. | -| `prepare_turn` | Prepare the turn. | -| `turn` | Run the turn. | -| `end_session` | End the agent session. | -| `settle_reuse` | Settle the session for reuse. | -| `stop_transport` | Stop the agent transport. | -| `sync_back` | Sync the workspace back. | -| `release_staging_lease` | Release the staging lease. | - -The payload never carries a command, an argument, a path, an environment value, -or a raw identifier. The event rides the `ctx.onEvent` run-event bridge and is -run-log-only. It needs no OTLP endpoint. - -## Workspace Restore and Settled Run Measurements - -The heartbeat controller writes these records with `eventType: "lifecycle"`, -`stream: "system"`, and `level: "info"`. The event name below is its `message` -and its payload's `phase`. They remain in the instance run log and add no -Paperclip Telemetry or OpenTelemetry export. - -| Event | Payload fields besides `phase` | Meaning | -| --- | --- | --- | -| `workspace_checkpoint` | `mode`, `scannedEntries`, `changedFiles`, `payloadBytes` | Metrics for a validated sparse checkpoint, or a full restore fallback. | -| `workspace.restore` | `durationMs` | Wall time around native workspace restoration. | -| `task.run.measured` | `durationMs`, `outcome`, `durableSettlement` | Wall time through terminal owner cleanup, with the persisted terminal run status and `durableSettlement: true`. | - -For `workspace_checkpoint`, `mode` is `"sparse"` or `"full"`. Sparse metrics -count snapshot entries in `scannedEntries`, regular files in the validated -downloaded payload in `changedFiles`, and their total bytes in `payloadBytes`. -Deleted entries and symlinks do not add to the regular-file count or byte total. -Full fallback reports all three counts as `null`: unavailable, not zero. A run -can emit multiple checkpoint records while restoring its repositories. These -payloads contain no paths, filenames, file contents, or credentials. - -`workspace.restore` is emitted when the timed restoration exits, including an -exception. It has no outcome field and does not by itself prove restore success. -Its producer is the native workspace sync `onPhase` callback. - -`task.run.measured` is emitted only after the controller observes a terminal run -with no remaining native runner ownership. Its duration starts at `startedAt`, -or `createdAt` if the run never started, and includes preparation, execution, -finalization, and cleanup. It is not model-only time; the creation-time fallback -also includes time spent queued. `outcome` is the actual terminal status, such as -`succeeded`, `failed`, or `cancelled`. Provider turn completion can precede this -settlement record. The record does not imply exactly-once emission across -controller recovery, and failure to write it is logged without changing the -already-settled outcome. - -## Storage outage waits - -A run waits out a file-storage outage on its critical path: when it starts its -run log, restores its saved workspace, or saves it. While it waits, the run's live -status says so, with phase `waiting_for_storage`, for example "Waiting for file -storage before starting the run log (45s so far)". That status lives in server -memory and on the live-event stream only. It writes nothing to the run row or -the run log during the wait. - -When the wait ends, the controller writes one record with `eventType: -"lifecycle"`, `stream: "system"`, and `level: "warn"`. Its message is "Waited Ns -for file storage before ." or, when storage stayed unavailable, "Waited Ns -for file storage before ; it was still unavailable." Its payload holds -`reason: "storage_outage_wait"`, `purpose` (the step), `waitedMs`, and -`recovered`. The record stays in the instance run log. - -When the outage outlasts that wait, the run fails. Only a run recorded with -`errorCode: "storage_unavailable"` gets a storage wait: a bounded retry -(`retryReason: "storage_unavailable_wait"`) that starts again after storage is -back. Its error says "Paperclip will retry automatically when storage is back" -only while that retry is scheduled. When the retry budget is spent, the error -says so and asks for Retry. When no storage wait follows, the error says -"Paperclip is not retrying it automatically. Retry or send a message to -continue." That happens when a native recovery already holds the run under -another code (for example `native_continuation_requires_reconciliation`, where -its files are kept and its unfinished turn is not replayed), or when the retry -is refused or cannot be written. A run saved by an earlier release with the -retry promise under another code reads with the same Retry text, with no -migration. - -## Durable storage timing - -A saved remote workspace is one encrypted archive in durable object storage. -Restoring it before a turn, and saving it after one, runs these steps as named -run phases: - -| Phase | Step | -| --- | --- | -| `storage.snapshot.download` | Get the encrypted archive from storage. | -| `storage.snapshot.decrypt` | Hash the downloaded archive and decrypt it to private scratch, at the same time on worker threads. | -| `storage.snapshot.verify_plaintext` | Hash the decrypted archive against its sealed digest while copying it beside its destination, on a worker thread. | -| `storage.snapshot.publish` | Link the verified copy to its destination. | -| `workspace.snapshot.sandbox_restore` | Upload and unpack the archive in the sandbox. | -| `workspace.snapshot.sandbox_upload` | Inside `sandbox_restore`: copy the archive into the sandbox. | -| `workspace.snapshot.sandbox_extract` | Inside `sandbox_restore`: verify and unpack the archive, then publish the restored folder. | -| `storage.snapshot.prior_attempts` | Check whether an earlier upload attempt already landed. | -| `storage.snapshot.encrypt` | Hash the archive, encrypt it and hash the encrypted file, in one pass on a worker thread. This is the only full read of a downloaded archive: one whose digest or size differs from the sandbox's receipt fails here, before any upload is recorded. | -| `storage.snapshot.digest` | Record the digest of the encrypted file. The `encrypt` pass computes it, so this step takes no time. | -| `storage.snapshot.put` | Upload the encrypted archive (multipart above 16 MiB). | -| `storage.snapshot.verify` | Confirm the stored object (its size when the backend checked every part, otherwise a full read-back). | - -A restore runs inside run preparation, so each of its steps of 50 ms or more is -a `run.performance.span` row of the run. A save runs after the turn. Its step -durations are recorded on its `remote_workspace_snapshot` workspace operation, -in `metadata.saveTiming.storage`: -- `priorAttemptCheckMs`, `encryptMs`, `digestMs`, `putMs` and `verifyMs`; -- `bytes`; -- `requests`, the storage requests the save sent. These are `requests`, - `attempts` (SDK attempts across them), `failedRequests`, `retryDelayMs` (SDK - waits between its own retries), `parts`, `throttled` (part sends the backend - rate-limited with 503, 429 or SlowDown) and `backoffMs` (the uploader's waits - before resending a part). - -A restore also records its step durations, `bytes` and `requests` on its -`remote_workspace_restore` workspace operation, in `metadata.restoreTiming` -(schema `paperclip.remote-workspace-restore-timing.v1`). A save that reuses an -earlier upload still reads it back; that read-back is counted in `verifyMs`. -A throttled part send counts in `throttled` even when it ends the retries, -and a multipart abort sent during cleanup counts in `requests`. - -These are counts and durations only: no object keys, paths or provider text. - -## System notices and final replies - -Automatic comments with a stored `system_notice` presentation, including -workspace-ready notices, remain available as progress information. They do not -occupy the run's final reply slot. Final response selection still preserves -ordinary explicit agent comments and excludes receipt-bound generated progress. - -## ACP terminal failure diagnostics - -The shared ACP adapter engine preserves typed terminal session failures in the -run error, the `acpx.error` transcript record, and -`heartbeat_runs.result_json.terminalSessionFailure`. The structured diagnostic -contains the provider category, title, and details. It works when raw provider -tracing is disabled. The existing UI and CLI render the diagnostic as an error, -not as assistant output or an automatic task response. -Issue continuation summaries and session-compaction handoffs retain only the -generic failure category; provider diagnostic prose is not copied into prompts. - -Both pinned ACPX patches pass complete title and detail strings to the in-memory -callback. The engine redacts configured environment values (including resolved -secrets with arbitrary variable names), launch environment values outside a -closed allowlist of public process settings, known boolean flags, and run identifiers, -connection URL passwords, the run API key, and common credential forms before -truncation. It removes control characters, -retains line breaks for JSON and stack traces, and preserves up to 4,096 title -characters and 24,576 detail characters. These bounds also keep the escaped -transcript JSON below the server's 64 KiB chunk limit. Longer fields end with an explicit -omission count and appear in `truncatedFields`. The error message includes the -same sanitized text. Ordinary run retrieval preserves the bounded structured -diagnostic even when multibyte text or other result fields exceed the result -byte budget. In that reduced response, title and details have 1 KiB and 8 KiB -byte budgets, including truncation markers. `retrievalTruncated` directs callers -to the full adapter-bounded text in the run error or transcript. Other provider -metadata and action payloads are not copied. - -When an ACP request fails with a JSON-RPC error, the engine also keeps the -provider explanation. It reads a string `data`, or the string `data.details`, or -the string `data.message`, in that order. It adds the explanation on a new line -after the error message unless the message already contains it. The combined -text passes through the same redaction and the 4,096-character title bound -before it becomes the run error, the summary, and the `acpx.error` `message`. -A numeric JSON-RPC error code appears as the optional integer `rpcCode` in -`errorMeta` and in the `acpx.error` record. Errors without these fields keep -their existing message and omit `rpcCode`. - -Recovery still uses the typed failure category and the adapter's existing -classifier. Provider warnings do not become failures, and timeouts or lost -control channels keep their authoritative failure messages. These diagnostics -stay in the instance's run records and configured run-log storage. They add no -Paperclip Telemetry or OpenTelemetry export. - -## Related instrumentation - -The sandbox duplex transport also writes one run-log event as one of its three -sinks. See the -[Sandbox Duplex Transport Instrumentation](observability.md#sandbox-duplex-transport-instrumentation) -section in the Observability contract. - -## Execution recovery - -Cancelled runs retain `resultJson.cancellation`: a closed `source` label -(`operator`, `queued_message`, `shutdown`, `provider`, `transport`, -`control_plane`, `controller_lease`, or `unknown`), whether the stop was expected, -the initiator, reason, and recording time. Recorded stop intent survives adapter -completion. `controller_lease` means the host stopped its own legacy adapter -after it lost the durable controller lease; the reason names a closed cause -(`renewal_rejected`, `renewal_deadline`, or `renewal_error`). The watchdog -retries a failed renewal until the lease deadline, so `renewal_error` means the -attempts still failed with an error when the deadline passed. -All legacy controllers in one server process renew through a single queue on -the private lease connection pool. Renewals that arrive while one is in flight -share the next statement, so a burst of runs costs one renewal statement per -round instead of one transaction per run, and a renewal waits behind at most -one other. A run whose row another transaction holds is skipped and retried -shortly, without delaying the other runs. A renewal only moves the lease -deadline later, so it does not wake live-run recovery; recovery rereads the -moved deadline when the earlier one comes due. -A lease lost during run setup (for example a rejected renewal while preparing -the workspace) settles the run `failed` with error code `controller_lease_lost`, -the same `controller_lease` cancellation evidence, and -`executionRecovery.providerWorkStarted: false`. Recovery retries it as transient -infrastructure instead of blocking on `setup_failed`. A setup-phase error after -provider dispatch began keeps `setup_failed`, so started work is never retried. -The local lifecycle event includes this evidence. Started cancellations without -an expected stop are also reported to Sentry; its cancellation diagnostics contain -only source, expectedness, initiator type, and the closed lease-loss cause, never -initiator IDs or reason text. -A lease loss also records `resultJson.controllerLeaseLoss`, including a loss -during setup that fails the run before adapter dispatch. It holds the closed -cause and the watchdog timings in milliseconds: `sinceRenewalMs` (since the -last successful renewal), `renewalPendingMs` (age of a renewal operation still -in progress, including its retries and retry waits, or `null`), `deadlineLateMs` -(how late the deadline callback ran, when the lease deadline ended the run, -otherwise `null`), and `renewalCount`. A long `renewalPendingMs` can mean a -stalled database query or repeated failed attempts with retry waits; the cause -label tells them apart (`renewal_deadline` or `renewal_error`). A long -`deadlineLateMs` points at a blocked event loop. Failure reports carry these -bounded numbers and the cause label only. -Historical ambiguous cancellations stay `unknown` and do not authorize replay. - -Provider tool-definition validation failures use -`provider_tool_definition_invalid` / `configuration`. Automatic retry and -continuation recovery stop until the configuration is repaired. Classification -uses raw provider diagnostics in memory before redaction; stored diagnostics -remain redacted and bounded. - -Provider identity diagnostics remain in the local run log. They record the notification method, expected and received thread/turn identifiers, and the classification (root, verified descendant, stale, unrelated informational, or invalid authoritative). They omit the original provider payload and credentials. Repeated informational notices are bounded. - -Ignored unrelated Codex notifications use `harness.diagnostic` with code -`codex_unrelated_information`. The payload retains only the bounded provider -method and expected/received thread and turn identifiers. Account updates, skill -changes, and unrelated thread information do not create a provider notice in -chat. Chat also omits the matching notice stored by older runners. Real provider -warnings and errors remain visible. - -Provider rate-limit windows use `provider.notice.recorded` with category -`provider.rate_limits` and scope `account`. Codex sends them as -`account/rateLimits/updated`. Claude sends them as `usage_update` metadata -(`_claude/rateLimit`) when it uses a subscription sign-in. The runner merges -sparse Codex updates and records a notice only when the windows change. The -notice has one `rateLimits` detail: a JSON snapshot with schema -`paperclip.provider.rate_limits.v1` (windows with used percent and reset time, -credits, and the reached window). Chat does not show the notice. The server -keeps the latest snapshot in the agent's runtime state under -`providerRateLimits`, and the task properties pane shows it as the assignee's -**Usage limit**. The snapshot is display data. It does not hold or start work; -the provider-limit wait reads typed run failures. - -Harness status notices use `provider.notice.recorded` with a status category. -Codex `error` notifications read `error.message`. A notification with -`willRetry: true` becomes a recoverable `provider_retry` notice. A final error -keeps category `error`. The `CodexErrorInfo` kind, the HTTP status, and the -reconnect attempt count become notice details. Claude compaction, model -fallback, and SDK informational banners arrive as ACP message text without a -message identity. The ACPX layer records them as `context_compaction`, -`model_fallback`, or `provider_informational` notices instead of answer text. -Compaction start and end share one notice identity and carry a `Phase` detail. -The task chat shows these notices as inline status chips. A retry chip fades -after later provider work. A final provider error shows the real provider -message. Generic warnings stay out of chat as before. - -Recovery lifecycle events retain the original structured failure code, retry attempt, next retry time, and predecessor/successor identifiers. Durable status delivery uses an idempotency marker; delivery grants no provider authority. Failed publication is retried without repeating provider work. These records are not first-party Telemetry. - -An unstarted legacy conversation retry waiting for execution cleanup retains -`resultJson.executionWait` with the issue, blocking run, and -`execution_owner_active` cause. A local lifecycle event records the wait and the -unchanged scheduled retry attempt once per blocking run. Repeated scheduler -checks update the same retry row without repeating that event. Promotion removes -this ownership-wait marker after cleanup; other execution holds retain their -existing dispatch gates. - -If execution-continuation setup finds that a task no longer exists, is closed, -or its owner changed, the existing cancellation settlement records -`continuation_task_ownership_changed`. The run and wake request become cancelled -before adapter dispatch, with the run-log message -`stale execution continuation cancelled before dispatch`. Immediate recovery is -suppressed. A comment the cancelled run carried that no run read is saved -first when an agent run closed the task: a person's comment, or another agent's -comment posted with `resume: true`. The release then reopens the task and -delivers that comment once. Another agent's comment without `resume: true` -stays communicative and does not reopen the task. Missing source context, authorization failures, and other setup -errors retain their failure classification. An untyped error with the same -message is also still a failure; cancellation requires the typed ownership guard. - -Bounded retry exhaustion writes one lifecycle receipt per run, retry reason, -scheduled attempt, and retry limit. Repeated or concurrent recovery checks reuse -that receipt, including receipts from earlier builds, without advancing the event -sequence or publishing another live event. Attention reads select the latest -matching receipt in PostgreSQL and project only the run's issue/task identifiers -from its context, so historical duplicate receipts cannot multiply run contexts -in server memory. Existing duplicate events do not require deletion or migration. - -### Workspace restore failures - -Legacy adapter results can carry `workspaceRestoreFailure` with the code -`restore_permission_denied`, `restore_lock_timeout`, `restore_unsafe_archive`, -or `restore_failed`. The heartbeat records `workspace_restore_failed` and keeps -the run failed and the workspace-finalization barrier closed. Available output, -usage, session metadata, and the previous execution outcome survive settlement. -`executionBeforeRestore` retains the earlier error code, exit code, signal, and -timeout flag. The ordinary redacted error field retains an earlier error message. - -The chat reports the restore phase separately from a missing final response. -A saved-plan link requires a stored document and its run-bound revision or a -matching run-bound review record. Older unclassified failures use neutral wording. Diagnostics show only -a validated relative member path, never an archive link target or host path. - -An unsafe archive or an outbound confinement refusal keeps the existing execution recovery hold, including across -conversation resets. It cannot start another model turn until an operator uses -the existing recovery action to record `executionReconciliation` with -`workspaceRepairEvidence` (20–12000 characters). This evidence must describe -verified safe staging or repair for the referenced failed run. It does not grant -plan approval. Saved comments, document revisions, and confirmation IDs and -states stay unchanged. Recovery uses the existing delivery identity and links -the successor to the original failed run. Repair does not reset the automatic -retry budget. Transient failures retain the existing -bounded retry policy. Archive confinement remains required. - -Sandbox restore tasks also write a `Workspace restore diagnostic` line to the -run log on failure. `phase` is `workspace` or `asset`, so a failed staged-asset -copy-back (such as credentials) can be distinguished from workspace restoration. The line -contains only an allowlisted OS/transport `errorCode` (otherwise `unknown`), an -optional numeric HTTP error status, and an optional bounded process exit code. -Up to four nested causes are inspected. Messages, URLs, filesystem paths, asset -names, credentials, and response bodies are excluded. Every failed outbound -task emits its own diagnostic; nested repository failures are logged once by -the enclosing workspace task. The original error and restore safety policy are -unchanged. These lines stay in the instance run log and its configured durable -storage, and are not new first-party telemetry events. - -Native sandbox `environmentSyncOut` errors preserve allowlisted error codes and -bounded HTTP/exit statuses across worker RPC for this diagnostic line. The host -revalidates the envelope and keeps the original failure and recovery policy. -Provider messages, paths, response bodies, credentials, and arbitrary error data -are not copied into the new envelope. Older workers can still report `unknown`. - -The optional `step` identifies the failed restore operation. For -`phase=workspace` and `step=git_integration`, `gitCommand` identifies one fixed -command family (`rev_parse`, `symbolic_ref`, `merge_base`, `merge_tree`, -`commit_tree`, `update_ref`, or `log`). `gitFailureKind` is `merge_conflict`, -`invalid_object`, `ref_conflict`, `permission_denied`, or `unknown`; it is a -bounded diagnostic clue, not a new recovery or retry decision. Only supported -exit/OS codes and recognized Git messages produce a specific classification. -No command arguments, stderr, filenames, repository URLs, or ref names are saved. -The same closed fields persist in `workspaceRestoreDiagnostic` and are -revalidated before projection into an enabled Sentry failure report. - -## Codex resume usage snapshot - -The native runner retains a bounded local `harness.diagnostic` event with code -`codex_resume_usage_snapshot`. It identifies `thread/tokenUsage/updated` as -`resume_usage_snapshot`, retains the reported thread and completed-turn IDs, -and records cumulative usage counters. It does not include provider credentials -or message content. The event establishes the accounting baseline; it is not a -new billable usage receipt or a user-facing provider warning. Other provider -identity checks remain in force. - -## OpenRouter serving host - -OpenRouter can route one model across many hosts, which may serve different -quantizations. OpenCode runs routed through OpenRouter record which host served -each model call. The runner's provider proxy reads the top-level `provider` and -`id` fields from the first streamed chunk, or from the whole JSON body when the -response is not streamed. It forwards every byte to OpenCode unchanged and -before it inspects them, so the first token is not delayed. It scans at most -the first 64 KiB of a stream and 1 MiB of a JSON body. It skips encoded bodies, -non-2xx responses, and any malformed or missing field. A recording failure -never affects the response or the turn. - -Each `item.completed` usage event (`payload.kind: "usage"`) carries two fields -in `payload.usage`: - -- `upstreamProvider`: the host that served the most recently finished model - call when the usage was reported, such as `Novita` or `DeepInfra`. -- `providerRequestId`: that call's OpenRouter generation ID, such as - `gen-1762981234-AbCdEfGh`. - -The proxy records a call's host when that call's response ends, not when the -request starts. OpenCode reports a step's usage only a few milliseconds before -it sends the next model request, so a step's usage still names its own call's -host while the next call is in flight. A call whose host the proxy could not -read records no host when it ends, and a session clears the recorded host when -it attaches another run. Such a call therefore carries neither field. It never -inherits an earlier call's or an earlier run's values. - -The runner's internal `usage.reported` event carries the same values, using its -existing `providerRequestId` field and an optional `upstreamProvider` field. -The run's `usageJson.providerRequestId` and its cost event name the first -generation ID the run reported. That value stays the same as later turns -report, so a late usage receipt still matches the charge. Remote Runner runs carry -the same ID in each saved usage receipt's producer event and in the receipt -itself. A receipt saved before this field existed still replays when the -Runner now names an ID. Per-call IDs and -hosts remain in the run log. Runs whose provider reports no host carry neither -field, and `usageJson.providerRequestId` stays `null` as before. - -These are run-log fields. They stay in the instance database and are not sent -to Paperclip Telemetry or OpenTelemetry. - -## AI subscription contention - -A fresh task execution cannot enter this wait. A run that already entered this -wait writes an informational `lifecycle` event to the local run log. Its -payload contains only `retryScheduled`, a boolean that reports -whether the scheduler created a retry. -The message distinguishes an automatic retry from work that is no longer eligible. -This pre-provider wait records `ai_connection_busy` on the cancelled run and does -not consume the provider-failure retry allowance. The event contains no credentials -and creates no Telemetry or OpenTelemetry export. - -## Agent directory waits - -A run whose instruction preparation finds another run of the same agent using -its files writes one informational `lifecycle` event to the local run log, with -`stream: "system"`. It records no file contents, paths or credentials, and -creates no Telemetry or OpenTelemetry export. - -- A resumed native run that stays on its own run writes the message "Another - run of this agent is still using its files; this run continues when it - finishes". Its payload has exactly two fields: `retryReasonCode`, always - `agent_directory_operation_busy`, and `nextAttemptAt`, the ISO time when the - same run is scheduled to try again. -- A run that has not yet started native execution is cancelled with error code - `agent_directory_busy` and writes "Another run of this agent is still using - its files. This task will continue automatically." when an - `agent_directory_busy` retry was scheduled, or "Another run of this agent is - still using its files, and this task could not be rescheduled." when it was - not. Its payload contains only `retryScheduled`, a boolean. - -Neither wait consumes the provider-failure retry allowance. Each stops after 30 -deferrals, after which the run follows the normal failure path. - -## Task history reads - -When a native run reads its task's comments through `get_task_history`, the -server writes one `task_history.read` event (`stream: "system"`, level `info`) -to that run's log. Its payload has one field, `comments`, which maps each -returned comment ID to that comment's `updatedAt` timestamp. It contains no comment bodies or -author identifiers. The event creates no Telemetry or OpenTelemetry export. - -Per-comment delivery (`deliveredQueuedCommentIds`) reads this event together -with the run's `run.result.accepted` event. A read alone is not delivery: the -run may have read the comment and set it aside. A comment counts as received -only when a run of the task's assignee succeeded after reading the comment's -current revision, and that run's accepted result cites the comment as a -`comment:` evidence reference (top-level `evidence[].ref` or a criterion's -`evidenceRefs`). The comment's queued wake then neither reopens the finished -task nor starts another turn. An edit after the read, a read by a run that did -not succeed, or a read the result does not cite leaves the comment owed, so it -is delivered once in a new turn. That turn's wake marks the comment -`readByEarlierTurn` when an earlier finished run read its current revision, and -the prompt asks the agent to check the current task state before acting so -already-applied work is not repeated. `get_task_history` returns -`arrivedDuringThisTurn` (the `comment:` refs of comments that other authors -posted or edited after the run started) with instructions to cite each applied -comment. The check reads only the assignee's succeeded runs of that task that -finished at or after the comment's current revision, through the existing -run-binding and `(company_id, run_id)` event indexes. A saved follow-up -resumed after a remote Stop uses the same rule. `taskHistoryReadsForRun` lists -the comments one run read, with each read's time and the comment revision it -saw, whatever the run's outcome. A failure to write the event leaves the -comment owed in the same way. - -When a run reads a finished child task, the server writes one -`child_task.read` event (`stream: "system"`, level `info`) to that run's log. -The reads that count are `get_task_context` on the parent task, and a -successful agent request with a run token for `GET /api/issues/:id`, -`GET /api/issues/:id/documents` or `GET /api/issues/:id/documents/:key` of a -child in `done`. The route writes the event only after the requested content -has loaded, so a 400 or 404 records nothing. The payload has `parentIssueId` -and `children`, which maps each finished child ID to the revision the run -received: its `status`, `updatedAt` and `statusDecisionId`. It contains no -titles, summaries or document bodies. The event creates no Telemetry or -OpenTelemetry export. - -`hasPendingNativeChildCompletion` reads this event. A child-completion wake no -longer keeps the parent's done claim open when the finishing run received that -child in `done` at the completion the wake reports: the received -`statusDecisionId` is the wake's completion decision, or the received -`updatedAt` is at or after that decision was saved. The event's own write time -does not count, so a delayed write of an older revision cannot deliver a newer -completion. The wake is then cancelled at release instead of starting another -parent turn. A read while the child was still open, a read of an older -revision, a read by another run, or a failure to write the event leaves the -child result owed. - -## Managed Agent File Save Receipts - -The server writes `instruction_save` after managed file collection or a warm -turn checkpoint. The payload includes the save state, instruction entry path, -storage warning, and error code/message. Agent-directory receipts identify -`contract: "agent_files"` and the applied candidate hash. Legacy instruction -receipts instead identify the saved revision. - -A validated warm checkpoint reports `saved` or `unchanged`, even though its -working directory remains owned by the live session. An unstable checkpoint -reports `pending_collection` until stopped collection produces a final receipt. -Successful checkpoints can include `checkpointStats`: `scannedEntries`, -`hashedBytes`, `copiedFiles`, and `copiedBytes`. These counts describe that -capture, not cumulative traffic or an atomic snapshot of background writers. -They contain no file contents. The receipt remains in the instance run log; -it adds no Paperclip Telemetry or OpenTelemetry export. - -If instruction-copy release throws, `instruction_cleanup` records a warning -with payload `{ "state": "deferred" }`. The existing working-copy recovery sweep -retries cleanup. This event preserves the run outcome and does not claim a file -save; `instruction_save` remains authoritative for collection. The event contains -no raw exception, host path, file contents, or lock-owner metadata. Failure to -write the warning must not replace the provider outcome or stop lease release. - -## Agent observability availability - -`agent_observability.unavailable` is one warning for a preparation snapshot that -cannot apply company export settings. Its payload contains only a closed reason -and the settings revision. It carries no endpoint, credential, secret reference -or resolver error. Recording the diagnostic never contacts a destination. - -## Runtime activity - -`runtime.activity.recorded` is a P1 Runner PRP event with a closed -`{ namespace, activityKey }` payload. `namespace` groups a producer's activity -types under a stable name. `activityKey` identifies an activity within that group. -Consumers use the pair to select a response to activity in the current run. - -For example, `{ namespace: "example.computer", activityKey: "desktop.used" }` -records successful desktop use. Core can map that activity to a Desktop viewer -for the task's running agent. The authenticated event envelope supplies the run, -session, source sequence and emission time. Core resolves the company, task and -agent through the run's existing authorization checks. - -Both activity identifiers contain 1–160 Unicode code points and exclude C0 -controls and DEL. Callers cannot supply run, task or agent identity in this payload. - -The ordinary authenticated Runner event is the durable record. Normal source-event -acknowledgement and replay deduplicate delivery. Existing run-read authorization -and secret redaction apply to saved and live events. Recording reported activity -is not independent proof that it happened and grants no resource permission. - -The native transport forwards runtime activity as a run-scoped fact. It retains -the authenticated run identity across warm session attachment. Provider messages -cannot impersonate this transport-owned notification. -New-run activity waits for warm attachment to finish. A failed attachment does -not retarget queued activity to the previous run. - -Runner accepts local activity records only with an opaque context -captured before the operation. Warm run attachment and process recovery rotate -that authority. Runner supplies the run identity from its active authenticated -binding. Optional hint failure does not fail the underlying operation. No output -text is scanned and no synthetic receipt is written. - -`GET /api/heartbeat-runs/:runId/events?view=plugin-panels` reads these events and -returns only installed agent task-panel contributions under normal run-read and -telemetry-read authorization. Ordinary PRP reads/live delivery retain run-secret -and current-user redaction. The task page uses one history read on mount/reconnect -and the existing live transcript stream, then makes a bounded plugin readiness -check. No panel discovery timer scans runs or provider state while connected. - -## Post-result failure diagnostics - -`native.post_result_failure` records `stage: post_result_execution`, a closed -`category` (`io_not_found`, `io_permission_denied`, `io_failure`, or -`execution_failure`), and `resultAlreadyAccepted: true`. It requires an accepted -selected result and exact current, unexpired controller authority. A busy diagnostic -row lock is skipped. Reporting is best effort under the existing 60-second control -bound; the sink can finish later and emit a late event. The `native_session_lease_lost` -control classification stays unchanged. The labels grant no recovery, result, -Stop, provider-work or payment authority and do not establish a hosted failure cause. +Run-log events are rows in the `heartbeat_run_events` table +(`packages/db/src/schema/heartbeat_run_events.ts`). They stay in the instance +database. They are not Paperclip Telemetry events and not OpenTelemetry +exports, and they need no operator endpoint. See `rules/data-paths.mdc` for how +the three data paths differ. + +This file does not list events or their payloads. Each event is defined where +it is written: search for the event name as a quoted string under `server/src` +and `packages` to find its producer and payload. Payload limits are defined in +`server/src/services/heartbeat/run-log.ts`; the storage codec lives in +`packages/db/src/run-event-payload.ts`. The server appends events through +`appendRunEvent` in `server/src/services/heartbeat.ts`. ## Payload storage -Structured events in this guide remain in `heartbeat_run_events`. When the -instance's existing file storage uses S3, new stdout/stderr NDJSON logs use the -same stack bucket and configured prefix under `run-logs/`, with existing run -references in Postgres. Local-disk instances remain local. Raw provider -traces have their own restricted, encrypted object path. See +Structured events remain in `heartbeat_run_events`. When the instance's existing +file storage uses S3, new stdout/stderr NDJSON logs use the same stack bucket +and configured prefix under `run-logs/`, with existing run references in +Postgres. Local-disk instances remain local. Raw provider traces have their own +restricted, encrypted object path. See [run evidence storage](run-evidence-storage.md) for batching, durability, configuration, migration, retention, and qualification boundaries. - -### Event commit stall diagnostics - -A Runner event commit stall records the first safe error code in the error and -up to three nested `cause` objects. This retains database and transport codes -when an adapter wraps the original error. Each code has the existing 80-character -identifier limit. If no safe code exists, the diagnostic uses the outer error's -safe name. It never copies exception messages, SQL, or query arguments. Cyclic or -long cause chains stop at the same four-object limit. - -This only improves the recorded cause. Event acknowledgement, replay integrity, -the bounded stall policy, and operator recovery requirements stay unchanged. - -A later successful continuation does not establish the cause of an earlier -failure that recorded only `Error`. - -### Failure cause of generic run failures - -A run that fails with a generic code (`adapter_failed`, `setup_failed`) keeps -that code, because callers compare against it, and records -`resultJson.failureCause`: one identifier of at most 80 characters -(`[A-Za-z0-9_.:-]`). For a thrown error it is the provider's HTTP status -(`provider_http_429`) or category, else the first safe code in the error and up -to three nested `cause` objects (`40001`, `ECONNRESET`, the same extraction as -the Runner's event commit stall diagnostics), else the error's class name. For -an adapter result without a code it is the error family, `signal_`, -`process_cancellation_failed`, `exit_code_` or `adapter_reported_error`. It -never copies message text. The server's failure log line carries the same -cause, and the board's run list shows it, for example "adapter failed (40001)". -Older rows have no cause and read as before. - -Run error text keeps SQL statements off the board: a Drizzle `Failed query:` -statement and its parameters, or unwrapped statement text with `$n` -placeholders, is replaced with "Database query failed ()". The wrapper -text before the statement is kept. - - -## Chat outcome read receipts - -Two run-log events record what the conversation agent saw before it replied. -`chat_handoff.completed_read` records that a run read a task it handed off, in -the done state, with the task ID and status version. `chat_interaction.declined_read` -records that a run listed its conversation's interactions and saw a declined tool -action, with the issue ID, interaction ID and status. Neither carries task text, -tool arguments or results. - -A completion or decline continuation that an earlier reply already reported -creates no run. The server checks for a reply by the same agent in the same -conversation, posted after the outcome, from a run that has one of these receipts -for that exact outcome. A completion reply must also link the task. The -completion delivery is then marked delivered with that reply as its response, -and a deferred wake ends `skipped` with the reason "Already reported by an -earlier reply in this conversation". -The conversation's handoff listing (`GET /api/issues/{id}/chat-handoffs`) shows -that reply as `completionReportCommentId` once the latest completion is delivered. diff --git a/doc/runner-api-tools.md b/doc/runner-api-tools.md index a4dc4081fc2..9ee52c8683f 100644 --- a/doc/runner-api-tools.md +++ b/doc/runner-api-tools.md @@ -343,7 +343,7 @@ idempotent redaction projection. Repeated projection preserves the original payload digest. Provider recovery uses the same projection to compare saved completion results with provider history. Changed private bodies still conflict. Restart preparation preserves the run's admitted REST credential scope, even -when the task mode changes. See [the run-log guide](run-log-events.md) for stored fields. +when the task mode changes. The stored fields are defined where the run-log event is written. The server-issued REST credential for the bound run is never returned to the model. Streaming, WebSocket, MCP and authentication diff --git a/maps/agent-access-map/README.md b/maps/agent-access-map/README.md index e5d633077ff..2fc8f2e87f3 100644 --- a/maps/agent-access-map/README.md +++ b/maps/agent-access-map/README.md @@ -79,7 +79,7 @@ receipt and inline response policy as newly created webhook secrets. Task image access requires the company boundary and the task permission; skill-test and low-trust agents do not need company-wide browsing authority. Completion replay applies the same idempotent projection to stored and expected -payloads; [the run-log guide](../../doc/run-log-events.md) describes its fields. +payloads; its fields are defined where the [run-log event](../../doc/run-log-events.md) is written. Sources: [API client](../../server/src/services/native-runtime/runner-api-client.ts), [catalog](../../server/src/services/native-runtime/runner-api-catalog.ts), diff --git a/maps/agent-access-map/runs-adapters.md b/maps/agent-access-map/runs-adapters.md index 8fc0e9e281c..6a7023064bd 100644 --- a/maps/agent-access-map/runs-adapters.md +++ b/maps/agent-access-map/runs-adapters.md @@ -287,7 +287,8 @@ The native first-text span ends after text is committed for the existing run-eve reader, including reasoning and completed responses. It exports only the bounded text kind and timing, with no text content or new agent access. This adds no agent API, credential access, run-log write, or recovery authority. -See [the tracing contract](../../doc/observability.md#controller-preparation-phases). +See the spans in +[`execution-preparation-timing.ts`](../../server/src/services/execution-preparation-timing.ts). Live latency verification remains separate from the existing access baseline. API: `present` β€” representative operation IDs: diff --git a/maps/feature-map/chat-channels.md b/maps/feature-map/chat-channels.md index 679eb9265e7..dfb8e215947 100644 --- a/maps/feature-map/chat-channels.md +++ b/maps/feature-map/chat-channels.md @@ -110,7 +110,8 @@ Eyes should appear after authorized task/comment persistence, independently of file preparation, subscription and agent startup. Duplicate delivery must keep one receipt; the final reply must remove only its source message's eyes. Record live mention-to-reaction timing separately from the PostgreSQL fixture tests and -the `chat.slack.receipt` transport span described in [Observability](../../doc/observability.md). +the `chat.slack.receipt` transport span in +[`receipt-timing.ts`](../../server/src/services/chat-channels/slack/receipt-timing.ts). During a top-level DM run, verify that the bot adds eyes to the source message and removes them when the final reply arrives. The pinned Slack SDK encodes a diff --git a/maps/feature-map/runs-adapters.md b/maps/feature-map/runs-adapters.md index 014a56832a1..a03ce83c1d8 100644 --- a/maps/feature-map/runs-adapters.md +++ b/maps/feature-map/runs-adapters.md @@ -44,7 +44,8 @@ For latency diagnosis with an operator OTLP endpoint, search the hashed run ID without restricting the root span name. Compare workspace preparation with its `workspace_` intervals. Measure controller readiness separately through `controller.startup.`. A recovery pause or deployment drain is not model -execution time. See the [tracing contract](../../doc/observability.md#controller-preparation-phases). +execution time. See the spans in +[`execution-preparation-timing.ts`](../../server/src/services/execution-preparation-timing.ts). These source-level diagnostics do not establish a live response-time budget. `task.time_to_first_token` measures accepted ingress through the first committed text that the task transcript can show, including thinking and completed diff --git a/packages/adapter-utils/src/http2-bridge-server.ts b/packages/adapter-utils/src/http2-bridge-server.ts index 3d96787ff25..4094292f0aa 100644 --- a/packages/adapter-utils/src/http2-bridge-server.ts +++ b/packages/adapter-utils/src/http2-bridge-server.ts @@ -132,9 +132,8 @@ export const HTTP2_BRIDGE_SERVER_OPTIONS: http2.ServerOptions = { * total, so no combination of concurrent streams, across every route, can * retain more than this many bytes at once. This value keeps the accepted * process ceiling at 1,073,741,824 bytes (1 GiB) β€” the same ceiling - * `doc/observability.md` already accepted before the per-body limit rose to - * 10 MiB β€” now enforced by this reservation instead of left as an unenforced - * document note. See the per-route budget comment above for the full + * the project already accepted before the per-body limit rose to 10 MiB β€” + * now enforced by this reservation instead of left as an unenforced note. See the per-route budget comment above for the full * accounting. */ export const HTTP2_BRIDGE_MAX_PROCESS_BODY_BYTES = 1024 * 1024 * 1024; diff --git a/packages/shared/src/telemetry/README.md b/packages/shared/src/telemetry/README.md index e792827b9f1..28125b5ab54 100644 --- a/packages/shared/src/telemetry/README.md +++ b/packages/shared/src/telemetry/README.md @@ -126,9 +126,8 @@ This document covers Paperclip Telemetry only. The generated Telemetry contract covers neither the Observability path nor the run-log path. Two other data paths document their own contract in their own file: -- [Observability](../../../../doc/observability.md) β€” the OpenTelemetry trace - path, the sandbox startup trace spans, and the sandbox duplex transport - instrumentation. +- [Observability](../../../../doc/observability.md) β€” the operator-enabled + OpenTelemetry trace path and Sentry error monitoring. - [Run-Log Events](../../../../doc/run-log-events.md) β€” events written to the local `heartbeat_run_events` table. diff --git a/rules/data-paths.mdc b/rules/data-paths.mdc index 0d3f28ca3c6..202c3732a51 100644 --- a/rules/data-paths.mdc +++ b/rules/data-paths.mdc @@ -49,10 +49,10 @@ Apply a review level that matches the path: - **Observability change (lighter review).** The operator endpoint gate stays in place. The no-operation behaviour stays when no endpoint is set. A privacy review is not necessary while the change stays inside the closed - span-attribute allowlist. + span-attribute allowlist defined in the code that opens the span. - **Run-log change (no extra review).** A run-log change needs neither review - level above, because the data stays in the instance database. Update - `doc/run-log-events.md` when an event is added or changed. + level above, because the data stays in the instance database. Define the + event and its payload where it is written; do not list it in a doc. ## Exclusion diff --git a/tests/runner-e2e/SUITES.md b/tests/runner-e2e/SUITES.md index f0b94e468f4..e1ec469dd01 100644 --- a/tests/runner-e2e/SUITES.md +++ b/tests/runner-e2e/SUITES.md @@ -1861,7 +1861,7 @@ Results can include a `startupReuse` entry for each run. The dashboard shows receipt-derived warmth, the environment/lease, workspace, Runner, provider process and session decisions, and observed provider launches. Private attempt evidence includes `snapshots/startup-reuse.json`. See the -[run-log contract](../../doc/run-log-events.md#startup-reuse-evidence) for the +[startup reuse receipt](../../packages/adapter-utils/src/startup-reuse.ts) for the closed states and reasons. Historical runs without receipts show unknown values. Provider-session continuity does not establish physical process reuse. diff --git a/tests/runner-e2e/source-sdk-public-inputs.json b/tests/runner-e2e/source-sdk-public-inputs.json index 980f64e64ff..a34d5f70074 100644 --- a/tests/runner-e2e/source-sdk-public-inputs.json +++ b/tests/runner-e2e/source-sdk-public-inputs.json @@ -351,7 +351,7 @@ "packages/shared/src/slack-tools.ts": "c53a75cd3099c6d8dcd1475e755aae589d4c88f6", "packages/shared/src/subscriptions.ts": "ef70e63395ca967bbc30776b71560e96eb354a75", "packages/shared/src/summary-slot.test.ts": "5f8e222dea425f0d5bff0adb5c683087bd7cf7fb", - "packages/shared/src/telemetry/README.md": "e792827b9f1e629ccdb66eecaad014cd02e532e8", + "packages/shared/src/telemetry/README.md": "28125b5ab542abc46c4bb6c044636a093f13584f", "packages/shared/src/telemetry/client-types.test.ts": "3b630edda4972b929ba4decd7457f226eb9cfc5b", "packages/shared/src/telemetry/client.test.ts": "3ba900cdab5a1ac1dcc1dcea382bb790af32d895", "packages/shared/src/telemetry/client.ts": "03a10a5550b542487543fbfd4a417e6eb316e11e",