mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 16:29:42 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Company traces let operators compare each request with its reply. > - OpenCode limits each text field to 8 KiB. > - A fresh session can put old history before the current request. > - The limit can remove the current request from the trace. > - This change selects the current task text first and links the retained submission. ## Linked Issues or Issue Description Refs #18098 and #18699. Neither merged change selects the current task request. **What existing behavior does this improve?** OpenCode company traces can contain only part of the old session history. **Current behavior** The exporter clips the full model envelope from its start. A sampled trace has a useful HOLD reply but a request that ends inside old history. A truncation flag is not proof that the task failed. **Proposed behavior** Select the task prompt before the text limit. Remove a valid leading generated handoff packet from the excerpt. Link the full retained run event through the existing API. [Source pattern](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/e6d8da04-ca3e-4bd9-bd90-737f7717a9e1). ### Hypothesis and thesis for a fix The baseline exporter spends its text budget on the envelope and old history. The new regression confirms this cause: the received OTLP input loses a short current request after a long handoff. Select the current request in the OpenCode projection before redaction and clipping. Keep storage and access checks with their existing owners. ## What Changed - Select task identity and prompt for known native envelopes. Put pending user messages first in continuation excerpts. - Omit only a valid leading generated handoff packet. Keep plain text and unknown formats on the existing path. - Add a relative run-events reference under the prompt capture switch. Readers must sign in to the source instance and page to the matching turn. - Keep raw tool argument values excluded. Keep redaction, opt-in, capture switches, company access and size limits. - Add exporter, proxy, installed-runtime and private-run retrieval tests. Fix the proxy test's session-directory lookup so it skips the output-custody directory. - Document the excerpt and full-event retrieval path. ## Verification - Baseline `6b6c94a4ea8f3d1b58767d67a7a2fc4ceaecda9c`: the new long-handoff regression failed because OTLP input did not contain the current request. - Same regression passes after the fix. Exporter, redaction and bundled proxy suites: 39 tests passed. - `PAPERCLIP_OPENCODE_QUALIFY=1 pnpm --dir packages/paperclip-runner exec vitest run src/drivers/opencode/company-observability-runtime.test.ts`: 2 passed. Real installed OpenCode 1.18.34 and real HTTP OTLP receiver; local SSE model fixture. Both capture settings pass. The model receives the full original submission. - Private-run API suite: 31 tests passed on base `340157d1d6ec01ad0fa55970e148c6cd332bbcb1` with these changes. Authorized readers retrieve a complete request over 8 KiB through pagination. Other readers receive 404. - Runner TypeScript source and test checks and TypeScript build passed. `git diff --check` passed. - `pnpm verify --explain` returned exit 2 and required broader CI. No full local-suite pass is claimed. - Initial harness attempts used the wrong Vitest working directory, lacked built plugin-sdk, selected the wrong proxy fixture directory, or expected 403 instead of the API's 404. These are not red regression results. The corrected checks above pass. - Review follow-up `dfe995a77f3eb3280a6b081b4df9428fac85f540` removes added code comments required by CONTRIBUTING.md. Exporter + installed OpenCode runtime suites reran: 30 tests passed, with the local SSE model and real HTTP OTLP receiver. `git diff --check` passed. The review thread is resolved. Greptile passed on this exact revision with confidence 5/5 and no actionable findings. Final-head typecheck, build, Runner and E2E checks passed; server test shards and the amd64 Runner image build remain in progress. No hosted model call or deployed recovery is claimed. ## Risks Known envelope excerpts change shape from serialized envelope text to readable task text. The handoff selector depends on the current generated packet format; unknown framing falls back to the full prompt excerpt. Long current requests still clip. Full-event access depends on source-instance permission and retention. Export remains best effort. The new reference grants no access and includes no credential. This PR is unmerged and not deployed. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use and code execution. The exact serving variant and context size are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Final verification (2026-11-14): exact head `dfe995a77f3eb3280a6b081b4df9428fac85f540` has 60 successful checks, one neutral security scan, and five conditional skips (cache write, publication/attestation, incremental selection, Storybook paths). Applicable typecheck, build, general and serialized tests, Runner tests, E2E and both Runner image architectures passed. [CI](https://github.com/paperclipai/paperclip/actions/runs/43572758306), [Runner images](https://github.com/paperclipai/paperclip/actions/runs/43572758233), [Greptile](https://github.com/paperclipai/paperclip/runs/132380487679). Greptile 5/5, no unresolved actionable findings. Unmerged and not deployed. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
159 lines
10 KiB
Markdown
159 lines
10 KiB
Markdown
# Company agent observability
|
|
|
|
Company Settings → General → Agent observability configures one company-owned
|
|
OTLP destination for Paperclip-managed Codex, Claude Code and native OpenCode executions. Export
|
|
is disabled by default. Choose Braintrust (US) or a custom HTTP(S) OTLP base URL;
|
|
use your own collector when you need fan-out. Braintrust EU and self-hosted
|
|
installations can use Custom OTLP with their required routing/authentication
|
|
headers. The receiver must be reachable from every execution environment;
|
|
`localhost` refers to the execution host, including a remote sandbox.
|
|
|
|
This is separate from Paperclip's first-party Telemetry and its content-free
|
|
[infrastructure tracing](observability.md). Configuring it does not change either data path.
|
|
|
|
## Capture and supported runtimes
|
|
|
|
The qualified versions are Codex 0.160.0 and Claude Code through ACP bridge
|
|
0.73.0 with Agent SDK 0.3.286. Both logs and traces use HTTP/protobuf. Prompts,
|
|
assistant responses and tool content default on. Additional tool details are a
|
|
Claude Code and native OpenCode control; Codex has no equivalent switch. Codex requests up to 64 KiB
|
|
of tool output (or zero when disabled); native event availability, truncation
|
|
and response caps still apply. Claude Code applies its own native limits.
|
|
Metrics, raw API bodies and Codex Guardian assessments are disabled.
|
|
|
|
Native OpenCode 1.18.34 uses a runner-owned HTTP/JSON exporter. Each completed,
|
|
failed or cancelled turn produces a trace with Paperclip company, agent, run and
|
|
turn IDs. Authenticated Paperclip tool dispatch outcomes, native tool outcomes and
|
|
message-level token/cost receipts are child spans;
|
|
repeated usage updates replace the same message receipt. Missing prices remain
|
|
unknown. This projection excludes hidden reasoning, raw provider payloads and
|
|
system instructions. Prompt, response, tool-detail and tool-content switches
|
|
apply before export. Content is bounded to 8 KiB per field and 64 tools plus 64
|
|
usage records per turn; truncation is marked. With both tool-detail and tool-content
|
|
capture enabled, Paperclip tool spans include `braintrust.input` diagnostics
|
|
(`paperclip.opencode.tool-diagnostics.v1`): the upstream `tools/call` method,
|
|
argument JSON type and types for a fixed list of envelope fields. These are
|
|
shape diagnostics, not a copy of the arguments. Arbitrary field names, argument
|
|
values, nested payloads and full model request bodies are not captured.
|
|
Authenticated schema rejections also produce failed tool spans without invoking
|
|
the controller handler. Conflicting request IDs produce separate failed items,
|
|
so a rejection cannot overwrite a previous dispatch outcome.
|
|
Paperclip tool spans use authenticated dispatch IDs and the name `tool <name>`.
|
|
Provider SSE records for those declared tools use `opencode tool <name>` and
|
|
remain available when failure occurs before dispatch, such as an oversized
|
|
request. These are separate observations, not correlated calls. The
|
|
`paperclip.tool.source` attribute distinguishes dispatch and provider records.
|
|
Native tools retain their provider spans without submitted-argument diagnostics. OpenCode's performance-only OTLP sink stays separate.
|
|
|
|
For known native task envelopes, OpenCode selects the task identifier, title and
|
|
prompt before the 8 KiB limit. A valid leading fresh-session handoff packet is
|
|
omitted from this excerpt so earlier history cannot hide the current request.
|
|
Continuation excerpts put pending user messages before recovery guidance.
|
|
Plain text and unknown envelope formats retain their existing text projection.
|
|
History omission also sets the turn's truncation flag; the flag can also mean a
|
|
text or row limit was reached.
|
|
|
|
When prompt capture is enabled, `paperclip.observability.input.events` gives a
|
|
relative URL on the originating Paperclip instance:
|
|
`/api/heartbeat-runs/<run-id>/events`. Sign in to that instance, read pages using
|
|
`afterSeq` and `limit`, and find the `turn.submitted` event for the trace's
|
|
`paperclip.turn.id`. Its `payload.prpEvent.payload.text` holds the retained
|
|
submission, including history, subject to existing redaction and retention.
|
|
The URL does not grant access. Run telemetry permission, company membership and
|
|
task privacy checks still apply. The reference is absent when prompt capture is
|
|
disabled. Tool argument values remain excluded from company traces.
|
|
|
|
The exporter sends once at turn completion with a three-second deadline, at most
|
|
four requests in flight and no idle polling or persistent queue. Closing drains
|
|
pending requests and marks an unfinished turn failed. Export failures emit a
|
|
sanitized diagnostic and do not change task outcomes. Abrupt process termination
|
|
can lose an unexported turn. Recovered in-flight turns without a new submission
|
|
are not replayed just to produce an observability record.
|
|
|
|
The native runner and legacy local adapters receive transient launch material.
|
|
Native Codex writes a private managed config.toml; legacy exec uses a unique
|
|
invocation profile, preserving a selected source profile. Its profile injection
|
|
requires the qualified executable version. Codex ACPX writes its isolated home.
|
|
The legacy ACPX adapter keeps export configuration in a private home per agent
|
|
and settings compatibility key, and shares conversation files with its stable managed home;
|
|
export changes do not change its logical conversation fingerprint.
|
|
Codex receives authentication through `OTEL_EXPORTER_OTLP_LOGS_HEADERS` and
|
|
`OTEL_EXPORTER_OTLP_TRACES_HEADERS` in its process environment. Its managed
|
|
configuration and invocation profiles contain empty header maps, so exporting
|
|
does not save the ingestion credential in those files. The bundled Rust OTLP
|
|
exporter percent-decodes environment header values; launch material encodes
|
|
them once to preserve literal percent sequences. Credentials remain literal
|
|
values in the company secret; config-file environment interpolation is not used.
|
|
A caller-owned/preconfigured Codex home cannot be automatically configured in
|
|
this version: its files remain untouched and execution emits a diagnostic.
|
|
Legacy OpenCode, other harnesses, personal Codex desktop sessions and hosted claude_managed are
|
|
outside this feature.
|
|
|
|
## Changes and process lifetime
|
|
|
|
Paperclip reads settings once when an execution begins preparation. That
|
|
snapshot governs instruction reservation, compatibility and launch. Already
|
|
preparing or running work retains its configuration. A changed settings
|
|
revision or resolved credential version retires an idle incompatible warm
|
|
process using existing ownership checks. Conversation compatibility is
|
|
separate, so a safe replacement can resume the saved conversation. Recovery
|
|
never replays an uncertain turn merely to apply new export settings.
|
|
|
|
Missing, disabled, invalid or unavailable configuration runs without company
|
|
export and reports a sanitized availability diagnostic where relevant. Native
|
|
export rejection/timeouts do not determine task success; exporter deadlines
|
|
are three seconds. OpenCode starts its bounded asynchronous request when the terminal event is
|
|
observed; event delivery and task settlement do not wait for the receiver.
|
|
|
|
## Credentials and access
|
|
|
|
Only board users with Manage connections permission can read or change this
|
|
configuration. Agents cannot retrieve it through the settings API. The destination stores no credential
|
|
values. Authentication headers live in one encrypted Paperclip-managed company
|
|
secret, bound to the system consumer company-observability. Updates rotate that
|
|
secret; removal uses the secret service to retire it. Braintrust project
|
|
routing is derived from the current project ID, independently of the API key.
|
|
|
|
Reads return configured status and header names. Blank existing custom header
|
|
values retain the saved secret at the same HTTP origin; a change of scheme, host,
|
|
or port requires every retained header value to be entered again. New/renamed headers require values. Removing a
|
|
row removes that header. API-key replacement is explicit, and Cancel clears
|
|
entered secrets. Revisions prevent another administrator's edit being silently
|
|
overwritten. Credential values are excluded from task environment manifests,
|
|
run/config snapshots, argv, activity records, recovery configuration and query
|
|
cache data. OpenCode keeps exporter credentials in the runner wrapper and excludes them
|
|
from its child environment and config. Codex and Claude Code receive headers through their
|
|
process environment, which their tools can inherit or inspect.
|
|
This feature does not isolate export credentials from agent code. Use a credential
|
|
limited to telemetry ingestion for the selected destination; do not supply a broad
|
|
organization administration key. A user-operated collector can hold upstream
|
|
credentials when a deployment requires a separate trust boundary.
|
|
|
|
## Connection tests
|
|
|
|
Saving never contacts the destination. Test saved settings sends one synthetic
|
|
root span and a correlated log with a unique marker from the control plane,
|
|
even when agent export is disabled. The receiver receives no production task
|
|
content. Both requests share a ten-second deadline, cannot redirect, and use
|
|
the deployment's guarded HTTP policy with DNS pinning and response-size limits.
|
|
Receipts contain sanitized HTTP/signal results and no receiver body text.
|
|
A receipt is retained only for the revision that was tested.
|
|
|
|
“Test payload accepted” means the server's synthetic payload was accepted. It
|
|
does not prove agent export, remote network reachability, downstream indexing,
|
|
or that a provider UI rendered the records. Harness exporters use their own
|
|
network clients under the execution environment's policy; the server's DNS
|
|
pinning applies only to the explicit control-plane test. Verify real tasks and
|
|
the destination's records before relying on a deployment.
|
|
|
|
## Verification
|
|
|
|
The focused suites cover schema bounds, concurrent saves, secret/header
|
|
lifecycle, authorization, protobuf transport and rejection, private-network
|
|
policy, timeout, revision fencing, environment projections, private native
|
|
configuration, warm compatibility, and UI conflict/company-switch behavior.
|
|
Storybook IDs agent-observability-01-settings, agent-observability-02-states and
|
|
agent-observability-03-journey use fake credentials and production components.
|
|
Real local/remote native export evidence must be reported separately from
|
|
these automated and synthetic checks.
|