Files
PaperClipAI/doc/agent-observability.md
T
DottaandPaperclip ac9459bae3 feat: configure company agent observability (#16874)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Companies use several agent harnesses and execution environments.
> - Operators need one place to select where native agent logs and
traces go.
> - An inference proxy would add a dependency to the agent request path.
> - This pull request supplies native OpenTelemetry settings to managed
Codex and Claude Code processes.
> - Company secrets protect destination credentials. Export failure does
not change the task result.

## Linked Issues or Issue Description

**Subsystem affected**

Company settings, shared contracts, database, server, adapters, and
runner.

**Problem or motivation**

An operator cannot configure a company destination for native agent logs
and traces. Manual harness configuration is hard to apply across local
and remote execution.

**Proposed solution**

Add one company destination: Braintrust or a custom OTLP HTTP/protobuf
endpoint. Apply a transient snapshot when execution preparation starts.
Keep credentials out of API reads, task environment manifests, recovery
files, and command arguments. Replace a warm process when its export
settings change and preserve its conversation.

**Alternatives considered**

Multiple destinations can use a user-operated collector for fan-out.
This change does not add a collector, an inference proxy, or transcript
scraping.

**Roadmap alignment**

This extends company settings, secret management, and support for
multiple harnesses. The maintainer requested this implementation.
Searches for observability and Braintrust found no duplicate company
destination feature. Related infrastructure tracing work in #16236 has a
separate content-free contract.

## What Changed

- Add strict schemas, revision checks, encrypted secret storage,
activity records, and board-only settings routes.
- Add explicit synthetic logs and traces tests with a bounded HTTP
request and sanitized results.
- Supply native Codex configuration and Claude Code environment values
through local and remote launch paths.
- Preserve Codex login refresh through the existing managed credential
link. Retain unsaved drafts across background reads. Require complete
custom header re-entry when the endpoint origin changes.
- Separate process export compatibility from conversation identity.
Isolate generated Codex configuration and preserve resumable sessions,
including remote home refresh after export changes.
- Advance the immutable Pi/Copilot source attestations for shared
environment and sidecar code changes. Keep their provider payloads
unchanged and retain prior identities for decoding.
- Redact destination credentials throughout launch diagnostics and HTTP
logs.
- Add the production settings form, Storybook states and journey,
documentation, and feature/access map entries.

## Verification

- Real managed local Codex 0.160.0 and Claude Code 2.1.286 tasks
exported to Braintrust. API readback and the Braintrust UI showed the
test records. A real remote Daytona Codex task exported through the
legacy `exec` path.
- Live tests exercised content capture off, export disabled, warm
conversation reuse, and a destination that accepted connections but
never replied. Agent work succeeded and export requests stopped at the
configured three-second timeout. The UI connection test returned “Test
payload accepted”; the saved API key stayed masked after reload.
- All current-head CI test lanes, typechecks, packaging/build checks,
and required aggregates passed on
`71e994b2be9c8fb6d6e938a4ad7cf4c95fdafe31` ([CI
run](https://github.com/paperclipai/paperclip/actions/runs/39319194318)).
Greptile completed on that exact commit at 5/5, with zero unresolved
review threads. The additional optional Paperclip Review check remains
queued waiting for its assigned agent.
- Local workspace typecheck and build passed. Focused verification
included 239 combined runner tests, 789 API/native-session tests, 107
Codex home/ACP/sandbox-site tests, 204 feature/privacy tests, and the
affected Rust suites. Migration, token, map, Storybook catalog, and
Storybook build checks passed during acceptance. These checks ran at the
relevant implementation or merge boundary; the current-head CI is the
complete final gate.
- Managed AI credential projections now receive enabled and disabled
company export settings; genuine caller-owned Codex homes remain
untouched. Two regressions reproduced the skip before the repair. All
276 engine/lifecycle tests and adapter-utils typecheck pass afterward.
The launched TOML retains provider/MCP settings, credential continuity,
and redaction.
- Settings saves, removal, and the real company-deletion owner acquire
secret mutation ownership before database locks. PostgreSQL NOWAIT
probes cover all three paths; concurrent save/rotation/removal/deletion
requests settle correctly. The affected suites passed 137 tests,
followed by all 32 service/connection cases after expanded deletion
coverage. Server and Codex adapter typechecks pass.
- Codex source profiles use the effective CLI argument list. A spawned
fixture CLI reproduced both ignored-argument failures; all three
precedence/fallback cases now pass.
- Real tar staging reproduced remote conversation loss in both API-key
and subscription modes before the repair. Both now pass: changing
managed settings preserves sessions, archives, native indexes, and
refreshed subscription credentials. A corrupt upload preserves the prior
config and conversation.
- Preserved failure evidence: initial CI caught stale Pi/Copilot source
attestations, a missing test-harness helper, and packaging fixture/probe
mismatches; those were corrected and covered by focused checks.
Superseded heads had intermittent attachment-composer timeouts despite
successful uploads; [the original trace and screenshot remain
available](https://github.com/paperclipai/paperclip/actions/runs/39309701089/artifacts/12215109203).
The same attachment case passed on the final head: [25/25 tests in shard
3](https://github.com/paperclipai/paperclip/actions/runs/39319194318/job/118288637051).
- A connection-fault fixture also exposed a stale pooled socket after
its hang case; closing fixture responses isolated scenarios and the full
affected suite passed.
- The broad local test run was not clean: database resource failures,
timeouts, an older cached module graph, and a stale optional-resource
manifest assumption were recorded. Focused checks passed after the
documented fixes. This does not establish a clean full local pass.
- Native remote Codex, remote Claude, and the full ACPX Codex bridge
remain unqualified. The direct Codex resume test does not establish full
bridge acceptance.

## Risks

- Native exporters perform network requests from each execution
environment. A server connection test does not prove remote
reachability.
- Prompts, assistant responses, and tool content are enabled by default
when supported. Native caps and harness-specific controls still apply.
- Caller-owned Codex homes are unsupported for automatic configuration
and remain untouched.
- Native harnesses and their tool processes can inspect export
credentials. The settings form and documentation advise using
ingestion-scoped credentials. Retained headers remain bound to their
saved scheme, host, and port; changing origin requires fresh values or
removal.
- Migration 0357 adds a company-scoped row and secret reference. It is
safe to reapply. Secret removal and company deletion use the existing
secret lifecycle.
- Slow or unavailable destinations must not block task settlement.
Export settings take effect at the next execution preparation boundary.

## Model Used

OpenAI Codex, GPT-6, with reasoning, tool use, code execution, and
browser verification. The exact serving model ID and context window size
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-19 15:36:04 -05:00

106 lines
6.3 KiB
Markdown

# Company agent observability
Company Settings → General → Agent observability configures one company-owned
OTLP destination for Paperclip-managed Codex and Claude Code executions. Export
is disabled by default. Choose Braintrust (US) or a custom HTTP(S) OTLP base URL;
use your own collector when you need fan-out. Braintrust EU and self-hosted
installations can use Custom OTLP with their required routing/authentication
headers. The receiver must be reachable from every execution environment;
`localhost` refers to the execution host, including a remote sandbox.
This is separate from Paperclip's first-party Telemetry and its content-free
[infrastructure tracing](observability.md). Configuring it does not change either data path.
## Capture and supported runtimes
The qualified versions are Codex 0.160.0 and Claude Code through ACP bridge
0.73.0 with Agent SDK 0.3.286. Both logs and traces use HTTP/protobuf. Prompts,
assistant responses and tool content default on. Additional tool details are a
Claude Code control; Codex has no equivalent switch. Codex requests up to 64 KiB
of tool output (or zero when disabled); native event availability, truncation
and response caps still apply. Claude Code applies its own native limits.
Metrics, raw API bodies and Codex Guardian assessments are disabled.
The native runner and legacy local adapters receive transient launch material.
Native Codex writes a private managed config.toml; legacy exec uses a unique
invocation profile, preserving a selected source profile. Its profile injection
requires the qualified executable version. Codex ACPX writes its isolated home.
The legacy ACPX adapter keeps export configuration in a private home per agent
and settings compatibility key, and shares conversation files with its stable managed home;
export changes do not change its logical conversation fingerprint.
Codex's bundled Rust OTLP exporter percent-decodes header values, so launch
material encodes them once to preserve literal percent sequences. Credentials
remain literal values in the company secret; environment interpolation is not used.
A caller-owned/preconfigured Codex home cannot be automatically configured in
this version: its files remain untouched and execution emits a diagnostic.
Other harnesses, personal Codex desktop sessions and hosted claude_managed are
outside this feature.
## Changes and process lifetime
Paperclip reads settings once when an execution begins preparation. That
snapshot governs instruction reservation, compatibility and launch. Already
preparing or running work retains its configuration. A changed settings
revision or resolved credential version retires an idle incompatible warm
process using existing ownership checks. Conversation compatibility is
separate, so a safe replacement can resume the saved conversation. Recovery
never replays an uncertain turn merely to apply new export settings.
Missing, disabled, invalid or unavailable configuration runs without company
export and reports a sanitized availability diagnostic where relevant. Native
export rejection/timeouts do not determine task success; exporter deadlines
are three seconds. Destination calls do not occur in usage, event-append,
tool-acknowledgment or settlement paths.
## Credentials and access
Only board users with Manage connections permission can read or change this
configuration. Agents cannot retrieve it through the settings API. The destination stores no credential
values. Authentication headers live in one encrypted Paperclip-managed company
secret, bound to the system consumer company-observability. Updates rotate that
secret; removal uses the secret service to retire it. Braintrust project
routing is derived from the current project ID, independently of the API key.
Reads return configured status and header names. Blank existing custom header
values retain the saved secret at the same HTTP origin; a change of scheme, host,
or port requires every retained header value to be entered again. New/renamed headers require values. Removing a
row removes that header. API-key replacement is explicit, and Cancel clears
entered secrets. Revisions prevent another administrator's edit being silently
overwritten. Credential values are excluded from task environment manifests,
run/config snapshots, argv, activity records, recovery configuration and query
cache data. Export credentials are necessarily accessible to the exporting
harness itself. Claude Code supplies these headers through its process environment,
which its tools can inherit or inspect. Codex can read its managed configuration.
This feature does not isolate export credentials from agent code. Use a credential
limited to telemetry ingestion for the selected destination; do not supply a broad
organization administration key. A user-operated collector can hold upstream
credentials when a deployment requires a separate trust boundary.
## Connection tests
Saving never contacts the destination. Test saved settings sends one synthetic
root span and a correlated log with a unique marker from the control plane,
even when agent export is disabled. The receiver receives no production task
content. Both requests share a ten-second deadline, cannot redirect, and use
the deployment's guarded HTTP policy with DNS pinning and response-size limits.
Receipts contain sanitized HTTP/signal results and no receiver body text.
A receipt is retained only for the revision that was tested.
“Test payload accepted” means the server's synthetic payload was accepted. It
does not prove agent export, remote network reachability, downstream indexing,
or that a provider UI rendered the records. Harness exporters use their own
network clients under the execution environment's policy; the server's DNS
pinning applies only to the explicit control-plane test. Verify real tasks and
the destination's records before relying on a deployment.
## Verification
The focused suites cover schema bounds, concurrent saves, secret/header
lifecycle, authorization, protobuf transport and rejection, private-network
policy, timeout, revision fencing, environment projections, private native
configuration, warm compatibility, and UI conflict/company-switch behavior.
Storybook IDs agent-observability-01-settings, agent-observability-02-states and
agent-observability-03-journey use fake credentials and production components.
Real local/remote native export evidence must be reported separately from
these automated and synthetic checks.