DottaandPaperclip
3bd709df8b
## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E tests exercise real agents, tools, setup and cleanup. > - Repeated runs need retained costs and traces, including failed attempts. > - A passing retry must not hide earlier spending or feedback. > - Recurring work also needs admission limits, private reports and failure detection. > - This change supplies those controls while keeping paid activation behind existing gates. ## Linked Issues or Issue Description Refs #16996 and #17158. This is the consolidated landing of the reviewed eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033, #17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096, #17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127, #17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository permits squash merges only. A local rehearsal showed that squashing each slice into master causes repeated conflicts in dependent slices. This PR preserves their commits and lands their combined implementation in one reviewed integration. Superseded slices will close after this merge. ## What Changed - Retain every attempt's cost receipt, including setup, failures, interruptions and retries. Separate reported amounts, estimates, missing costs and reserved capacity in reports and dashboards. - Add bounded request admission, exact model pricecards, gateway error accounting, cache policy and reconciliation evidence. Refuse unsupported routes and requests that cannot fit the allowance. - Integrate Braintrust gateway traces and OTLP correlation, with secret filtering and experiment imports. Record phase timing, including setup, feedback capture and cleanup. - Collect complaint and suggestion evidence after each attempt, including recovery after failures. Retain explicit review dispositions independently from the product grade. - Add gated daily and weekly selections, measured forecasts, setup/readiness checks, wall-time limits and retained monitor notifications. Include declared non-model overhead in admission while reporting the existing subscription separately. - Keep recurring history private, bound uploads and retention, and use isolated archive environments. Keep public publication an explicit operation. Report and publication jobs install only the committed lockfile, with scripts disabled and isolated stores; stale pins fail closed. - Integrate current master, preserve its Product E2E journey billing and privacy checks, and test that shared setup receipts do not double-count estimated agent usage. Preserve upstream fixture repairs. ## Verification - Complete integration baseline `0649a5ce7544ab20825c2183d2f421157db49f24` includes master `a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head as ancestors. - On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012 Vitest tests passed, two skipped, across 193 files; all 137 Node source-contract tests passed. - Focused billing and Product E2E journey contract checks: 24 tests passed. They cover estimates, unknown receipts, single-owner setup costs, browser-only execution and private publication guards. - Monitor and workflow wiring: all 96 Node tests passed. `node scripts/check-token-gates.mjs`: all gates clean. - `pnpm test:e2e:runner:typecheck` passes after the standard Runner TypeScript build refreshed local declarations. The first attempt rejected the new master `runner_prepared_launch` union member because this checkout retained an older generated declaration; the source union already contained that member. - Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes dependency re-resolution from archive-authorized reporting jobs after the security scan flagged it. All 15 workflow-security tests pass after updating the old two-install assertion. - Hosted [CI run 39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180) passes on the current repaired head, including typecheck, build, general tests, Runner checks, browser shards and canary dry run. Greptile is 5/5 on this exact head with no unresolved threads. The security scan passes after the fixed lockfile-resolution thread was resolved. - Paperclip Review returned a terminal neutral check but an incomplete 0-file assessment on the repaired head: its platform blocked both diff responses as prompt-injection content. This is not a successful code review. No bypass or fabricated pass was attempted; current-head Greptile, security and hosted CI are independent gates. - No paid test, reservation release or recurring-run activation accompanies this landing. Live route qualification and invoice reconciliation remain separate evidence. <details> <summary>Retained earlier validation failures</summary> Earlier broad local runs had two 15-second test timeouts (CLI poll cadence and Retry validation), one embedded-database setup timeout, and later PostgreSQL semaphore exhaustion. The exact timed-out tests passed separately without larger timeouts; the broad-run causes were not established. Those local failures remain distinct from subsequent hosted CI results. The independent #17158 fix records its Muse fixture collision, its UUID correction and hosted verification. Local Muse execution was also blocked by macOS embedded-PostgreSQL startup. No historical failure is treated as a successful paid qualification. </details> ## Risks - Admission is a local control, not a provider invoice cap. Missing provider receipts or resource charges retain unknown amounts and holds; estimates do not become billed dollars. - The bounded path admits only documented local suites and provider routes. Ordinary legacy or separate Runner protocol campaigns have different controls. Merging this code does not qualify additional routes. - Recurrence still needs the existing enable gates and current qualification/allowance evidence. Archive permissions and resources must match the documented private-environment contract. - Legacy reports may lack complete attempt or phase evidence. Reports preserve that limitation and do not infer zero usage from missing data. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact deployment variant and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks; full-suite host limitation disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
6.6 KiB
Company agent observability
Company Settings → General → Agent observability configures one company-owned
OTLP destination for Paperclip-managed Codex and Claude Code executions. Export
is disabled by default. Choose Braintrust (US) or a custom HTTP(S) OTLP base URL;
use your own collector when you need fan-out. Braintrust EU and self-hosted
installations can use Custom OTLP with their required routing/authentication
headers. The receiver must be reachable from every execution environment;
localhost refers to the execution host, including a remote sandbox.
This is separate from Paperclip's first-party Telemetry and its content-free infrastructure tracing. Configuring it does not change either data path.
Capture and supported runtimes
The qualified versions are Codex 0.160.0 and Claude Code through ACP bridge 0.73.0 with Agent SDK 0.3.286. Both logs and traces use HTTP/protobuf. Prompts, assistant responses and tool content default on. Additional tool details are a Claude Code control; Codex has no equivalent switch. Codex requests up to 64 KiB of tool output (or zero when disabled); native event availability, truncation and response caps still apply. Claude Code applies its own native limits. Metrics, raw API bodies and Codex Guardian assessments are disabled.
The native runner and legacy local adapters receive transient launch material.
Native Codex writes a private managed config.toml; legacy exec uses a unique
invocation profile, preserving a selected source profile. Its profile injection
requires the qualified executable version. Codex ACPX writes its isolated home.
The legacy ACPX adapter keeps export configuration in a private home per agent
and settings compatibility key, and shares conversation files with its stable managed home;
export changes do not change its logical conversation fingerprint.
Codex receives authentication through OTEL_EXPORTER_OTLP_LOGS_HEADERS and
OTEL_EXPORTER_OTLP_TRACES_HEADERS in its process environment. Its managed
configuration and invocation profiles contain empty header maps, so exporting
does not save the ingestion credential in those files. The bundled Rust OTLP
exporter percent-decodes environment header values; launch material encodes
them once to preserve literal percent sequences. Credentials remain literal
values in the company secret; config-file environment interpolation is not used.
A caller-owned/preconfigured Codex home cannot be automatically configured in
this version: its files remain untouched and execution emits a diagnostic.
Other harnesses, personal Codex desktop sessions and hosted claude_managed are
outside this feature.
Changes and process lifetime
Paperclip reads settings once when an execution begins preparation. That snapshot governs instruction reservation, compatibility and launch. Already preparing or running work retains its configuration. A changed settings revision or resolved credential version retires an idle incompatible warm process using existing ownership checks. Conversation compatibility is separate, so a safe replacement can resume the saved conversation. Recovery never replays an uncertain turn merely to apply new export settings.
Missing, disabled, invalid or unavailable configuration runs without company export and reports a sanitized availability diagnostic where relevant. Native export rejection/timeouts do not determine task success; exporter deadlines are three seconds. Destination calls do not occur in usage, event-append, tool-acknowledgment or settlement paths.
Credentials and access
Only board users with Manage connections permission can read or change this configuration. Agents cannot retrieve it through the settings API. The destination stores no credential values. Authentication headers live in one encrypted Paperclip-managed company secret, bound to the system consumer company-observability. Updates rotate that secret; removal uses the secret service to retire it. Braintrust project routing is derived from the current project ID, independently of the API key.
Reads return configured status and header names. Blank existing custom header values retain the saved secret at the same HTTP origin; a change of scheme, host, or port requires every retained header value to be entered again. New/renamed headers require values. Removing a row removes that header. API-key replacement is explicit, and Cancel clears entered secrets. Revisions prevent another administrator's edit being silently overwritten. Credential values are excluded from task environment manifests, run/config snapshots, argv, activity records, recovery configuration and query cache data. Export credentials are necessarily accessible to the exporting harness itself. Both Codex and Claude Code receive these headers through their process environment, which their tools can inherit or inspect. This feature does not isolate export credentials from agent code. Use a credential limited to telemetry ingestion for the selected destination; do not supply a broad organization administration key. A user-operated collector can hold upstream credentials when a deployment requires a separate trust boundary.
Connection tests
Saving never contacts the destination. Test saved settings sends one synthetic root span and a correlated log with a unique marker from the control plane, even when agent export is disabled. The receiver receives no production task content. Both requests share a ten-second deadline, cannot redirect, and use the deployment's guarded HTTP policy with DNS pinning and response-size limits. Receipts contain sanitized HTTP/signal results and no receiver body text. A receipt is retained only for the revision that was tested.
“Test payload accepted” means the server's synthetic payload was accepted. It does not prove agent export, remote network reachability, downstream indexing, or that a provider UI rendered the records. Harness exporters use their own network clients under the execution environment's policy; the server's DNS pinning applies only to the explicit control-plane test. Verify real tasks and the destination's records before relying on a deployment.
Verification
The focused suites cover schema bounds, concurrent saves, secret/header lifecycle, authorization, protobuf transport and rejection, private-network policy, timeout, revision fencing, environment projections, private native configuration, warm compatibility, and UI conflict/company-switch behavior. Storybook IDs agent-observability-01-settings, agent-observability-02-states and agent-observability-03-journey use fake credentials and production components. Real local/remote native export evidence must be reported separately from these automated and synthetic checks.