mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-24 18:19:05 +02:00
DottaandPaperclip
3bd709df8b
## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E tests exercise real agents, tools, setup and cleanup. > - Repeated runs need retained costs and traces, including failed attempts. > - A passing retry must not hide earlier spending or feedback. > - Recurring work also needs admission limits, private reports and failure detection. > - This change supplies those controls while keeping paid activation behind existing gates. ## Linked Issues or Issue Description Refs #16996 and #17158. This is the consolidated landing of the reviewed eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033, #17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096, #17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127, #17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository permits squash merges only. A local rehearsal showed that squashing each slice into master causes repeated conflicts in dependent slices. This PR preserves their commits and lands their combined implementation in one reviewed integration. Superseded slices will close after this merge. ## What Changed - Retain every attempt's cost receipt, including setup, failures, interruptions and retries. Separate reported amounts, estimates, missing costs and reserved capacity in reports and dashboards. - Add bounded request admission, exact model pricecards, gateway error accounting, cache policy and reconciliation evidence. Refuse unsupported routes and requests that cannot fit the allowance. - Integrate Braintrust gateway traces and OTLP correlation, with secret filtering and experiment imports. Record phase timing, including setup, feedback capture and cleanup. - Collect complaint and suggestion evidence after each attempt, including recovery after failures. Retain explicit review dispositions independently from the product grade. - Add gated daily and weekly selections, measured forecasts, setup/readiness checks, wall-time limits and retained monitor notifications. Include declared non-model overhead in admission while reporting the existing subscription separately. - Keep recurring history private, bound uploads and retention, and use isolated archive environments. Keep public publication an explicit operation. Report and publication jobs install only the committed lockfile, with scripts disabled and isolated stores; stale pins fail closed. - Integrate current master, preserve its Product E2E journey billing and privacy checks, and test that shared setup receipts do not double-count estimated agent usage. Preserve upstream fixture repairs. ## Verification - Complete integration baseline `0649a5ce7544ab20825c2183d2f421157db49f24` includes master `a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head as ancestors. - On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012 Vitest tests passed, two skipped, across 193 files; all 137 Node source-contract tests passed. - Focused billing and Product E2E journey contract checks: 24 tests passed. They cover estimates, unknown receipts, single-owner setup costs, browser-only execution and private publication guards. - Monitor and workflow wiring: all 96 Node tests passed. `node scripts/check-token-gates.mjs`: all gates clean. - `pnpm test:e2e:runner:typecheck` passes after the standard Runner TypeScript build refreshed local declarations. The first attempt rejected the new master `runner_prepared_launch` union member because this checkout retained an older generated declaration; the source union already contained that member. - Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes dependency re-resolution from archive-authorized reporting jobs after the security scan flagged it. All 15 workflow-security tests pass after updating the old two-install assertion. - Hosted [CI run 39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180) passes on the current repaired head, including typecheck, build, general tests, Runner checks, browser shards and canary dry run. Greptile is 5/5 on this exact head with no unresolved threads. The security scan passes after the fixed lockfile-resolution thread was resolved. - Paperclip Review returned a terminal neutral check but an incomplete 0-file assessment on the repaired head: its platform blocked both diff responses as prompt-injection content. This is not a successful code review. No bypass or fabricated pass was attempted; current-head Greptile, security and hosted CI are independent gates. - No paid test, reservation release or recurring-run activation accompanies this landing. Live route qualification and invoice reconciliation remain separate evidence. <details> <summary>Retained earlier validation failures</summary> Earlier broad local runs had two 15-second test timeouts (CLI poll cadence and Retry validation), one embedded-database setup timeout, and later PostgreSQL semaphore exhaustion. The exact timed-out tests passed separately without larger timeouts; the broad-run causes were not established. Those local failures remain distinct from subsequent hosted CI results. The independent #17158 fix records its Muse fixture collision, its UUID correction and hosted verification. Local Muse execution was also blocked by macOS embedded-PostgreSQL startup. No historical failure is treated as a successful paid qualification. </details> ## Risks - Admission is a local control, not a provider invoice cap. Missing provider receipts or resource charges retain unknown amounts and holds; estimates do not become billed dollars. - The bounded path admits only documented local suites and provider routes. Ordinary legacy or separate Runner protocol campaigns have different controls. Merging this code does not qualify additional routes. - Recurrence still needs the existing enable gates and current qualification/allowance evidence. Archive permissions and resources must match the documented private-environment contract. - Legacy reports may lack complete attempt or phase evidence. Reports preserve that limitation and do not infer zero usage from missing data. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact deployment variant and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks; full-suite host limitation disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
110 lines
6.6 KiB
Markdown
110 lines
6.6 KiB
Markdown
# Company agent observability
|
|
|
|
Company Settings → General → Agent observability configures one company-owned
|
|
OTLP destination for Paperclip-managed Codex and Claude Code executions. Export
|
|
is disabled by default. Choose Braintrust (US) or a custom HTTP(S) OTLP base URL;
|
|
use your own collector when you need fan-out. Braintrust EU and self-hosted
|
|
installations can use Custom OTLP with their required routing/authentication
|
|
headers. The receiver must be reachable from every execution environment;
|
|
`localhost` refers to the execution host, including a remote sandbox.
|
|
|
|
This is separate from Paperclip's first-party Telemetry and its content-free
|
|
[infrastructure tracing](observability.md). Configuring it does not change either data path.
|
|
|
|
## Capture and supported runtimes
|
|
|
|
The qualified versions are Codex 0.160.0 and Claude Code through ACP bridge
|
|
0.73.0 with Agent SDK 0.3.286. Both logs and traces use HTTP/protobuf. Prompts,
|
|
assistant responses and tool content default on. Additional tool details are a
|
|
Claude Code control; Codex has no equivalent switch. Codex requests up to 64 KiB
|
|
of tool output (or zero when disabled); native event availability, truncation
|
|
and response caps still apply. Claude Code applies its own native limits.
|
|
Metrics, raw API bodies and Codex Guardian assessments are disabled.
|
|
|
|
The native runner and legacy local adapters receive transient launch material.
|
|
Native Codex writes a private managed config.toml; legacy exec uses a unique
|
|
invocation profile, preserving a selected source profile. Its profile injection
|
|
requires the qualified executable version. Codex ACPX writes its isolated home.
|
|
The legacy ACPX adapter keeps export configuration in a private home per agent
|
|
and settings compatibility key, and shares conversation files with its stable managed home;
|
|
export changes do not change its logical conversation fingerprint.
|
|
Codex receives authentication through `OTEL_EXPORTER_OTLP_LOGS_HEADERS` and
|
|
`OTEL_EXPORTER_OTLP_TRACES_HEADERS` in its process environment. Its managed
|
|
configuration and invocation profiles contain empty header maps, so exporting
|
|
does not save the ingestion credential in those files. The bundled Rust OTLP
|
|
exporter percent-decodes environment header values; launch material encodes
|
|
them once to preserve literal percent sequences. Credentials remain literal
|
|
values in the company secret; config-file environment interpolation is not used.
|
|
A caller-owned/preconfigured Codex home cannot be automatically configured in
|
|
this version: its files remain untouched and execution emits a diagnostic.
|
|
Other harnesses, personal Codex desktop sessions and hosted claude_managed are
|
|
outside this feature.
|
|
|
|
## Changes and process lifetime
|
|
|
|
Paperclip reads settings once when an execution begins preparation. That
|
|
snapshot governs instruction reservation, compatibility and launch. Already
|
|
preparing or running work retains its configuration. A changed settings
|
|
revision or resolved credential version retires an idle incompatible warm
|
|
process using existing ownership checks. Conversation compatibility is
|
|
separate, so a safe replacement can resume the saved conversation. Recovery
|
|
never replays an uncertain turn merely to apply new export settings.
|
|
|
|
Missing, disabled, invalid or unavailable configuration runs without company
|
|
export and reports a sanitized availability diagnostic where relevant. Native
|
|
export rejection/timeouts do not determine task success; exporter deadlines
|
|
are three seconds. Destination calls do not occur in usage, event-append,
|
|
tool-acknowledgment or settlement paths.
|
|
|
|
## Credentials and access
|
|
|
|
Only board users with Manage connections permission can read or change this
|
|
configuration. Agents cannot retrieve it through the settings API. The destination stores no credential
|
|
values. Authentication headers live in one encrypted Paperclip-managed company
|
|
secret, bound to the system consumer company-observability. Updates rotate that
|
|
secret; removal uses the secret service to retire it. Braintrust project
|
|
routing is derived from the current project ID, independently of the API key.
|
|
|
|
Reads return configured status and header names. Blank existing custom header
|
|
values retain the saved secret at the same HTTP origin; a change of scheme, host,
|
|
or port requires every retained header value to be entered again. New/renamed headers require values. Removing a
|
|
row removes that header. API-key replacement is explicit, and Cancel clears
|
|
entered secrets. Revisions prevent another administrator's edit being silently
|
|
overwritten. Credential values are excluded from task environment manifests,
|
|
run/config snapshots, argv, activity records, recovery configuration and query
|
|
cache data. Export credentials are necessarily accessible to the exporting
|
|
harness itself. Both Codex and Claude Code receive these headers through their
|
|
process environment, which their tools can inherit or inspect.
|
|
This feature does not isolate export credentials from agent code. Use a credential
|
|
limited to telemetry ingestion for the selected destination; do not supply a broad
|
|
organization administration key. A user-operated collector can hold upstream
|
|
credentials when a deployment requires a separate trust boundary.
|
|
|
|
## Connection tests
|
|
|
|
Saving never contacts the destination. Test saved settings sends one synthetic
|
|
root span and a correlated log with a unique marker from the control plane,
|
|
even when agent export is disabled. The receiver receives no production task
|
|
content. Both requests share a ten-second deadline, cannot redirect, and use
|
|
the deployment's guarded HTTP policy with DNS pinning and response-size limits.
|
|
Receipts contain sanitized HTTP/signal results and no receiver body text.
|
|
A receipt is retained only for the revision that was tested.
|
|
|
|
“Test payload accepted” means the server's synthetic payload was accepted. It
|
|
does not prove agent export, remote network reachability, downstream indexing,
|
|
or that a provider UI rendered the records. Harness exporters use their own
|
|
network clients under the execution environment's policy; the server's DNS
|
|
pinning applies only to the explicit control-plane test. Verify real tasks and
|
|
the destination's records before relying on a deployment.
|
|
|
|
## Verification
|
|
|
|
The focused suites cover schema bounds, concurrent saves, secret/header
|
|
lifecycle, authorization, protobuf transport and rejection, private-network
|
|
policy, timeout, revision fencing, environment projections, private native
|
|
configuration, warm compatibility, and UI conflict/company-switch behavior.
|
|
Storybook IDs agent-observability-01-settings, agent-observability-02-states and
|
|
agent-observability-03-journey use fake credentials and production components.
|
|
Real local/remote native export evidence must be reported separately from
|
|
these automated and synthetic checks.
|