PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-28 12:04:41 +01:00

Commit Graph

Author	SHA1	Message	Date
DottaandPaperclip	6c9b87d092	trace(server): expose repository admission waits (#17040 ) ## Thinking Path > - Paperclip manages work performed by AI agents. > - The controller prepares a task workspace before it starts a provider turn. > - Repository admission also runs when a checkout already exists. > - One observed reuse took 16.877 seconds. Its child spans explained only 3.556 seconds. > - The missing detail covered repeated authority checks, catalog work, and publication waits. > - This change exposes those operations under the existing execution attempt. > - Operators can expand the slow phase before choosing a performance fix. ## Linked Issues or Issue Description Refs #16963, #16330, and #16515. The provider-routing work in #17019 has a different scope. What existing behavior does this improve? Repository-admission traces for new and reused checkouts. Subsystem affected server/ orchestration and operator-configured OpenTelemetry tracing. Current behavior The admission span can contain a long interval without child spans. Existing project checks, HTTP requests, and checkout lookup do not explain current-task checks, database awaits, credential refresh, or publication locks. Proposed behavior Expose nested spans for current binding and task access, individual identity reads, catalog construction, credentials, grant transactions, filesystem validation, Git inspection, and directory-lock acquisition, work, and release. Reason and benefit The trace can distinguish repeated checks, remote calls, database awaits, and directory contention. A database-await span includes driver, network, pool, and server time. It is not isolated SQL execution time. Breaking changes None. No API, schema, permission, retry, lock, or repository behavior changes. ## What Changed - Add fixed admission phases beneath the existing execution attempt. Keep the existing project, checkout, and HTTP detail. - Trace current-task authorization stages only under workspace access. Unrelated decisions cannot consume that detail budget before admission. - Trace GitHub catalog reads, credentials, provider metadata, grant row locking and writes, transaction settlement, and webhook work. - Separate directory-lock acquisition, the protected operation, and release without changing the adapter lock contract. - Document the hierarchy and timing limits. Keep endpoint gating, span caps, and hashed run correlation. Add no span attributes or exception text. ## Verification - Nine new tests passed with real PostgreSQL, real Git, and SDK-exported spans. They cover directory contention, grant-row blocking, all three binding checks, reuse without cloning, dirty files, private-task denial, receipt repair, conflict retry, endpoint-off behavior, and parent IDs. - Twenty earlier unrelated authorization decisions do not exhaust the admission detail budget. - Existing authorization and trust suites: 105 tests passed. Existing timing/context suites: 26 tests passed. Existing repository-access tests: 6 tests passed. - `pnpm -r typecheck` and `pnpm build` passed on commit `5a986d8c86644dfb7ee39b3b832e2d7c86532598`. - `pnpm test:run` could not complete locally. The temporary five-connection cluster returned `sorry, too many clients already` in chat fixtures, and the run was interrupted. It is not a pass. The complete standard CI workflow passed on the same commit, including all 12 server shards and all three chat shards. - Final current-head checks: 54 success, two neutral, two skipped, zero pending or failing. The [CI aggregate](https://github.com/paperclipai/paperclip/actions/runs/39522187940/job/118971261973) passed. [Greptile](https://github.com/paperclipai/paperclip/pull/17040#issuecomment-6305066520) and [Paperclip Review](https://github.com/paperclipai/paperclip/pull/17040#issuecomment-6305017182) both assessed this exact commit at 5/5 with no actionable findings or unresolved threads. - `node scripts/check-module-boundaries.mjs` and `git diff --check` passed. - This macOS host exhausted its SysV semaphore quota. Database tests used a temporary command-scoped initdb preload with five connections, fewer workers, and a larger lock table. No repository test infrastructure or shared host settings changed. Initial skipped tests, focused-fixture failures, and clock-order assertion failures were corrected and rerun; they are not counted as passes. The complete local run remained connection-limited and was interrupted. - Local SDK export proves parentage and privacy. Hosted Honeycomb ingestion remains unverified until deployment. No deployment occurred. ## Risks Span recording requires an operator-configured OTLP endpoint. Each fixed detail name remains capped at 16 occurrences per attempt. Later operations still execute under their enclosing phase. Timing wrappers preserve the existing operations and their errors. No migration is required. The change does not itself reduce admission latency. ## Model Used OpenAI Codex, based on GPT-6, with tool use, code execution, and delegated agents. The exact hosted model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 19:50:51 -05:00
DottaandPaperclip	25b16e65fc	fix(observability): detail workspace and connector preparation (#16454 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server prepares a repository, sandbox workspace, and connector access before an agent can reply. > - Existing traces measure large preparation phases but leave important work inside those phases unclear. > - Repository source materialization and Browser Use discovery can do more work than their callers suggest. > - New detail must preserve access checks, run ownership, and the existing work order. > - This pull request adds fixed OpenTelemetry spans inside those operations. > - Operators can use one deployed trace to select the next performance fix. ## Linked Issues or Issue Description Refs: #16419, #16425, #16428. What existing behavior does this improve? Startup traces for server workspace preparation, native sandbox synchronization, and connector assignment resolution. Current behavior Workspace and connector aggregate spans can take many seconds. The trace does not separate private source cloning, seed reuse, archive work, progress persistence, or each connector resolver. Browser Use resource selection builds the full authorized tool catalog. The configuration and native backend can each resolve connector assignments. Proposed behavior Show the awaited work as fixed child spans. Separate discovery queue wait from catalog construction. Preserve the initiating run context when another run releases a scheduler slot. Add no new lifecycle event writes for workspace detail callbacks. Reason and benefit Operators need evidence about the hosted critical path before another performance change. These spans distinguish filesystem work, provider transfer, authorization, transaction entry, locks, and reporting overhead. Breaking changes None. Authorization, query order, transaction ownership, synchronization, existing progress events, and task results retain their contracts. The change adds opt-in Observability spans under the existing OTLP endpoint gate. ## What Changed - Trace private source directory setup, Git credentials, clone, requested ref, source identity, and checkout publication. - Trace sandbox source capture, seed cache/replay, shallow clone and archive, overlay copy/archive, verification, upload, warm stamps, and descriptor persistence. - Separate existing awaited progress reporting from useful workspace work. Keep new detail callbacks trace-only. - Trace each connector resolver and Browser Use catalog/grant work. Split discovery queueing, descriptor sources, and policy evaluation. - Trace Slack authority, credential reads, catalog synchronization, transaction entry/settlement, and advisory locks. - Update the real filesystem staging trace expectations and the Honeycomb investigation guide. - Add a real scheduler regression for a queued job released by another run. Assert run ownership, parent spans, and exactly one completed queue span. ## Verification - Passed: server and adapter-utils TypeScript checks (`pnpm --filter @paperclipai/server exec tsc --noEmit` and `pnpm --filter @paperclipai/adapter-utils exec tsc --noEmit`). - Passed: 252 checks across 9 focused files. These cover real filesystem staging, native workspace synchronization, trace ownership, endpoint gating, and workspace preparation. - Passed after review: 26 trace-context, tracing-failure, and discovery-scheduler checks across 3 files (these overlap the first run). The new queued-discovery regression fails when context binding is removed and passes after restoration. An initial submillisecond timestamp assertion was replaced with explicit span-completion order to avoid SDK clock precision assumptions. - Passed on implementation commit `0e68968a4bd61001893d833f281f12bb110edbe1`: `pnpm -r typecheck` and `pnpm build`. The later commit updates only an engine test expectation. - The first CI workspace lane found an old engine trace-tree expectation. It expected the earlier span set. The updated assertion now checks the new detail spans, their parents, and single completion. Its targeted executor check passes. - The full test suite, build, release registry, and end-to-end gates pass in CI on final head `40e37d89ef679109d60a96bb5731a0616efeaa43`: 55 passing checks and 2 optional Storybook skips. The full suite was not duplicated locally after the focused run. - Greptile reviewed the final head at 5/5. No unresolved review threads remain. - After deployment, filter Honeycomb by the hashed run ID and verify `service.version`. Inspect `heartbeat.prepare.workspace`, `runtime_selection.workspace_sync`, and the connector assignment parents. The guide in `doc/observability.md` explains each new family and the nested intervals. - No hosted run has used this new instrumentation yet. Local checks cannot establish hosted latency improvement. ## Risks - Tracing adds spans when an operator configures an OTLP endpoint. Detail remains capped at 16 spans per phase per execution attempt. - Parent and child durations overlap. The documentation explains how to avoid counting them twice. - A discovery job can start in another run's scheduler continuation. The bound callback preserves the initiating async context. - New spans export fixed names and the existing hashed run ID. They add no paths, repository URLs, connector resource IDs, credentials, tool schemas, or exception text. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact served model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-15 12:21:32 -05:00
DottaandPaperclip	cccde7122b	Trace execution preparation and durable response persistence (#16419 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server prepares an execution environment before it starts an agent. > - Operators need to distinguish preparation, provider wait, and response persistence. > - Existing preparation phases start separate traces, and the native session trace ends before the final response is saved. > - This pull request adds an execution-attempt trace and detailed spans at the existing work boundaries. > - The benefit is a measurable startup and settlement path without changing execution policy. ## Linked Issues or Issue Description What existing behavior does this improve? Operator-configured OpenTelemetry tracing for agent preparation, startup, and durable response storage. Subsystem affected Server orchestration, native Runner startup, and the Honeycomb run link. Current behavior Preparation phases start independent roots. Workspace lock waits, authorization groups, runtime asset work, provider-pack fallback, and the gap after provider completion are hard to attribute. Proposed behavior Use a server-owned async context for each execution attempt. Parent nested work to that attempt. Link the separately sampled native session because its ingress and queue history can begin earlier. Record transaction entry, lock waits, grouped authorization, asset preparation, artifact fallback, copyback, and response commit. Reason and benefit Operators can identify expensive work from recorded boundaries instead of inferring the cause from elapsed time. Nested and concurrent durations remain visible and must not be added together. Breaking changes No API or execution-policy change. New Observability span names and closed count attributes are documented. The existing local run-performance schema also records the additional provider-pack and harness measurements. Refs: #16178, #16205, #16330. ## What Changed - Add `task.execution_attempt` with same-run async ownership, independent root sampling, and native-session links. - Measure workspace transaction entry and settlement, advisory and row waits, source scans, reader groups, notifications, and writes. - Measure runtime selection and each concurrent instructions, skills, and MCP branch. Separate manifest, cache, materialization, and verification work. - Measure provider-pack discovery, verified reuse or upload, final verification, and remote harness state checks. - Measure workspace settlement, accounting, liveness, response authorization, comment commit, and publication enqueue. - Preserve the silent endpoint-off path, including when another library has registered a tracer. Bound repeated detail spans to 16 per name per attempt. - Extend the Honeycomb run link and document the actual coverage and remaining gaps. ## Verification - After resolving the conflict with retained-reader batching on `b98cdaea531c8447637cd4dcc19c61b3f5f41844`, all 61 targeted query, authorization, workspace-preparation and trace-boundary tests passed. The server typecheck passed. Reader tests still bound query count, avoid loading retained run payloads and enforce fresh project grants on every publication attempt. - Before that base integration, the tracing/exporter group passed 855 tests with 2 optional-SDK skips across 10 files and no inherited OTLP endpoint. It covers real PostgreSQL waits/rollback, durable response commit, assembled heartbeat preparation, concurrent parentage, sampling and tracer failure isolation. Native workspace/history tests passed 23 cases. These results remain supporting evidence for the unchanged tracing paths. - Full workspace typecheck and build passed during implementation. Full sharded CI, release validation, E2E and Apex 5/5 passed on prior head `8cebad80f3856dbcff6f3a6d5d34880ff77a20f2`. Fresh CI, security and Apex review are required on the current head before merge. - `pnpm check:token-gates` passed. No styles changed. - Earlier CI runs exposed exact-source-expression assertions changed by wrappers, missing run hashes on transaction-boundary spans, and sampling fixtures without an enabled endpoint. The fixes preserve the calls, add the declared hash and span cap, and enable tracing explicitly in the sampling fixtures. Focused repairs passed; the full prior-head CI passed after those repairs. Earlier failed run: https://github.com/paperclipai/paperclip/actions/runs/38729131366. - The full local `pnpm test:run` was stopped after the base advanced during that process. Its mixed-version results are not final-head evidence. A full local-suite pass is not claimed. - The old-head Superagent scan stalled at 35%. The current head has a fresh scan; no security pass is inferred from the old pending result. - No deployed Honeycomb acceptance is claimed by this code change. ## Risks - Enabled tracing creates more spans. Repeated detail spans are bounded, and inventories use group spans. - The attempt and native session remain linked traces. Historical queue time can precede the attempt, and nested intervals overlap. - A committed comment proves durable server storage. Publication enqueue and realtime dispatch do not prove external delivery or browser paint. - New custom attributes use only the existing run hash and validated file/byte counts. No SQL, paths, permissions, content, or exception text is added. ## Model Used OpenAI GPT-6 through Codex. The serving model ID and context window size are not exposed in this session. The agent used reasoning, tools, code editing, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-15 08:13:53 -05:00

Author

SHA1

Message

Date

DottaandPaperclip

6c9b87d092

trace(server): expose repository admission waits (#17040 )

## Thinking Path

> - Paperclip manages work performed by AI agents.
> - The controller prepares a task workspace before it starts a provider
turn.
> - Repository admission also runs when a checkout already exists.
> - One observed reuse took 16.877 seconds. Its child spans explained
only 3.556 seconds.
> - The missing detail covered repeated authority checks, catalog work,
and publication waits.
> - This change exposes those operations under the existing execution
attempt.
> - Operators can expand the slow phase before choosing a performance
fix.

## Linked Issues or Issue Description

Refs #16963, #16330, and #16515. The provider-routing work in #17019 has
a different scope.

**What existing behavior does this improve?**

Repository-admission traces for new and reused checkouts.

**Subsystem affected**

server/ orchestration and operator-configured OpenTelemetry tracing.

**Current behavior**

The admission span can contain a long interval without child spans.
Existing project checks, HTTP requests, and checkout lookup do not
explain current-task checks, database awaits, credential refresh, or
publication locks.

**Proposed behavior**

Expose nested spans for current binding and task access, individual
identity reads, catalog construction, credentials, grant transactions,
filesystem validation, Git inspection, and directory-lock acquisition,
work, and release.

**Reason and benefit**

The trace can distinguish repeated checks, remote calls, database
awaits, and directory contention. A database-await span includes driver,
network, pool, and server time. It is not isolated SQL execution time.

**Breaking changes**

None. No API, schema, permission, retry, lock, or repository behavior
changes.

## What Changed

- Add fixed admission phases beneath the existing execution attempt.
Keep the existing project, checkout, and HTTP detail.
- Trace current-task authorization stages only under workspace access.
Unrelated decisions cannot consume that detail budget before admission.
- Trace GitHub catalog reads, credentials, provider metadata, grant row
locking and writes, transaction settlement, and webhook work.
- Separate directory-lock acquisition, the protected operation, and
release without changing the adapter lock contract.
- Document the hierarchy and timing limits. Keep endpoint gating, span
caps, and hashed run correlation. Add no span attributes or exception
text.

## Verification

- Nine new tests passed with real PostgreSQL, real Git, and SDK-exported
spans. They cover directory contention, grant-row blocking, all three
binding checks, reuse without cloning, dirty files, private-task denial,
receipt repair, conflict retry, endpoint-off behavior, and parent IDs.
- Twenty earlier unrelated authorization decisions do not exhaust the
admission detail budget.
- Existing authorization and trust suites: 105 tests passed. Existing
timing/context suites: 26 tests passed. Existing repository-access
tests: 6 tests passed.
- `pnpm -r typecheck` and `pnpm build` passed on commit
`5a986d8c86644dfb7ee39b3b832e2d7c86532598`.
- `pnpm test:run` could not complete locally. The temporary
five-connection cluster returned `sorry, too many clients already` in
chat fixtures, and the run was interrupted. It is not a pass. The
complete standard CI workflow passed on the same commit, including all
12 server shards and all three chat shards.
- Final current-head checks: 54 success, two neutral, two skipped, zero
pending or failing. The [CI
aggregate](https://github.com/paperclipai/paperclip/actions/runs/39522187940/job/118971261973)
passed.
[Greptile](https://github.com/paperclipai/paperclip/pull/17040#issuecomment-6305066520)
and [Paperclip
Review](https://github.com/paperclipai/paperclip/pull/17040#issuecomment-6305017182)
both assessed this exact commit at 5/5 with no actionable findings or
unresolved threads.
- `node scripts/check-module-boundaries.mjs` and `git diff --check`
passed.
- This macOS host exhausted its SysV semaphore quota. Database tests
used a temporary command-scoped initdb preload with five connections,
fewer workers, and a larger lock table. No repository test
infrastructure or shared host settings changed. Initial skipped tests,
focused-fixture failures, and clock-order assertion failures were
corrected and rerun; they are not counted as passes. The complete local
run remained connection-limited and was interrupted.
- Local SDK export proves parentage and privacy. Hosted Honeycomb
ingestion remains unverified until deployment. No deployment occurred.

## Risks

Span recording requires an operator-configured OTLP endpoint. Each fixed
detail name remains capped at 16 occurrences per attempt. Later
operations still execute under their enclosing phase. Timing wrappers
preserve the existing operations and their errors. No migration is
required. The change does not itself reduce admission latency.

## Model Used

OpenAI Codex, based on GPT-6, with tool use, code execution, and
delegated agents. The exact hosted model identifier and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-20 19:50:51 -05:00

DottaandPaperclip

25b16e65fc

fix(observability): detail workspace and connector preparation (#16454 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server prepares a repository, sandbox workspace, and connector
access before an agent can reply.
> - Existing traces measure large preparation phases but leave important
work inside those phases unclear.
> - Repository source materialization and Browser Use discovery can do
more work than their callers suggest.
> - New detail must preserve access checks, run ownership, and the
existing work order.
> - This pull request adds fixed OpenTelemetry spans inside those
operations.
> - Operators can use one deployed trace to select the next performance
fix.

## Linked Issues or Issue Description

Refs: #16419, #16425, #16428.

**What existing behavior does this improve?**

Startup traces for server workspace preparation, native sandbox
synchronization, and connector assignment resolution.

**Current behavior**

Workspace and connector aggregate spans can take many seconds. The trace
does not separate private source cloning, seed reuse, archive work,
progress persistence, or each connector resolver. Browser Use resource
selection builds the full authorized tool catalog. The configuration and
native backend can each resolve connector assignments.

**Proposed behavior**

Show the awaited work as fixed child spans. Separate discovery queue
wait from catalog construction. Preserve the initiating run context when
another run releases a scheduler slot. Add no new lifecycle event writes
for workspace detail callbacks.

**Reason and benefit**

Operators need evidence about the hosted critical path before another
performance change. These spans distinguish filesystem work, provider
transfer, authorization, transaction entry, locks, and reporting
overhead.

**Breaking changes**

None. Authorization, query order, transaction ownership,
synchronization, existing progress events, and task results retain their
contracts. The change adds opt-in Observability spans under the existing
OTLP endpoint gate.

## What Changed

- Trace private source directory setup, Git credentials, clone,
requested ref, source identity, and checkout publication.
- Trace sandbox source capture, seed cache/replay, shallow clone and
archive, overlay copy/archive, verification, upload, warm stamps, and
descriptor persistence.
- Separate existing awaited progress reporting from useful workspace
work. Keep new detail callbacks trace-only.
- Trace each connector resolver and Browser Use catalog/grant work.
Split discovery queueing, descriptor sources, and policy evaluation.
- Trace Slack authority, credential reads, catalog synchronization,
transaction entry/settlement, and advisory locks.
- Update the real filesystem staging trace expectations and the
Honeycomb investigation guide.
- Add a real scheduler regression for a queued job released by another
run. Assert run ownership, parent spans, and exactly one completed queue
span.

## Verification

- Passed: server and adapter-utils TypeScript checks (`pnpm --filter
@paperclipai/server exec tsc --noEmit` and `pnpm --filter
@paperclipai/adapter-utils exec tsc --noEmit`).
- Passed: 252 checks across 9 focused files. These cover real filesystem
staging, native workspace synchronization, trace ownership, endpoint
gating, and workspace preparation.
- Passed after review: 26 trace-context, tracing-failure, and
discovery-scheduler checks across 3 files (these overlap the first run).
The new queued-discovery regression fails when context binding is
removed and passes after restoration. An initial submillisecond
timestamp assertion was replaced with explicit span-completion order to
avoid SDK clock precision assumptions.
- Passed on implementation commit
`0e68968a4bd61001893d833f281f12bb110edbe1`: `pnpm -r typecheck` and
`pnpm build`. The later commit updates only an engine test expectation.
- The first CI workspace lane found an old engine trace-tree
expectation. It expected the earlier span set. The updated assertion now
checks the new detail spans, their parents, and single completion. Its
targeted executor check passes.
- The full test suite, build, release registry, and end-to-end gates
pass in CI on final head `40e37d89ef679109d60a96bb5731a0616efeaa43`: 55
passing checks and 2 optional Storybook skips. The full suite was not
duplicated locally after the focused run.
- Greptile reviewed the final head at 5/5. No unresolved review threads
remain.
- After deployment, filter Honeycomb by the hashed run ID and verify
`service.version`. Inspect `heartbeat.prepare.workspace`,
`runtime_selection.workspace_sync`, and the connector assignment
parents. The guide in `doc/observability.md` explains each new family
and the nested intervals.
- No hosted run has used this new instrumentation yet. Local checks
cannot establish hosted latency improvement.

## Risks

- Tracing adds spans when an operator configures an OTLP endpoint.
Detail remains capped at 16 spans per phase per execution attempt.
- Parent and child durations overlap. The documentation explains how to
avoid counting them twice.
- A discovery job can start in another run's scheduler continuation. The
bound callback preserves the initiating async context.
- New spans export fixed names and the existing hashed run ID. They add
no paths, repository URLs, connector resource IDs, credentials, tool
schemas, or exception text.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, and code
execution. The exact served model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-15 12:21:32 -05:00

DottaandPaperclip

cccde7122b

Trace execution preparation and durable response persistence (#16419 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server prepares an execution environment before it starts an
agent.
> - Operators need to distinguish preparation, provider wait, and
response persistence.
> - Existing preparation phases start separate traces, and the native
session trace ends before the final response is saved.
> - This pull request adds an execution-attempt trace and detailed spans
at the existing work boundaries.
> - The benefit is a measurable startup and settlement path without
changing execution policy.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Operator-configured OpenTelemetry tracing for agent preparation,
startup, and durable response storage.

**Subsystem affected**

Server orchestration, native Runner startup, and the Honeycomb run link.

**Current behavior**

Preparation phases start independent roots. Workspace lock waits,
authorization groups, runtime asset work, provider-pack fallback, and
the gap after provider completion are hard to attribute.

**Proposed behavior**

Use a server-owned async context for each execution attempt. Parent
nested work to that attempt. Link the separately sampled native session
because its ingress and queue history can begin earlier. Record
transaction entry, lock waits, grouped authorization, asset preparation,
artifact fallback, copyback, and response commit.

**Reason and benefit**

Operators can identify expensive work from recorded boundaries instead
of inferring the cause from elapsed time. Nested and concurrent
durations remain visible and must not be added together.

**Breaking changes**

No API or execution-policy change. New Observability span names and
closed count attributes are documented. The existing local
run-performance schema also records the additional provider-pack and
harness measurements.

Refs: #16178, #16205, #16330.

## What Changed

- Add `task.execution_attempt` with same-run async ownership,
independent root sampling, and native-session links.
- Measure workspace transaction entry and settlement, advisory and row
waits, source scans, reader groups, notifications, and writes.
- Measure runtime selection and each concurrent instructions, skills,
and MCP branch. Separate manifest, cache, materialization, and
verification work.
- Measure provider-pack discovery, verified reuse or upload, final
verification, and remote harness state checks.
- Measure workspace settlement, accounting, liveness, response
authorization, comment commit, and publication enqueue.
- Preserve the silent endpoint-off path, including when another library
has registered a tracer. Bound repeated detail spans to 16 per name per
attempt.
- Extend the Honeycomb run link and document the actual coverage and
remaining gaps.

## Verification

- After resolving the conflict with retained-reader batching on
`b98cdaea531c8447637cd4dcc19c61b3f5f41844`, all 61 targeted query,
authorization, workspace-preparation and trace-boundary tests passed.
The server typecheck passed. Reader tests still bound query count, avoid
loading retained run payloads and enforce fresh project grants on every
publication attempt.
- Before that base integration, the tracing/exporter group passed 855
tests with 2 optional-SDK skips across 10 files and no inherited OTLP
endpoint. It covers real PostgreSQL waits/rollback, durable response
commit, assembled heartbeat preparation, concurrent parentage, sampling
and tracer failure isolation. Native workspace/history tests passed 23
cases. These results remain supporting evidence for the unchanged
tracing paths.
- Full workspace typecheck and build passed during implementation. Full
sharded CI, release validation, E2E and Apex 5/5 passed on prior head
`8cebad80f3856dbcff6f3a6d5d34880ff77a20f2`. Fresh CI, security and Apex
review are required on the current head before merge.
- `pnpm check:token-gates` passed. No styles changed.
- Earlier CI runs exposed exact-source-expression assertions changed by
wrappers, missing run hashes on transaction-boundary spans, and sampling
fixtures without an enabled endpoint. The fixes preserve the calls, add
the declared hash and span cap, and enable tracing explicitly in the
sampling fixtures. Focused repairs passed; the full prior-head CI passed
after those repairs. Earlier failed run:
https://github.com/paperclipai/paperclip/actions/runs/38729131366.
- The full local `pnpm test:run` was stopped after the base advanced
during that process. Its mixed-version results are not final-head
evidence. A full local-suite pass is not claimed.
- The old-head Superagent scan stalled at 35%. The current head has a
fresh scan; no security pass is inferred from the old pending result.
- No deployed Honeycomb acceptance is claimed by this code change.

## Risks

- Enabled tracing creates more spans. Repeated detail spans are bounded,
and inventories use group spans.
- The attempt and native session remain linked traces. Historical queue
time can precede the attempt, and nested intervals overlap.
- A committed comment proves durable server storage. Publication enqueue
and realtime dispatch do not prove external delivery or browser paint.
- New custom attributes use only the existing run hash and validated
file/byte counts. No SQL, paths, permissions, content, or exception text
is added.

## Model Used

OpenAI GPT-6 through Codex. The serving model ID and context window size
are not exposed in this session. The agent used reasoning, tools, code
editing, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-15 08:13:53 -05:00