Commit Graph
6 Commits
Author SHA1 Message Date
Nicky LeachandPaperclip 1d571de5f9 Add shared native turn attribution boundaries (#18213)
## Thinking Path

> - Paperclip manages work performed by AI agents.
> - Native execution crosses the plugin worker, Runner, sidecar, and
durable event boundaries.
> - Existing outer spans leave repeated preparation and admission work
unattributed.
> - Optimizations need a common baseline with accurate ownership and
event meanings.
> - This change extends the existing opt-in tracing contract with
bounded timing receipts.
> - It preserves execution policy and keeps provider internals opaque
where the API provides no timing proof.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Attribution of native task preparation, retained/cold Runner handoff,
and completion-to-text work.

**Current behavior**
Worker spans use a flat provider contract. Workspace calls lack a
consistent dispatch envelope. Submitted-to-started timing is named as a
provider queue even though it includes other work. The transport does
not expose a shared breakdown of its waits.

**Proposed improvement**
Add invocation-scoped nesting, bounded numeric remote receipts,
current-run transport timing, durable receipt intervals, and distinct
completion/text milestones. Reuse the existing exporter. No latency
optimization is included.

Related public work: #17842 consolidates existing startup tracing tests;
it does not add this runtime coverage.

## What Changed

- Extend the optional SDK tracer with nested async scopes and completed
timing receipts. Keep the host responsible for parent identity,
capability checks, attributes, clocks, depth, and count limits.
- Add dispatch and bounded host-read timings for environment and
workspace providers.
- Resolve transport parents on each call so retained sessions do not
retain an earlier run's trace.
- Split cold/resume/attach/prepare/registration/command/started-event
waits. Add opt-in numeric ACPX Runner/sidecar status and prompt-start
receipts.
- Distinguish first completion request, validated tool receipt, final
result commit, first nonempty text, and event persistence. Rename the
submitted-to-started interval to match its meaning.
- Link completed earlier remote lifecycles to their host-owned observer
rather than placing them under a later polling request. Preserve
sampling, timestamp limits and budgets.
- Document the versioned coverage map, causal clock limits, and baseline
acceptance requirements.

## Verification

- Current head: GitHub CI is fully green. Standard Greptile and Apex are
both 5/5, with no unresolved review threads. Full-suite validation was
performed in CI; follow-up local validation was limited to affected
packages and tests.

- Review follow-up: early rejection, missing provider capability and
startup-parent regressions were reproduced and fixed. The affected
worker/SDK/timing set passes 128 tests; 28 timing/context tests, SDK
typecheck and direct server TypeScript checking pass. The server
typecheck wrapper reached an unrelated Runner rebuild and stopped
because Cargo was absent from PATH; full Runner/image validation remains
in CI.

- CI failure accounting: the earlier Turbo cache/force conflict and
Linux OpenCode shell assertion were reproduced and diagnosed. A direct
proc probe ruled out permissions; a portable shell comparison identified
Bash skipping BASH_ENV with stripped SHLVL and socket stdin. Upstream
#18230 now supplies the CI and ignored-stdin fixture fixes; this PR
adopts them without duplicate CI changes. The strict Linux OOM assertion
passed before this source-equivalent merge.
- Retry follow-up: the three insertion/replay/failure timing cases and
server typecheck pass. The existing database regression now asserts a
single marker across identical result replay; the current-head CI
integration gate passes.
- Upstream CI alignment: all three real-Turbo 2.11.10 integration cases
pass. The repository policy gate rejected committing a generated
lockfile; the final PR contains no lockfile edit and uses the maintained
inline stale-lockfile recovery.
- Linked-lifecycle follow-up: 44 focused host/export tests pass (three
optional SDK-dependent cases skip); SDK and server typechecks pass. The
API contract test verifies an explicit root context, host-owned observer
link and unsampled suppression.
- SDK typecheck and focused worker/invocation tests: 30 passed,
including clock changes, bounded legacy IPC, and manual-span
creation-scope parentage.
- Focused reviewed host/native/transaction-boundary checks: 44 passed.
The 86-test worker-manager suite passes after updating its exact
assertions for the host-owned timing budget found by CI. Earlier
preparation/first-text checks also passed.
- Affected server and Runner TypeScript checks pass after building local
workspace declarations.
- Merge follow-up: the profile/candidate startup subset passes 94 tests,
the four Grok source-identity checks pass, and Runner TypeScript
checking passes. A subsequent execution-factory merge passes 21 focused
timing/trace tests and server TypeScript checking.
- Pinned Rust build and the ACPX Rust subset pass. The hostile receipt
projection test passes. Source profile compatibility checks: 127 passed,
one existing skip. Grok identity checks: four passed.
- Broad transport/ACPX suite: 410 passed after building all fake
provider binaries. Earlier attempts failed on missing
Runner/fake-provider outputs; building the required fixtures resolved
those failures.
- The first CI run correctly rejected stale source attestations. After
integrating current master, Grok advances 13→14 and Copilot 84→85;
master’s historical fixtures and Pi 35 stay byte-identical. Provider
dependencies, models and permission defaults are unchanged. Identity
generation/checks now pass.
- The added real-Postgres result lock regression could not initialize
its local embedded cluster. The insertion/replay/failure unit
regressions pass. The current-head CI integration gate passes.
- Comprehensive repository checks and image builds are delegated to CI,
as requested for this instrumentation pass. No deployment, model run,
collector-ingestion proof, or latency improvement is claimed.

## Risks

The new SDK methods are optional. Legacy span calls and endpoint-off
behavior remain supported. The host accepts only closed timing names and
bounded attributes. Remote durations are anchored at local receipt;
transport/scheduling uncertainty remains explicit. Trace trees have
count/depth limits. The first rollout must inspect actual collector
exports before treating the baseline as accepted. Fresh live
qualification for the new source identities remains a rollout
requirement. Revert these commits to remove the new measurements; no
schema migration is required.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and local code/tool
execution. The exact runtime model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the enhancement
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal issue
details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Standard Greptile and Apex are both 5/5, with no unresolved
findings
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-29 17:45:56 -07:00
DottaandPaperclip 9f3fc2094f Fix shared workspace seed capacity and trace deferred copyback (#18181)
Apply the existing company quota to company-shared workspace seed buckets. Preserve individual seed limits and recheck size after verification.

Trace each deferred native workspace finalization check and its resumed copyback under the same hashed run ID. Keep existing admission, ownership and retry behavior.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-29 12:44:34 -05:00
3cd3af5d75 Trace native preparation and OpenCode execution phases (#17918)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Each native run prepares its workspace and skills, then starts its
provider.
> - A long preparation span does not show whether inventory refresh, a
cache lock, repository reads, or file checks caused the delay.
> - One hosted run spent 81 seconds in this phase. The same deployed
service and pinned files took 2.55 seconds in a separate diagnostic
process.
> - Another hosted turn spent 69 seconds before its first output. A
matched public-workspace local model replay took 7.64 seconds.
> - We need skill preparation and provider-internal measurements before
choosing another fix.
> - This pull request adds skill spans, a real-service benchmark, and
opt-in filtered provider timing.
> - The benefit is a faster and more precise diagnosis loop.

## Linked Issues or Issue Description

Refs #17620. That PR owns the batched read optimization. This PR adds
diagnostics and does not change its read policy.
Refs #17907. That PR protects long cache lock transactions from the
database idle timeout.

**What happened?**

Native skill configuration can take much longer than matched standalone
resolution. Provider execution also has long gaps with no internal
timings. Existing spans cannot separate inventory refresh, cache
acquisition, repository reads, context loading, Git snapshots, or model
streaming.

**Expected behavior**

An operator-configured trace shows these phases under the existing run
context. Without an OTLP endpoint, the new spans stay off. A local
benchmark can reproduce the service path with real PostgreSQL, files, an
agent actor, and a responsible user.

**Steps to reproduce**

1. Run `pnpm --filter @paperclipai/server exec tsx
scripts/benchmark-native-skill-preparation.ts --actor=agent
--staging-shape=true`.
2. Inspect cold and warm phase timings, SQL counts, and file
verification results.
3. Set the existing operator OTLP configuration and add
`--export-otlp=true` to inspect the same hierarchy in the collector.

## What Changed

- Add bounded skill inventory, refresh wait, cache acquisition, row,
version, repository, file check, and consumer spans.
- Keep the existing endpoint gate, attribute allowlist, cache locks,
authorization, file validation, and durable pin boundary.
- Add a standalone benchmark with a full repository fixture, immutable
snapshot input, calibrated local transport delay, and optional OTLP
export.
- Verify every delivered file and executable bit outside the timed
section. Create its fixture agent through the lifecycle API.
- Add an opt-in authenticated loopback sink for OpenCode. Discard raw
attributes, logs and content. Retain only closed phase names and
timestamps.
- Carry one bounded diagnostic batch through the real proxy, Rust
normalization, durable journal and server trace projection. Group
provider phases under their execution parent.
- Freeze the Date clock in an existing delegated Slack rate-limit test.
This fixes a CI failure when consecutive requests cross a rate-window
boundary. It does not change rate-limit behavior.
- Document phase boundaries and the difference between SDK startup and
confirmed ingestion.

## Verification

- Skill service, runtime selection, concurrency, cache and timing
suites: 122 tests passed with real PostgreSQL and migrations.
Enabled-span integration tests also passed. They verify parent
relationships and a real advisory-lock timeout.
- A captured 208-file repository produced 209 verified runtime files.
The benchmark uses real agent and responsible-user authorization. The
lifecycle fixture correction passed with the staging-shaped inventory.
- Affected OpenCode driver, collector and transport tests: 72 passed.
The tests cover authentication, content removal, size/count bounds,
disabled behavior and the rebuilt native Runner boundary. One Rust
projection test and 18 server trace tests also passed. A held run-log
sink proves that a full 128-span diagnostic batch adds no duplicate
database writes before terminal consumption.
- The two CI-failing delegated Slack rate-limit cases passed against
real PostgreSQL with only the Date clock frozen.
- Runner TypeScript and server TypeScript checks passed. Rust formatting
and module-boundary checks passed.
- Real Honeycomb readback confirmed the 14-span skill waterfall and a
40-span provider waterfall. Provider phases have the correct execution
parent. The provider export uses synthetic model replies and is
instrumentation proof only.
- Earlier full local checks failed on missing package-local executables,
then on the default Node heap and a missing generated service worker.
The dependency links were repaired. The server check passed with a
larger heap. The full local test attempt could not start embedded
PostgreSQL on this host and was stopped after retaining its failures;
real database coverage uses an isolated database on an existing local
server. These attempts do not count as a full local pass.
- CI on the prior revision passed typecheck, build, Runner and all E2E
shards. It failed the benchmark lifecycle boundary and one existing
rate-window test. Those fixes passed CI. A later Runner assertion
exposed the Copilot source identity pin on the shared transport file.
The diagnostic flag now uses the existing configured-environment
projection, preserving that file byte-for-byte. All 66 selected
installation and native transport tests passed. New-head CI and review
remain required.
- The under-15-second hosted response target remains unmet. This PR adds
diagnostics and does not claim a hosted latency fix.

## Risks

- Skill spans and Honeycomb export retain the operator endpoint gate.
OpenCode phase collection requires a separate explicit diagnostic flag.
Without an endpoint, its closed measurements stay in the instance run
log.
- Raw OpenCode OTLP can contain prompts and authorization attributes.
The new sink rejects unauthenticated requests and discards all
attributes before any measurement reaches PRP or the operator collector.
Request, event and per-phase limits bound memory and journal growth.
- Collection does not wait for a provider flush. Late or oversized
exports can be absent; these measurements are not a complete profiling
guarantee.
- The benchmark's inventory contents and transport delay are explicit
fixtures. They do not reproduce server contention or prove hosted
response time.
- The one durable diagnostic batch remains the local evidence. Its child
span export does not add per-phase run-log rows.
- No database migration, authorization policy, model selection or
inference transport changes. The Rust Runner and OpenCode proxy must be
shipped together for the new opt-in diagnostic.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, shell execution,
browser tools, and Honeycomb queries. The session does not expose a
verifiable backend deployment ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Paperclip <paperclip@paperclip.ing>
2026-10-27 14:53:32 -05:00
DottaandPaperclip 023f4a5d0c fix: preserve repository cache negotiation refs (#16687)
## Thinking Path

> - Paperclip manages agent work and its execution environments.
> - A fresh repository task needs an independent checkout before its
runner can start.
> - The object cache from #16487 should reduce repeated network
transfer.
> - The cache kept Git objects but dropped remote-tracking refs during
its bare clone.
> - Git needs those tips to negotiate which objects it already has.
> - This pull request preserves the tips in a versioned mirror cache and
records explicit cache outcomes.
> - The benefit is less repeated transfer and clearer evidence when a
cache does not help.

## Linked Issues or Issue Description

Refs #16487 and #16454. No duplicate cache-negotiation fix was found in
the public PR search.

**What happened?**

A valid object cache could make a fresh clone slower. A controlled
staging probe took 45.87 seconds and received 184.39 MB even though the
reference already held the objects. An earlier probe took 52.33 seconds.
An uncached clone took 20.94 seconds.

`git clone --bare` copied the pristine checkout's objects, local
branches, and tags, but omitted its remote-tracking refs. The source had
about 1,200 remote branches. Git could not use the missing tips to
negotiate those objects.

**Expected behavior**

Keep the pristine reference's negotiation tips. Each task must still
authenticate online, use the current remote refs, and end with
independent Git objects.

**Steps to reproduce**

1. Clone a repository with many remote branches into a fresh checkout.
2. Publish a reference with `git clone --bare --no-hardlinks` from that
checkout.
3. Clone the remote with `--reference-if-able` and `--dissociate`.
4. Compare with a reference that retains the remote-tracking tips. Use
Git Trace2 and count received pack bytes.

**Paperclip version or commit**

The staging runtime was `27ab902d3db2770ada79c9f42a9c06e1d346f34b`, with
Git 2.47.3. The cache implementation was unchanged from #16487.

**Deployment mode**

Docker on a persistent application volume. The probe used the
application file owner and the public Paperclip repository. It did not
start agent tasks or alter the application cache.

## What Changed

- Publish pristine object caches with `--mirror` to retain
remote-tracking tips.
- Put cache format version 2 in the key and receipt. The first checkout
builds a new generation. Existing generations stay under the current
storage limits.
- Keep the existing online authentication, current remote refs,
credential and company isolation, independent objects, quotas, and
fallback behavior.
- Add closed Honeycomb cache outcomes and a boolean that states whether
a reference was supplied to clone.
- Document that a lookup hit or supplied reference does not prove Git
avoided transfer.

## Verification

- Controlled staging component experiment: a disposable reference copy
held exactly the same 210,057 objects as the original. It added 1,187
advertised branch tips whose commits were already in those objects.
- Before: 45.87 seconds and 184.39 MB received, one sample in this
comparison.
- After: 12.20 and 14.58 seconds; median 13.39 seconds. Received pack
sizes were 1.99 and 1.95 MB.
- All comparison clones used the same current remote HEAD and ended
without an alternates file.
- The remaining repack took 9.72–9.81 seconds. Checkout file writing
took 0.86–0.96 seconds.
- The staging experiment restored tips on a disposable cache copy. It
did not deploy this patch or measure complete agent latency.
- Confirmed both staging scratch locations were empty afterward. The
original cache timestamps stayed unchanged. The temporary operator SSH
key was revoked and its local files removed.
- TypeScript 5.9.3 transpilation reported zero syntax diagnostics across
the four changed TypeScript files. `git diff --check` passed.
- The new negotiation regression fails with the old `--bare` publisher
and passes with the mirror publisher.
- Focused Vitest suites: 26 tests passed across repository-object-cache
and execution-preparation-timing. These cover independent objects,
current refs, unavailable/revoked remotes, scope separation, invalid
storage, tracing isolation, and endpoint-off behavior.
- The first full focused run passed 24 tests but failed its existing
localhost HTTP fixture because the sandbox denied `listen` on
`127.0.0.1`. The rerun with local loopback permission passed all 26
tests.
- Server semantic typechecking passed with `tsc -p server/tsconfig.json
--noEmit` after building the required workspace declarations. Initial
attempts lacked generated dependency outputs; the completed dependency
build and rerun passed.
- A full application build was not run locally. CI passed: 55 checks
passed and two optional checks were skipped. Greptile is 5/5 on the
current head, with both findings resolved.

## Risks

- Version 2 causes one cold cache rebuild. Old cache generations retain
their disk usage, and the existing quota can prevent publication.
- A cache still needs a full repack to produce an independent task
checkout. This PR does not remove that cost.
- The component experiment establishes the negotiation effect on
staging. It does not establish an end-to-end agent speedup or the
under-30-second goal.
- The new trace fields remain behind the existing OTLP gate. Their
values are closed enums or a boolean. They export no refs, paths, URLs,
cache keys, credentials, or error text.

## Model Used

OpenAI Codex, GPT-6 family, with code and tool use. The exact serving
model ID and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-18 21:28:20 -05:00
DottaandPaperclip 0e58f29eb7 fix(observability): trace pre-dispatch controller waits (#16515)
Measure usage startup, MCP handoff, dispatch admission, and budget reservation with the existing endpoint-gated tracer. Preserve outer attempt context across asynchronous provider dispatch.

Validated with 135 focused tests, full typecheck/build, fresh full CI/E2E, Superagent, and Apex 5/5 on 9b28865bb3.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-15 20:30:05 -05:00
DottaandPaperclip ebceb2a744 trace(server): time workspace, native setup and startup recovery (#16330)
## Thinking Path

> - Paperclip manages agents and their work.
> - The controller prepares workspaces before it starts a Runner.
> - A slow run can spend 30–80 seconds inside the combined workspace phase.
> - Startup recovery can also delay queued work for minutes.
> - Native runs also have gaps before Runner startup and after the provider result.
> - Existing traces do not identify the awaited steps inside those intervals.
> - This change adds closed timing boundaries around the existing operations.
> - Operators can locate the wait without changing execution or exposing content.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Operator-configured OpenTelemetry diagnosis of controller preparation and startup recovery. Related: #16178, #16236 and #16285 introduced the existing preparation timing and sampling contract. #16306 and #16312 reduce separate configuration and copyback work.

**Current behavior**

The workspace interval combines source resolution, trust checks, provisioning, database persistence, binding and repository admission. Startup awaits accounting and native recovery before queued work resumes, but neither has a separate duration. Native backend setup and post-turn cleanup also contain unmeasured awaits. CPU profiles and a fast database probe cannot identify which await is slow.

**Proposed behavior**

Record workspace steps and native controller setup and cleanup with the existing hashed run ID. Record startup phases as independent, operator-sampled root spans without custom attributes. Keep the same operation order, results, failures and authorization.

**Reason and benefit**

These boundaries distinguish source work, database writes, repository admission and restart recovery. This is a diagnosis change. It does not claim a response-time improvement or a solved latency budget.

## What Changed

- Add 14 closed workspace step names around 16 existing awaits. Repeated updates and bindings produce separate spans.
- Time startup tool reviews, gateway sessions, delivery-worker readiness, accounting, plugin readiness, native recovery, hot restart, orphan reaping, retries, queued work and goal recovery.
- Time native GitHub setup, wake binding, backend construction, connector assignments, MCP discovery, tool authority, companion selection and binary resolution. Time selected post-turn GitHub, coordinator and instruction cleanup awaits.
- Move preparation timing to a pure shared module. Keep the existing heartbeat import as an alias. Both heartbeat and native timing skip tracer lookup when the endpoint is unset or blank, including when another library registered a provider.
- Tracing lookup, callback, status and end faults cannot repeat work or replace its value or error.
- Add real PostgreSQL workspace timing coverage and silent no-op/fault controls. Update the tracing contract and paired run maps.
- Keep span attributes closed. Startup spans add none; workspace steps use only the existing run hash. No commands, paths, returned records or exception messages are recorded.

## Verification

- Diagnostic-gap baseline: the new real workspace test fails against the unchanged workspace implementation because the expected step spans are absent. It passes with the wrappers.
- Final affected coverage: **804 distinct cases pass** across seven complete suites: 698 native-executor cases, 39 real PostgreSQL workspace cases, 61 timing/context/startup cases, and six execution-configuration integration cases. The final native and unit invocations pass with no selected skips. The workspace suite passed in the initial six-file invocation on the same final workspace source.
- Preserve the initial six-file attempt as failed: **7 failed and 789 passed**. All seven errors occurred during embedded PostgreSQL startup. The log reports local macOS shared-memory exhaustion. A fresh 15-case database group passes, followed by the complete 698-case native suite. These runs overlap; do not add their counts.
- The new native timing assertion fails against unchanged native code because the expected spans are absent. It passes with the wrappers and retains the credential-isolation assertions.
- Full `pnpm -r typecheck` and `pnpm build`: passed on the final production source. Earlier nullable-closure errors are fixed with erased assertions inside existing guards. The failed attempts remain recorded.
- Normalized AST comparison: removing only timing wrappers, timing imports/local handles and erased assertions restores the original workspace, startup and native executor programs. This checks call order and policy expressions; it is not live acceptance.
- The expanded head first failed two configuration integration assertions because their test providers had no endpoint. Configure the endpoint explicitly in those two tracing tests, and restore it after each test. All six cases then pass with the original credential cleanup, redaction and exact span assertions. Production is unchanged by this repair.
- The first PR head failed one unchanged attachment test at its 15-second deadline in CI. The complete 32-case attachment suite passes locally with that deadline. Do not label the failed CI attempt green. The expanded head needs fresh full CI.
- `node scripts/check-maps.mjs`: passed, 35 areas and 176 capability IDs. `node scripts/storybook-catalog.mjs`: 283 files and 2,728 stories. Structural checks only; no browser, play or Storybook build claim.
- `git diff --check`: passed. No added prose comments, lockfile changes or new 1,000-line crossings.
- A duplicate full local test run was not started. Fresh exact-head full CI and E2E pass on `15a683bb2419960f611584f8c07cdcfc9ce3af0b`. All 56 check runs are successful or optional skipped. Superagent and legacy Snyk pass. Completed Apex review is 5/5 on this same head, with zero unresolved review threads.

Commands:

```sh
pnpm exec vitest run --project @paperclipai/server --maxWorkers=1 --fileParallelism=false server/src/services/native-runtime/native-session-executor.test.ts server/src/services/execution-preparation-timing.test.ts server/src/startup-recovery-timing.test.ts server/src/services/heartbeat/preparation-timing.test.ts server/src/services/heartbeat/preparation-timing-context.test.ts server/src/services/heartbeat/workspace-preparation.test.ts
pnpm -r typecheck
pnpm build
node scripts/check-maps.mjs
node scripts/storybook-catalog.mjs
```

## Risks

- Configured operators receive more spans. They still follow the configured sampler and exporter.
- Startup spans measure the awaited phase, including remaining readiness waits. They do not measure earlier worker construction or a provider turn.
- Workspace and native setup intervals overlap their containing phases. Do not add them to the parent duration. Native timing covers named awaits, not every statement or every error path.
- These spans are not yet deployed or tested against a live latency run. No first-party Telemetry event, run-log write, schema or API changes are included.

## Model Used

OpenAI GPT-6 through Codex, with tool use and code execution. The exact served variant and context-window size are not exposed to this task.

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before requesting merge

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-14 17:16:07 -05:00