Commit Graph
3 Commits
Author SHA1 Message Date
DottaandPaperclip 023f4a5d0c fix: preserve repository cache negotiation refs (#16687)
## Thinking Path

> - Paperclip manages agent work and its execution environments.
> - A fresh repository task needs an independent checkout before its
runner can start.
> - The object cache from #16487 should reduce repeated network
transfer.
> - The cache kept Git objects but dropped remote-tracking refs during
its bare clone.
> - Git needs those tips to negotiate which objects it already has.
> - This pull request preserves the tips in a versioned mirror cache and
records explicit cache outcomes.
> - The benefit is less repeated transfer and clearer evidence when a
cache does not help.

## Linked Issues or Issue Description

Refs #16487 and #16454. No duplicate cache-negotiation fix was found in
the public PR search.

**What happened?**

A valid object cache could make a fresh clone slower. A controlled
staging probe took 45.87 seconds and received 184.39 MB even though the
reference already held the objects. An earlier probe took 52.33 seconds.
An uncached clone took 20.94 seconds.

`git clone --bare` copied the pristine checkout's objects, local
branches, and tags, but omitted its remote-tracking refs. The source had
about 1,200 remote branches. Git could not use the missing tips to
negotiate those objects.

**Expected behavior**

Keep the pristine reference's negotiation tips. Each task must still
authenticate online, use the current remote refs, and end with
independent Git objects.

**Steps to reproduce**

1. Clone a repository with many remote branches into a fresh checkout.
2. Publish a reference with `git clone --bare --no-hardlinks` from that
checkout.
3. Clone the remote with `--reference-if-able` and `--dissociate`.
4. Compare with a reference that retains the remote-tracking tips. Use
Git Trace2 and count received pack bytes.

**Paperclip version or commit**

The staging runtime was `27ab902d3db2770ada79c9f42a9c06e1d346f34b`, with
Git 2.47.3. The cache implementation was unchanged from #16487.

**Deployment mode**

Docker on a persistent application volume. The probe used the
application file owner and the public Paperclip repository. It did not
start agent tasks or alter the application cache.

## What Changed

- Publish pristine object caches with `--mirror` to retain
remote-tracking tips.
- Put cache format version 2 in the key and receipt. The first checkout
builds a new generation. Existing generations stay under the current
storage limits.
- Keep the existing online authentication, current remote refs,
credential and company isolation, independent objects, quotas, and
fallback behavior.
- Add closed Honeycomb cache outcomes and a boolean that states whether
a reference was supplied to clone.
- Document that a lookup hit or supplied reference does not prove Git
avoided transfer.

## Verification

- Controlled staging component experiment: a disposable reference copy
held exactly the same 210,057 objects as the original. It added 1,187
advertised branch tips whose commits were already in those objects.
- Before: 45.87 seconds and 184.39 MB received, one sample in this
comparison.
- After: 12.20 and 14.58 seconds; median 13.39 seconds. Received pack
sizes were 1.99 and 1.95 MB.
- All comparison clones used the same current remote HEAD and ended
without an alternates file.
- The remaining repack took 9.72–9.81 seconds. Checkout file writing
took 0.86–0.96 seconds.
- The staging experiment restored tips on a disposable cache copy. It
did not deploy this patch or measure complete agent latency.
- Confirmed both staging scratch locations were empty afterward. The
original cache timestamps stayed unchanged. The temporary operator SSH
key was revoked and its local files removed.
- TypeScript 5.9.3 transpilation reported zero syntax diagnostics across
the four changed TypeScript files. `git diff --check` passed.
- The new negotiation regression fails with the old `--bare` publisher
and passes with the mirror publisher.
- Focused Vitest suites: 26 tests passed across repository-object-cache
and execution-preparation-timing. These cover independent objects,
current refs, unavailable/revoked remotes, scope separation, invalid
storage, tracing isolation, and endpoint-off behavior.
- The first full focused run passed 24 tests but failed its existing
localhost HTTP fixture because the sandbox denied `listen` on
`127.0.0.1`. The rerun with local loopback permission passed all 26
tests.
- Server semantic typechecking passed with `tsc -p server/tsconfig.json
--noEmit` after building the required workspace declarations. Initial
attempts lacked generated dependency outputs; the completed dependency
build and rerun passed.
- A full application build was not run locally. CI passed: 55 checks
passed and two optional checks were skipped. Greptile is 5/5 on the
current head, with both findings resolved.

## Risks

- Version 2 causes one cold cache rebuild. Old cache generations retain
their disk usage, and the existing quota can prevent publication.
- A cache still needs a full repack to produce an independent task
checkout. This PR does not remove that cost.
- The component experiment establishes the negotiation effect on
staging. It does not establish an end-to-end agent speedup or the
under-30-second goal.
- The new trace fields remain behind the existing OTLP gate. Their
values are closed enums or a boolean. They export no refs, paths, URLs,
cache keys, credentials, or error text.

## Model Used

OpenAI Codex, GPT-6 family, with code and tool use. The exact serving
model ID and context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-18 21:28:20 -05:00
DottaandPaperclip 0e58f29eb7 fix(observability): trace pre-dispatch controller waits (#16515)
Measure usage startup, MCP handoff, dispatch admission, and budget reservation with the existing endpoint-gated tracer. Preserve outer attempt context across asynchronous provider dispatch.

Validated with 135 focused tests, full typecheck/build, fresh full CI/E2E, Superagent, and Apex 5/5 on 9b28865bb3.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-15 20:30:05 -05:00
DottaandPaperclip ebceb2a744 trace(server): time workspace, native setup and startup recovery (#16330)
## Thinking Path

> - Paperclip manages agents and their work.
> - The controller prepares workspaces before it starts a Runner.
> - A slow run can spend 30–80 seconds inside the combined workspace phase.
> - Startup recovery can also delay queued work for minutes.
> - Native runs also have gaps before Runner startup and after the provider result.
> - Existing traces do not identify the awaited steps inside those intervals.
> - This change adds closed timing boundaries around the existing operations.
> - Operators can locate the wait without changing execution or exposing content.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Operator-configured OpenTelemetry diagnosis of controller preparation and startup recovery. Related: #16178, #16236 and #16285 introduced the existing preparation timing and sampling contract. #16306 and #16312 reduce separate configuration and copyback work.

**Current behavior**

The workspace interval combines source resolution, trust checks, provisioning, database persistence, binding and repository admission. Startup awaits accounting and native recovery before queued work resumes, but neither has a separate duration. Native backend setup and post-turn cleanup also contain unmeasured awaits. CPU profiles and a fast database probe cannot identify which await is slow.

**Proposed behavior**

Record workspace steps and native controller setup and cleanup with the existing hashed run ID. Record startup phases as independent, operator-sampled root spans without custom attributes. Keep the same operation order, results, failures and authorization.

**Reason and benefit**

These boundaries distinguish source work, database writes, repository admission and restart recovery. This is a diagnosis change. It does not claim a response-time improvement or a solved latency budget.

## What Changed

- Add 14 closed workspace step names around 16 existing awaits. Repeated updates and bindings produce separate spans.
- Time startup tool reviews, gateway sessions, delivery-worker readiness, accounting, plugin readiness, native recovery, hot restart, orphan reaping, retries, queued work and goal recovery.
- Time native GitHub setup, wake binding, backend construction, connector assignments, MCP discovery, tool authority, companion selection and binary resolution. Time selected post-turn GitHub, coordinator and instruction cleanup awaits.
- Move preparation timing to a pure shared module. Keep the existing heartbeat import as an alias. Both heartbeat and native timing skip tracer lookup when the endpoint is unset or blank, including when another library registered a provider.
- Tracing lookup, callback, status and end faults cannot repeat work or replace its value or error.
- Add real PostgreSQL workspace timing coverage and silent no-op/fault controls. Update the tracing contract and paired run maps.
- Keep span attributes closed. Startup spans add none; workspace steps use only the existing run hash. No commands, paths, returned records or exception messages are recorded.

## Verification

- Diagnostic-gap baseline: the new real workspace test fails against the unchanged workspace implementation because the expected step spans are absent. It passes with the wrappers.
- Final affected coverage: **804 distinct cases pass** across seven complete suites: 698 native-executor cases, 39 real PostgreSQL workspace cases, 61 timing/context/startup cases, and six execution-configuration integration cases. The final native and unit invocations pass with no selected skips. The workspace suite passed in the initial six-file invocation on the same final workspace source.
- Preserve the initial six-file attempt as failed: **7 failed and 789 passed**. All seven errors occurred during embedded PostgreSQL startup. The log reports local macOS shared-memory exhaustion. A fresh 15-case database group passes, followed by the complete 698-case native suite. These runs overlap; do not add their counts.
- The new native timing assertion fails against unchanged native code because the expected spans are absent. It passes with the wrappers and retains the credential-isolation assertions.
- Full `pnpm -r typecheck` and `pnpm build`: passed on the final production source. Earlier nullable-closure errors are fixed with erased assertions inside existing guards. The failed attempts remain recorded.
- Normalized AST comparison: removing only timing wrappers, timing imports/local handles and erased assertions restores the original workspace, startup and native executor programs. This checks call order and policy expressions; it is not live acceptance.
- The expanded head first failed two configuration integration assertions because their test providers had no endpoint. Configure the endpoint explicitly in those two tracing tests, and restore it after each test. All six cases then pass with the original credential cleanup, redaction and exact span assertions. Production is unchanged by this repair.
- The first PR head failed one unchanged attachment test at its 15-second deadline in CI. The complete 32-case attachment suite passes locally with that deadline. Do not label the failed CI attempt green. The expanded head needs fresh full CI.
- `node scripts/check-maps.mjs`: passed, 35 areas and 176 capability IDs. `node scripts/storybook-catalog.mjs`: 283 files and 2,728 stories. Structural checks only; no browser, play or Storybook build claim.
- `git diff --check`: passed. No added prose comments, lockfile changes or new 1,000-line crossings.
- A duplicate full local test run was not started. Fresh exact-head full CI and E2E pass on `15a683bb2419960f611584f8c07cdcfc9ce3af0b`. All 56 check runs are successful or optional skipped. Superagent and legacy Snyk pass. Completed Apex review is 5/5 on this same head, with zero unresolved review threads.

Commands:

```sh
pnpm exec vitest run --project @paperclipai/server --maxWorkers=1 --fileParallelism=false server/src/services/native-runtime/native-session-executor.test.ts server/src/services/execution-preparation-timing.test.ts server/src/startup-recovery-timing.test.ts server/src/services/heartbeat/preparation-timing.test.ts server/src/services/heartbeat/preparation-timing-context.test.ts server/src/services/heartbeat/workspace-preparation.test.ts
pnpm -r typecheck
pnpm build
node scripts/check-maps.mjs
node scripts/storybook-catalog.mjs
```

## Risks

- Configured operators receive more spans. They still follow the configured sampler and exporter.
- Startup spans measure the awaited phase, including remaining readiness waits. They do not measure earlier worker construction or a provider turn.
- Workspace and native setup intervals overlap their containing phases. Do not add them to the parent duration. Native timing covers named awaits, not every statement or every error path.
- These spans are not yet deployed or tested against a live latency run. No first-party Telemetry event, run-log write, schema or API changes are included.

## Model Used

OpenAI GPT-6 through Codex, with tool use and code execution. The exact served variant and context-window size are not exposed to this task.

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before requesting merge

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-14 17:16:07 -05:00