mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-12 06:10:44 +01:00
DottaandClaude Opus 5.5
b32ef9f111
## Thinking Path - The native speed budgets (plan invariant 8) judge how fast an agent responds. They judged `task.time_to_first_token`: time to the first deliverable **text**. - In the clean local restart storm (w-20261108T153612-s9401), Claude breached its TTFT budget (cold p50/p95 17.4/34.7 s against 16.5/30). The per-run spans showed this was not a control-plane wait: in all 49 cold runs (and 76% of warm), Claude started a tool call before writing any text. Its first visible activity came at 7.4 s p50 from run creation; first text came at 19.6 s. Codex never calls a tool first. - So first text charges model tool work as latency for tool-first adapters. What the user experiences as "the agent responded" is the first visible activity: a text delta, a tool-call start, or a thinking indicator. - The coordinator decided: budgets judge time to first visible agent activity from run creation, with the budget numbers unchanged; first text stays a reported metric with no budget for tool-first adapters. ## Linked Issues or Issue Description **What happened?** Local storm s9401: the Claude TTFT budget breached on every aggregate, while the control-plane steps (queue 0.05 s, prepare 0.3 s, runner startup ~4 s, first provider event 1.3–1.8 s) were well within budget. The rest of the first-text wait was Claude streaming tool input (about 5.5 s per call) and running tools before it wrote text. **Steps to reproduce** 1. Run a Claude task whose first action is a tool call (most real tasks). 2. Read the run log: `provider.time_to_first_agent_event` ends at the tool call; `task.time_to_first_token` ends many seconds later at the first text. 3. The speed budget judges the later one and reports a breach. **Expected behavior** The budget judges the first visible agent activity from run creation (`task.time_to_first_agent_activity`). Time to first text is still recorded and reported. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: when the first meaningful agent event commits (the existing `firstMeaningfulAgentEventKind`: text delta, tool-call start, or reasoning), record `task.time_to_first_agent_activity`. It uses the same detached anchor resolution as TTFT (accepted comment, answered question, durable wake request, or run creation), so it adds nothing to the turn path. - `server/src/services/native-runtime/native-speed-budgets.ts`: the metric is now named `time_to_first_activity` (was `ttft`), with identical budget numbers. The type comment documents the definition and why first text is unbudgeted. - `server/src/services/native-runtime/native-run-trace.ts`: the budget check judges `task.time_to_first_agent_activity` from run creation, keeping the existing input-wait handling. `task.time_to_first_token` is no longer budgeted. - `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`: report first activity from run creation (the budgeted value) next to TTFT. - Docs: `doc/observability.md` (new span) and `doc/plans/2026-11-07-ttft-baseline.md` (metric table). - Tests: - A tool-first turn records first activity at the tool call (5 s) while TTFT records first text (16 s). - A tool-first trace with slow first text but fast first activity is within budget. - The existing budget tests now judge the activity span. Observability path (AGENTS.md section 5.7): the new run-log span uses only the existing `eventKind` and `anchor` attributes. Its name survives run-log redaction intact; checked with `redactEventPayload`. The operator OTLP gate is unchanged. It is not Telemetry. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts`: 44/44 pass. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "first visible activity|first assistant text once|first-token timing"`: 7/7 pass. - `npx vitest run --root tests/restart-chaos`: 159/159 pass; `tsc` on `tests/restart-chaos` is clean. ## Risks - **A metric rename.** Consumers of `run.performance.budget_exceeded` see `metric: "time_to_first_activity"` instead of `"ttft"`. In-repo there are none besides the tests; Phase 3's chaos speed grade switches to the new span. - **A looser budget for slow first text.** Intentional: first text is reported, not judged. A regression in control-plane latency still shows up in first activity. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool use and code execution, via Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>