Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-12 22:56:58 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,128 Commits 1,922 Branches 2,445 Tags
fe72bc5f81dd3bf2722daf5bf714aa8d451b8374
Commit Graph
4 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 6fd46c0025 perf(budgets): judge Paperclip's share of restart recovery; report the provider's session load (#20183) 
## Thinking Path

- Speed budgets protect the time to first visible agent activity, and
restart recovery is one of the judged metrics
(`native-speed-budgets.ts`, wave grader).
- In the B1 concurrency-6 chaos wave (`w-20261109T045320-s9807`), Codex
`restart_recovery` was p50 8.6 s and p95 11.2 s, against a budget of 6.5
s and 8.0 s. At concurrency 4 (K13) it was 3.8 s and 5.0 s.
- The preparation spans on a copy of the kept database break it down:
- Paperclip's share (ready → claim → prepare → runtime stage → runner
launch) is a steady 1.2–2.0 s.
- `provider.session.load`, the Codex app-server boot plus
`thread/resume` of long saved sessions, takes 3.0–6.3 s, 65–75% of each
recovery.
- Resumes run concurrently, and the worktree lock and the startup
reconciler are not on the path.
- Judging the full latency therefore mixes our code with provider and
session-size effects. Budget the part we control, so a regression in our
code shows up regardless of session size or concurrency, and report the
rest.

## Linked Issues or Issue Description

**What happened?**
`restart_recovery` breached its budget whenever many long provider
sessions resumed at once, although Paperclip's own recovery work stayed
within about 2 s. The breach did not point at anything Paperclip could
fix, and a real regression in our code could hide inside the
provider-load variance.

**Steps to reproduce**
1. Run a local chaos wave at concurrency 6 with long-running Codex tasks
and K1/K2/K4 faults (for example B1, seed 9807).
2. Compare `restart_recovery` in `speed.json` with
`provider.session.load` in the recovered runs' run logs.

**Expected behavior**
The budget judges Paperclip's share of a recovery. The provider's
session load and the full latency are reported but not judged.

## What Changed

- `server/src/services/native-runtime/native-speed-budgets.ts`:
- New budgeted metric `restart_recovery_control_plane`: from `ready` to
the start of `provider.session.load`, or to the next provider event when
nothing reloads (for example a reattached runner).
- Its budget is 2.5 s / 4.0 s for claude, codex and opencode. Measured:
1.2–2.0 s per run at concurrency 6.
  - The full `restart_recovery` stays in the table for reporting.
  - Breach payloads carry `restartTotalMs` and `providerLoadMs`.
- `server/src/services/native-runtime/native-run-trace.ts`:
- The per-run check records when `provider.session.load` starts, keeps
the recovery measurement, and judges it when the run finishes.
- It judges after the deferred diagnostics drain, so a provider span
committed after the provider event still counts.
  - It emits `restart_recovery_control_plane` breaches only.
- `tests/restart-chaos/ttft-baseline-stats.ts`: each recovery sample
gains `controlPlaneMs` and `providerLoadMs`. The provider-load start is
the span event's write time minus its duration.
- `tests/restart-chaos/wave-speed.ts`:
- Three restart rows per adapter: `restart_recovery` (report-only),
`restart_recovery_control_plane` (budgeted) and `restart_provider_load`
(report-only).
  - All three have outage time removed.
- `tests/restart-chaos/ttft-baseline.ts`: the recovery table adds a
"Paperclip share" column.
- Docs: `doc/run-log-events.md` (metric and payload) and the
speed-budget table in
`doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`.

## Verification

- `server`: `npx vitest run
src/services/native-runtime/native-speed-budgets.test.ts`, 18 passed.
Two new tests:
  - a recovery dominated by provider load does not breach;
- a slow Paperclip share breaches with `restartTotalMs` and
`providerLoadMs`, even when the provider span is committed after the
provider event.

The existing recovery tests moved to the new metric. All six were red
before the trace change.
- `tests/restart-chaos`: `npx vitest run --config
tests/restart-chaos/vitest.config.ts wave-speed.test.ts
ttft-baseline-stats.test.ts`, 29 passed. New tests cover the sample
split and the grader rows (a judged share, a report-only provider load,
zero breaches).
- Server typecheck: no errors in the touched files.

## Risks

- **Lower sensitivity to provider-side slowdowns.** A provider
regression in session load no longer breaches. It stays visible in the
report-only rows and in breach payloads.
- **Runs with no `provider.session.load` span** (reattach, or an adapter
that doesn't emit it) are judged on the full latency against the
control-plane budget. That's conservative.
- **Run-log consumers that filter on `metric: "restart_recovery"`** see
`restart_recovery_control_plane` from now on. The doc notes the change
date.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-09 01:22:28 -06:00
DottaandClaude Opus 5.5 b32ef9f111 perf(budgets): judge time to first visible agent activity; keep time to first text as a reported metric (#20077) 
## Thinking Path

- The native speed budgets (plan invariant 8) judge how fast an agent
responds. They judged `task.time_to_first_token`: time to the first
deliverable **text**.
- In the clean local restart storm (w-20261108T153612-s9401), Claude
breached its TTFT budget (cold p50/p95 17.4/34.7 s against 16.5/30). The
per-run spans showed this was not a control-plane wait: in all 49 cold
runs (and 76% of warm), Claude started a tool call before writing any
text. Its first visible activity came at 7.4 s p50 from run creation;
first text came at 19.6 s. Codex never calls a tool first.
- So first text charges model tool work as latency for tool-first
adapters. What the user experiences as "the agent responded" is the
first visible activity: a text delta, a tool-call start, or a thinking
indicator.
- The coordinator decided: budgets judge time to first visible agent
activity from run creation, with the budget numbers unchanged; first
text stays a reported metric with no budget for tool-first adapters.

## Linked Issues or Issue Description

**What happened?**
Local storm s9401: the Claude TTFT budget breached on every aggregate,
while the control-plane steps (queue 0.05 s, prepare 0.3 s, runner
startup ~4 s, first provider event 1.3–1.8 s) were well within budget.
The rest of the first-text wait was Claude streaming tool input (about
5.5 s per call) and running tools before it wrote text.

**Steps to reproduce**
1. Run a Claude task whose first action is a tool call (most real
tasks).
2. Read the run log: `provider.time_to_first_agent_event` ends at the
tool call; `task.time_to_first_token` ends many seconds later at the
first text.
3. The speed budget judges the later one and reports a breach.

**Expected behavior**
The budget judges the first visible agent activity from run creation
(`task.time_to_first_agent_activity`). Time to first text is still
recorded and reported.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: when
the first meaningful agent event commits (the existing
`firstMeaningfulAgentEventKind`: text delta, tool-call start, or
reasoning), record `task.time_to_first_agent_activity`. It uses the same
detached anchor resolution as TTFT (accepted comment, answered question,
durable wake request, or run creation), so it adds nothing to the turn
path.
- `server/src/services/native-runtime/native-speed-budgets.ts`: the
metric is now named `time_to_first_activity` (was `ttft`), with
identical budget numbers. The type comment documents the definition and
why first text is unbudgeted.
- `server/src/services/native-runtime/native-run-trace.ts`: the budget
check judges `task.time_to_first_agent_activity` from run creation,
keeping the existing input-wait handling. `task.time_to_first_token` is
no longer budgeted.
- `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`:
report first activity from run creation (the budgeted value) next to
TTFT.
- Docs: `doc/observability.md` (new span) and
`doc/plans/2026-11-07-ttft-baseline.md` (metric table).
- Tests:
- A tool-first turn records first activity at the tool call (5 s) while
TTFT records first text (16 s).
- A tool-first trace with slow first text but fast first activity is
within budget.
  - The existing budget tests now judge the activity span.

Observability path (AGENTS.md section 5.7): the new run-log span uses
only the existing `eventKind` and `anchor` attributes. Its name survives
run-log redaction intact; checked with `redactEventPayload`. The
operator OTLP gate is unchanged. It is not Telemetry.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-speed-budgets.test.ts
src/services/native-runtime/native-run-trace.test.ts`: 44/44 pass.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "first
visible activity|first assistant text once|first-token timing"`: 7/7
pass.
- `npx vitest run --root tests/restart-chaos`: 159/159 pass; `tsc` on
`tests/restart-chaos` is clean.

## Risks

- **A metric rename.** Consumers of `run.performance.budget_exceeded`
see `metric: "time_to_first_activity"` instead of `"ttft"`. In-repo
there are none besides the tests; Phase 3's chaos speed grade switches
to the new span.
- **A looser budget for slow first text.** Intentional: first text is
reported, not judged. A regression in control-plane latency still shows
up in first activity.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-08 12:32:42 -06:00
DottaandClaude Opus 5.5 f35eb1e5ef fix(observability): judge the TTFT budget from run creation; report the wait behind a live turn (#19941) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Invariant 8 of the durable-ownership plan says turns must start
fast. #19863 added per-run speed budgets that write
`run.performance.budget_exceeded` to the run log.
> - The Phase 2 chaos data (#19927) showed follow-up TTFT about 3x over
the claude budget.
> - The cause is measurement scope. A follow-up comment posted during a
live turn waits for that turn to end, and TTFT is anchored at the
comment. The budgets came from idle follow-ups.
> - This pull request judges the TTFT budget from run creation and
reports the wait separately.
> - The benefit is that budget breaches point at real start-up slowness,
not at time spent behind a turn the user is already watching.

## Linked Issues or Issue Description

Refs #19863 (speed budgets), #19927 (Phase 2 data), #19852 (startup
sub-spans), #19888 (warm default).

**What happened?**
In two local storm waves, C1 follow-up runs breached the claude warm
TTFT budget (p50 27.6–29.9 s against 9.5 s). Their spans show
`comment.to_run_created` at 6–62 s (waiting for the live turn), then
session resume at 1.4–3.8 s and submit-to-first-event at 2–5 s. Codex
without the wait was about 4.8 s, the same as the warm-follow-up
baseline.

**What did you expect?**
The budget should judge start-up latency, which is what its baseline
measured. The wait behind a live turn should be visible but not judged.

**Steps to reproduce**
1. Run a local chaos storm with C1 tasks.
2. Read `run.performance.budget_exceeded` rows with metric `ttft` for
`issue_commented` runs, and compare them with each run's
`comment.to_run_created` span.

## What Changed

- `native-run-trace.ts`: the TTFT budget check measures from run
creation (start of `heartbeat.queue`, or end of the ingress span). The
TTFT span itself does not change.
- `native-speed-budgets.ts`: breaches can carry `inputWaitMs` (TTFT
only).
- `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`:
the wait behind a live turn is its own measured step (`inputWait`), next
to the full TTFT and the TTFT from run creation. The TTFT table gains a
"Wait behind live turn" column. The full comment-to-first-token time
stays the user-visible metric.
- `doc/run-log-events.md` and the plan's speed-budget section describe
the rule.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-speed-budgets.test.ts
src/services/native-runtime/native-run-trace.test.ts`: 43 pass, 2 new.
`pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/ttft-baseline-stats.test.ts`: 12 pass, 1 new. They
cover a follow-up under budget after a 60 s wait (no breach), a
follow-up over budget (breach with `inputWaitMs`), and queueing after
run creation still judged.
- Server typecheck passes.

## Risks

- Low. This is a run-log-only change (AGENTS.md §5.7): no Telemetry and
no span attributes change.
- A slow wait before run creation is no longer a TTFT budget breach. The
TTFT span still records it, and the breach payload names it. Admission
holds after run creation are still judged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 20:27:41 -06:00
DottaandClaude Opus 5.5 99529f9d71 feat(observability): local speed budgets and a run-log budget check (#19863) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When someone messages an agent, the reply must start as fast as
possible, and restart-safety work must not slow it
> - The durable ownership plan makes this invariant 8 ("fast turns") and
asks Phase 1 to turn the Phase 0 baseline into p50/p95 budgets per
adapter
> - Without budgets, a regression in time to first token, agent boot or
restart recovery shows up only when someone reruns the baseline by hand
> - This pull request records the budgets and adds a per-run check that
writes a run-log row when a run exceeds its p95 budget
> - The benefit is that every later phase can show "no regression
against the budgets" from the run log, without new waits on the turn
path

## Linked Issues or Issue Description

Refs #19831 (plan, "Performance constraint", rule 4, and invariant 8).
Refs #19846 (the Phase 0 baseline these budgets come from). Stacked on
#19861.

**Problem**
The plan requires p50/p95 budgets per adapter for TTFT, agent boot and
restart recovery latency, and a check against them. Neither exists.

**Expected behavior**
Budgets are recorded in the plan and in code. A run that exceeds a p95
budget gets one run-log row that names the metric, the measurement and
the budget.

**Impact**
Phase 2–5 exit criteria can cite budget breaches from chaos waves and
staging. The check adds no wait before `turn.start`.

## What Changed

- New `server/src/services/native-runtime/native-speed-budgets.ts`: the
local budget table (codex, claude, opencode) for TTFT and agent boot
(cold and warm) and restart recovery latency. Each budget is 1.2× the
baseline, rounded up to 0.5 s. `evaluateSpeedBudget` judges one run
against p95. An unknown start kind uses the looser budget. Remote runs
and adapters without a baseline are not judged.
- `native-run-trace.ts`: the trace takes an optional `budget` subject.
It checks `task.time_to_first_token`, agent boot (end of
`heartbeat.queue` to the end of the first `runner.turn.submit` or
`runner.session.resume`) and restart recovery once each. It writes
`run.performance.budget_exceeded` through `defer`, so the write is
detached from the event path. `finish` now drains diagnostics that defer
more work.
- `native-session-executor.ts`: passes the adapter and environment to
the trace, labels cold or warm at the existing `provider_session` reuse
receipts, and checks restart recovery at a recovered run's first
provider event, measured from the startup recovery `ready` time.
- Docs: budget tables in
`doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` ("Speed
budgets (Phase 1)"), and the new event in `doc/run-log-events.md`.

## Verification

- `npx vitest run
src/services/native-runtime/native-speed-budgets.test.ts
src/services/native-runtime/native-run-trace.test.ts` (in `server/`): 34
tests pass. They cover:
  - every budget has p50 ≤ p95;
  - the provider-to-adapter mapping;
  - p95 judgement, and the looser budget when the start kind is unknown;
- one breach for a slow boot, and none for a second submit or a fast
TTFT;
  - a deferred TTFT breach is written before the trace closes;
  - restart recovery is checked once;
  - remote runs and runs with no budget subject write nothing.
- `npx tsc --noEmit -p server`: no errors.

## Risks

- Low risk. This is a run-log change only (AGENTS.md §5.7, no extra
review). The OTLP and Telemetry paths are unchanged.
- The baseline samples are small (2–12 per cell) on a shared host.
Budgets carry 20% headroom, and the codex restart budget (8.0 s p95) is
tight. Expect some breach rows on a loaded machine. They are diagnostics
and change no behavior.
- Restart-recovery budgets use run 4 of the baseline, as merged in
#19846.
- Restart recovery is measured from the startup recovery `ready` time to
the first provider event, which matches the baseline definition.
- Previous-release story: no persisted run state changes. The new event
is additive.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 15:27:42 -06:00