Commit Graph
6207 Commits
Author SHA1 Message Date
Dotta c455a206a8 fix(evals): preserve original native observation clocks for eligible historical finals 2026-10-28 11:15:52 -05:00
Dotta 3e1c87117e fix(evals): seed active prerequisite tasks with valid owners 2026-10-28 11:03:18 -05:00
Dotta a1c5811363 fix(evals): retain executable cancellation proof at final boundary 2026-10-28 10:56:32 -05:00
Dotta b740827954 test(evals): refresh pricing and artifact valid observations 2026-10-28 10:33:20 -05:00
Dotta 22dca95926 fix(evals): avoid cancelling consumed privacy response bodies 2026-10-28 10:22:02 -05:00
Dotta cb31ebccf0 test(e2e): preserve secret proposal original observation clock 2026-10-28 09:57:44 -05:00
Dotta e386933481 test(e2e): bind secret proposal review to expanded card 2026-10-28 09:46:16 -05:00
Dotta 8bbf313552 test(e2e): keep long-thread measurements within original clock 2026-10-28 09:46:16 -05:00
Dotta ed24a4a444 Retain certified source SDK evidence for plugin fixtures 2026-10-28 09:46:16 -05:00
Dotta 7170ca4b81 test(e2e): preserve original approval and session clocks 2026-10-28 09:12:28 -05:00
Dotta a477b2c5d8 fix(evals): preserve original task and inbox observation budgets 2026-10-28 08:17:14 -05:00
Dotta efa3382b21 test(e2e): fingerprint executed plugin CLI source and loader 2026-10-28 08:15:28 -05:00
Dotta 250ba62791 test(e2e): bind installed team to selected OpenCode runtime 2026-10-28 08:15:27 -05:00
DottaandPaperclip 1c6021a4e6 fix(evals): bind inline secret helper to actual consumer task
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 08:15:27 -05:00
DottaandPaperclip 0e4ebca61c fix(evals): bind approval wait to actual task workspace
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 08:15:27 -05:00
DottaandPaperclip dea44755b0 fix(evals): prepare prerequisite people through public membership routes
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 08:15:26 -05:00
Dotta 81987af644 Accept lifecycle-ready running workers in catalog readiness 2026-10-28 08:15:26 -05:00
Dotta 8cd9e8bc6b fix(evals): keep fast response initial observation on original clock 2026-10-28 07:26:09 -05:00
DottaandPaperclip 1c7cbdc25b test(evals): preserve artifact outcome without extra task status gate
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 07:20:35 -05:00
DottaandPaperclip 3dab66775d fix(evals): bind answer race completion and ordinary draft evidence
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 07:20:35 -05:00
Dotta af01c4f6d3 fix(evals): await fast response save before model readback 2026-10-28 07:19:59 -05:00
Dotta 9ea0c8a61d fix(evals): retain original clock for seven prepared human journeys 2026-10-28 02:31:43 -05:00
Dotta d282ae2eee fix(evals): share original observation clock across selected native phases 2026-10-28 02:01:44 -05:00
Dotta 81abf04b21 fix(evals): qualify owned OpenCode post-stop proxy cleanup 2026-10-28 01:32:20 -05:00
Dotta 15fb2318bf fix(evals): preserve human origin on consumed queue probes 2026-10-28 01:32:20 -05:00
Dotta b70439f387 fix(evals): keep deadline boundary timeout strictly positive 2026-10-28 01:32:20 -05:00
Dotta b0eff403b8 fix(evals): share original budget for instance and confirmation waits 2026-10-28 01:32:19 -05:00
Dotta e232200747 test(e2e): pin configurable auxiliary tests to requested model 2026-10-28 00:55:14 -05:00
Dotta 1e686fa0f3 fix(evals): bind routine failure evidence to selected profile 2026-10-28 00:53:18 -05:00
Dotta 78783dda57 fix(evals): bind routine history grading to observed provider 2026-10-28 00:47:54 -05:00
Dotta 765dd51e0f test(e2e): retain context custody across budget handoff 2026-10-28 00:37:55 -05:00
Dotta fa7304ae5e test(e2e): share original budget across company followups 2026-10-28 00:37:55 -05:00
Dotta 6eca7f6bcc test(e2e): verify returned historical artifact copies 2026-10-28 00:25:04 -05:00
Dotta 2c6d746015 fix(evals): retain contracted delivery negation in date grading 2026-10-28 00:25:04 -05:00
Dotta fefc475470 fix(evals): accept exact natural historical delivery dates 2026-10-28 00:25:04 -05:00
Dotta 8b29ca5489 fix(evals): enable real workspace view before lifecycle journeys 2026-10-28 00:21:59 -05:00
Dotta 81bba1dd0d fix(evals): scope design approval claims to contrast clauses 2026-10-27 23:00:30 -05:00
Dotta 89063699af fix(evals): recognize explicit design revision ordering 2026-10-27 23:00:29 -05:00
Dotta 22dd6a1280 test(evals): report effective project-defaults timeout policy 2026-10-27 22:36:30 -05:00
DottaandPaperclip d1f983e395 test(evals): distinguish team pricing from negated per-user charges
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 9434442348 test(evals): reject waived required skills without confusing unrelated history
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 96dcd15449 test(evals): reject modal contradictions of required plan features
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 021c155682 test(evals): ground team requirements in actual skill identities
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 913ffeec7c test(evals): accept substantive single-page plan paraphrases
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 33f99c6c5a docs(evals): preserve the composed operating section heading
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:18:36 -05:00
DottaandPaperclip b797e1cc29 test(evals): read delivered task HTML with the authenticated fixture actor
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:18:00 -05:00
DottaandPaperclip 1ba267de60 test(evals): use production chat timeout within finite cell budgets
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:18:00 -05:00
Dotta 01c585d965 docs(evals): record refreshed merged-baseline chat pilot 2026-10-27 21:08:53 -05:00
DottaandPaperclip 0195fcb016 Merge pinned production for the post-custody chat pilot
Preserve frozen 244-scenario YAML intent and 191-capability coverage. Production baseline: 39e811c9c5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:03:19 -05:00
DottaandPaperclip 39e811c9c5 fix(runner): prepare diagnostic proxy and reduce bundled skill queries (#17941)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native turns reconcile company skills and select verified remote
runtime packs.
> - Catalog imports repeat folder transactions for skills in the same
bundled category.
> - Diagnostic requests can also reuse an older image proxy that lacks
the phase collector.
> - This pull request reuses categories within one import and selects
the current proxy for fresh diagnostics.
> - Each import has its own category map, and recovery keeps its
admitted runtime identity.
> - The result is less preparation work and a runtime that can provide
the requested measurements.

## Linked Issues or Issue Description

Refs #17918.

**What happened?**

A hosted run retained `PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` but
emitted no provider phase batch. Its saved pack authority selected an
older image proxy. The deployed server proxy contained the collector.
The image proxy did not. Bundled inventory reconciliation also repeats
folder transactions for skills in the same category. The paired local
benchmark exposes that preparation cost.

**Expected behavior**

A fresh diagnostic request must use a verified proxy that supports the
current collector. Bundled skill reconciliation should resolve each
category once per import.

**Steps to reproduce**

1. Use a protected image pack with the same OpenCode interface revision
and an older proxy hash.
2. Enable `PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` for a fresh
remote run.
3. Observe that the current backend accepts the older proxy and cannot
emit the requested phase batch.

## What Changed

- Reuse each bundled category during one import. Every skill still runs
its existing validation, access lookup, value comparison, and write
path.
- Compare the installed proxy hash with the current build when a fresh
run requests diagnostics.
- Keep compatible-pack reuse for ordinary runs and exact admitted
identity for recovery.
- Cover selection through the real backend, including recovery after
diagnostics are enabled.
- Document runtime preparation costs and admission limits for diagnostic
requests.

The skill importer owns category reuse. The provider verifier owns the
artifact comparison. The native backend passes the resolved diagnostic
requirement at the existing admission seam. The changes stay in those
existing owners and share one latency investigation and rollout.

## Verification

- Before the fix, the new verifier and backend cases failed because both
accepted a stale proxy.
- After the fix, 63 focused cases passed in 8.10 seconds. They cover
normal reuse, current diagnostic proxy selection, stale proxy rejection,
selected ACPX profiles, and admitted recovery identity.
- 110 skill and folder service cases passed against isolated real
PostgreSQL databases. Only cluster bootstrap used the existing local
PostgreSQL service. Queries, migrations, authorization, file handling,
and transactions ran on their real paths.
- A paired benchmark used the same immutable skill snapshot,
authenticated actor, 170-skill inventory, and 3.5 ms delay in each
network direction. It checked every delivered file and executable bit.
Warm median fell from 3.319 s to 2.482 s (25.2%). SQL calls fell by 81
in each matched sample, from 241–243 to 160–162. The bundled phase
median fell from 2.596 s to 1.721 s. The final measurement uses commit
`37eac8d8de`; only the skill importer differs from baseline
`7951455073`. These local results are not hosted acceptance.
- `pnpm -r --workspace-concurrency=1 typecheck` passed with the pinned
pnpm 9.15.4 and a 6 GiB Node heap. Earlier attempts exposed a subprocess
toolchain mismatch, a local IPC restriction, and a fixture type error.
Each was corrected before the passing run.
- Module boundary and whitespace checks passed.
- `pnpm build` passed for all workspaces.
- `pnpm test:run` did not complete. Embedded PostgreSQL initialization
failed after repeated startup attempts. The owned test subtree was
stopped after preserving the failure log. This is not a full local test
pass. On final commit `37eac8d8de`, all 53 successful check contexts
passed; two optional Storybook checks were skipped. Greptile is 5/5 with
no unresolved findings. The draft awaits merge approval.
- The catalog change has a measured local gain. The proxy change fixes
missing phase capture. Real hosted response latency remains above the
target and needs a new matched measurement after delivery.

## Risks

The category map belongs to one import and one company. The importer
still resolves current skill data on every invocation.

Fresh diagnostic runs can prepare a current verified pack, which can add
startup time. Admission can fail when matching runtime assets are
unavailable. Ordinary runs keep the existing fast compatibility path.
Recovery keeps the already admitted identity, even if diagnostics are
enabled later. The operator endpoint gate, collector limits, and closed
span fields are unchanged.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, shell execution,
browser tools, and Honeycomb queries. The session does not expose a
verifiable backend deployment ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 21:00:05 -05:00