Commit Graph
6206 Commits
Author SHA1 Message Date
Dotta 3e1c87117e fix(evals): seed active prerequisite tasks with valid owners 2026-10-28 11:03:18 -05:00
Dotta a1c5811363 fix(evals): retain executable cancellation proof at final boundary 2026-10-28 10:56:32 -05:00
Dotta b740827954 test(evals): refresh pricing and artifact valid observations 2026-10-28 10:33:20 -05:00
Dotta 22dca95926 fix(evals): avoid cancelling consumed privacy response bodies 2026-10-28 10:22:02 -05:00
Dotta cb31ebccf0 test(e2e): preserve secret proposal original observation clock 2026-10-28 09:57:44 -05:00
Dotta e386933481 test(e2e): bind secret proposal review to expanded card 2026-10-28 09:46:16 -05:00
Dotta 8bbf313552 test(e2e): keep long-thread measurements within original clock 2026-10-28 09:46:16 -05:00
Dotta ed24a4a444 Retain certified source SDK evidence for plugin fixtures 2026-10-28 09:46:16 -05:00
Dotta 7170ca4b81 test(e2e): preserve original approval and session clocks 2026-10-28 09:12:28 -05:00
Dotta a477b2c5d8 fix(evals): preserve original task and inbox observation budgets 2026-10-28 08:17:14 -05:00
Dotta efa3382b21 test(e2e): fingerprint executed plugin CLI source and loader 2026-10-28 08:15:28 -05:00
Dotta 250ba62791 test(e2e): bind installed team to selected OpenCode runtime 2026-10-28 08:15:27 -05:00
DottaandPaperclip 1c6021a4e6 fix(evals): bind inline secret helper to actual consumer task
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 08:15:27 -05:00
DottaandPaperclip 0e4ebca61c fix(evals): bind approval wait to actual task workspace
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 08:15:27 -05:00
DottaandPaperclip dea44755b0 fix(evals): prepare prerequisite people through public membership routes
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 08:15:26 -05:00
Dotta 81987af644 Accept lifecycle-ready running workers in catalog readiness 2026-10-28 08:15:26 -05:00
Dotta 8cd9e8bc6b fix(evals): keep fast response initial observation on original clock 2026-10-28 07:26:09 -05:00
DottaandPaperclip 1c7cbdc25b test(evals): preserve artifact outcome without extra task status gate
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 07:20:35 -05:00
DottaandPaperclip 3dab66775d fix(evals): bind answer race completion and ordinary draft evidence
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 07:20:35 -05:00
Dotta af01c4f6d3 fix(evals): await fast response save before model readback 2026-10-28 07:19:59 -05:00
Dotta 9ea0c8a61d fix(evals): retain original clock for seven prepared human journeys 2026-10-28 02:31:43 -05:00
Dotta d282ae2eee fix(evals): share original observation clock across selected native phases 2026-10-28 02:01:44 -05:00
Dotta 81abf04b21 fix(evals): qualify owned OpenCode post-stop proxy cleanup 2026-10-28 01:32:20 -05:00
Dotta 15fb2318bf fix(evals): preserve human origin on consumed queue probes 2026-10-28 01:32:20 -05:00
Dotta b70439f387 fix(evals): keep deadline boundary timeout strictly positive 2026-10-28 01:32:20 -05:00
Dotta b0eff403b8 fix(evals): share original budget for instance and confirmation waits 2026-10-28 01:32:19 -05:00
Dotta e232200747 test(e2e): pin configurable auxiliary tests to requested model 2026-10-28 00:55:14 -05:00
Dotta 1e686fa0f3 fix(evals): bind routine failure evidence to selected profile 2026-10-28 00:53:18 -05:00
Dotta 78783dda57 fix(evals): bind routine history grading to observed provider 2026-10-28 00:47:54 -05:00
Dotta 765dd51e0f test(e2e): retain context custody across budget handoff 2026-10-28 00:37:55 -05:00
Dotta fa7304ae5e test(e2e): share original budget across company followups 2026-10-28 00:37:55 -05:00
Dotta 6eca7f6bcc test(e2e): verify returned historical artifact copies 2026-10-28 00:25:04 -05:00
Dotta 2c6d746015 fix(evals): retain contracted delivery negation in date grading 2026-10-28 00:25:04 -05:00
Dotta fefc475470 fix(evals): accept exact natural historical delivery dates 2026-10-28 00:25:04 -05:00
Dotta 8b29ca5489 fix(evals): enable real workspace view before lifecycle journeys 2026-10-28 00:21:59 -05:00
Dotta 81bba1dd0d fix(evals): scope design approval claims to contrast clauses 2026-10-27 23:00:30 -05:00
Dotta 89063699af fix(evals): recognize explicit design revision ordering 2026-10-27 23:00:29 -05:00
Dotta 22dd6a1280 test(evals): report effective project-defaults timeout policy 2026-10-27 22:36:30 -05:00
DottaandPaperclip d1f983e395 test(evals): distinguish team pricing from negated per-user charges
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 9434442348 test(evals): reject waived required skills without confusing unrelated history
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 96dcd15449 test(evals): reject modal contradictions of required plan features
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 021c155682 test(evals): ground team requirements in actual skill identities
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 913ffeec7c test(evals): accept substantive single-page plan paraphrases
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 33f99c6c5a docs(evals): preserve the composed operating section heading
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:18:36 -05:00
DottaandPaperclip b797e1cc29 test(evals): read delivered task HTML with the authenticated fixture actor
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:18:00 -05:00
DottaandPaperclip 1ba267de60 test(evals): use production chat timeout within finite cell budgets
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:18:00 -05:00
Dotta 01c585d965 docs(evals): record refreshed merged-baseline chat pilot 2026-10-27 21:08:53 -05:00
DottaandPaperclip 0195fcb016 Merge pinned production for the post-custody chat pilot
Preserve frozen 244-scenario YAML intent and 191-capability coverage. Production baseline: 39e811c9c5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:03:19 -05:00
DottaandPaperclip 39e811c9c5 fix(runner): prepare diagnostic proxy and reduce bundled skill queries (#17941)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native turns reconcile company skills and select verified remote
runtime packs.
> - Catalog imports repeat folder transactions for skills in the same
bundled category.
> - Diagnostic requests can also reuse an older image proxy that lacks
the phase collector.
> - This pull request reuses categories within one import and selects
the current proxy for fresh diagnostics.
> - Each import has its own category map, and recovery keeps its
admitted runtime identity.
> - The result is less preparation work and a runtime that can provide
the requested measurements.

## Linked Issues or Issue Description

Refs #17918.

**What happened?**

A hosted run retained `PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` but
emitted no provider phase batch. Its saved pack authority selected an
older image proxy. The deployed server proxy contained the collector.
The image proxy did not. Bundled inventory reconciliation also repeats
folder transactions for skills in the same category. The paired local
benchmark exposes that preparation cost.

**Expected behavior**

A fresh diagnostic request must use a verified proxy that supports the
current collector. Bundled skill reconciliation should resolve each
category once per import.

**Steps to reproduce**

1. Use a protected image pack with the same OpenCode interface revision
and an older proxy hash.
2. Enable `PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` for a fresh
remote run.
3. Observe that the current backend accepts the older proxy and cannot
emit the requested phase batch.

## What Changed

- Reuse each bundled category during one import. Every skill still runs
its existing validation, access lookup, value comparison, and write
path.
- Compare the installed proxy hash with the current build when a fresh
run requests diagnostics.
- Keep compatible-pack reuse for ordinary runs and exact admitted
identity for recovery.
- Cover selection through the real backend, including recovery after
diagnostics are enabled.
- Document runtime preparation costs and admission limits for diagnostic
requests.

The skill importer owns category reuse. The provider verifier owns the
artifact comparison. The native backend passes the resolved diagnostic
requirement at the existing admission seam. The changes stay in those
existing owners and share one latency investigation and rollout.

## Verification

- Before the fix, the new verifier and backend cases failed because both
accepted a stale proxy.
- After the fix, 63 focused cases passed in 8.10 seconds. They cover
normal reuse, current diagnostic proxy selection, stale proxy rejection,
selected ACPX profiles, and admitted recovery identity.
- 110 skill and folder service cases passed against isolated real
PostgreSQL databases. Only cluster bootstrap used the existing local
PostgreSQL service. Queries, migrations, authorization, file handling,
and transactions ran on their real paths.
- A paired benchmark used the same immutable skill snapshot,
authenticated actor, 170-skill inventory, and 3.5 ms delay in each
network direction. It checked every delivered file and executable bit.
Warm median fell from 3.319 s to 2.482 s (25.2%). SQL calls fell by 81
in each matched sample, from 241–243 to 160–162. The bundled phase
median fell from 2.596 s to 1.721 s. The final measurement uses commit
`37eac8d8de`; only the skill importer differs from baseline
`7951455073`. These local results are not hosted acceptance.
- `pnpm -r --workspace-concurrency=1 typecheck` passed with the pinned
pnpm 9.15.4 and a 6 GiB Node heap. Earlier attempts exposed a subprocess
toolchain mismatch, a local IPC restriction, and a fixture type error.
Each was corrected before the passing run.
- Module boundary and whitespace checks passed.
- `pnpm build` passed for all workspaces.
- `pnpm test:run` did not complete. Embedded PostgreSQL initialization
failed after repeated startup attempts. The owned test subtree was
stopped after preserving the failure log. This is not a full local test
pass. On final commit `37eac8d8de`, all 53 successful check contexts
passed; two optional Storybook checks were skipped. Greptile is 5/5 with
no unresolved findings. The draft awaits merge approval.
- The catalog change has a measured local gain. The proxy change fixes
missing phase capture. Real hosted response latency remains above the
target and needs a new matched measurement after delivery.

## Risks

The category map belongs to one import and one company. The importer
still resolves current skill data on every invocation.

Fresh diagnostic runs can prepare a current verified pack, which can add
startup time. Admission can fail when matching runtime assets are
unavailable. Ordinary runs keep the existing fast compatibility path.
Recovery keeps the already admitted identity, even if diagnostics are
enabled later. The operator endpoint gate, collector limits, and closed
span fields are unchanged.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, shell execution,
browser tools, and Honeycomb queries. The session does not expose a
verifiable backend deployment ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 21:00:05 -05:00
DottaandPaperclip 2cd747d1b6 perf(server): batch connection install capability checks (#17945)
## Thinking Path

> - Paperclip manages AI agents and their access to external
connections.
> - The connection grants screen describes which agents can receive an
installation.
> - Each candidate currently repeats authorization for every retained
installation.
> - A company installation makes this work grow with the square of the
roster size.
> - This pull request checks retained targets once inside the existing
service owner.
> - The benefit is fewer authorization calls with the same scoped
capability results.

## Linked Issues or Issue Description

**What happened?**

The grants GET repeats connection preparation and retained-target checks
for each candidate agent. A local 100-agent company-install fixture made
50,300 SQL calls and 10,000 target lookups in one capability-description
operation. Open PR searches found no matching capability fan-out fix.

**Expected behavior**

The description must preserve retained-installation obligations and each
candidate's permission. It should evaluate retained targets once per
description. Writes must still authorize fresh.

**Steps to reproduce**

Run the committed benchmark with
`--baseline-ref=f6bc653f7f745adaafa26393852fb2f5a99256a4`. It creates
disposable synthetic PostgreSQL fixtures and loads the actual pre-fix
service from Git. It compares every positional capability, reason, and
operation.

**Paperclip version or commit**

Baseline `f6bc653f7f745adaafa26393852fb2f5a99256a4`.

**Deployment mode**

Built from source. Local real PostgreSQL proof. No hosted rollout.

## What Changed

- Add `describeAgentInstallCapabilities` to connection management.
Ordinary authorization and the batch share preparation, target checks,
and fallback completion.
- Use the batch only for direct saved non-AI connections and board
actors. Complex resources, delegated actors, and unusual saved target
IDs keep the ordinary evaluator. Fallback consumes the first preparation
once.
- Replace only the grants route's editable-agent loop. Keep
configure/use checks, identity filtering, audience disclosure, and write
authority on their existing paths.
- Add scoped-grant, retained-target, error-order, freshness, fallback,
full HTTP response, and real mutation regression coverage. Add a
reproducible benchmark and evidence notes.

## Verification

Real PostgreSQL 18.6 Linux Docker, Node 26.4.0. Two warmups and five
measured samples per arm, in alternating order. These are
service-component measurements with the same SQL-count instrumentation
in both arms. They do not measure the whole hosted endpoint.

| Fixture | Old median | Batch median | SQL calls old → batch | Target
lookups old → batch |
| --- | ---: | ---: | ---: | ---: |
| 100 agents, company install | 24.753 s | 246.39 ms | 50,300 → 503 |
10,000 → 100 |
| 100 agents, partial install | 6.339 s | 236.37 ms | 13,075 → 502 |
2,575 → 100 |
| 1 agent, company install | 5.13 ms | 4.82 ms | 8 → 8 | 1 → 1 |

All eight fixture shapes and every measured iteration had complete
positional equality. One-agent timing ranges overlap. Full ranges,
database settings, and limits are in
`doc/plans/2026-10-27-connection-grants-batch-verification.md`.

- **Passed:** `pnpm -r typecheck` and `pnpm build`, using pinned pnpm
9.15.4.
- **Passed:** 116 focused tests: all 83 connection-management
integration cases use real PostgreSQL and HTTP; 33 batch regressions
execute real policy over synthetic query transport. The integration
cases keep existing database creation, migrations, clients, services,
and assembled HTTP server, with disposable Docker PostgreSQL.
- **Passed:** complete grants GET parity across six human-role fixtures;
real install PUT succeeds before revocation, then fresh GET and actual
PUT deny after revocation without changing saved installs.
- **Passed:** a real PostgreSQL red/green regression for empty saved
target selectors, including candidate-specific decisions and later
reached infrastructure faults.
- **Passed:** independent old/new evaluator comparisons, including
recursive and delegated fallback, run-context reuse, continuation,
failure order, and principal/company isolation.
- **Local full-suite limit:** `pnpm test:run` was attempted and stopped
after native PostgreSQL fixtures could not initialize. An isolated
failed native case reproduced that startup error. The host cannot
allocate PostgreSQL SysV semaphores, including outside the sandbox. The
focused suites use an existing constructor seam to start disposable
Docker PostgreSQL. This is not a complete local full-suite pass.
- **Passed:** all 16 native-runtime cases that failed to start
PostgreSQL on the host, rerun with the disposable Docker provider. The
full local workflow remains incomplete.
- `git diff --check` passes. Greptile is 5/5 with no actionable findings
on final head `41dcf3ae029b6c60999ca6208136399b0af663d0`. All final-head
CI gates passed, including the full PR workflow. The two Storybook
checks are expected skips because this change has no UI changes.

Normal reproduction:

```sh
node --import ./server/node_modules/tsx/dist/esm/index.mjs scripts/benchmark-connection-install-capabilities.ts --baseline-ref=f6bc653f7f745adaafa26393852fb2f5a99256a4
cd server
pnpm exec vitest run src/__tests__/connection-install-capabilities.test.ts src/__tests__/connection-management.integration.test.ts
```

## Risks

Retained facts are observed once during one descriptive call. A mid-call
revoke or deletion can leave a stale displayed choice. The next
description and every mutation or human continuation read fresh
authority. The batch returns no reusable authorization context.

The bound applies to target evaluations in the eligible shape. Scoped
grant payloads and hierarchy reads can still grow inside each policy
decision. Complex resources and agent callers keep existing costs.
Hosted latency improvement remains unverified. There is no migration,
provider protocol change, new worker, persistent cache, or UI contract
change.

## Model Used

OpenAI Codex with code editing, execution, and test tools. Design, test,
and independent review agents used exact model IDs `gpt-6.1-sol` and
`gpt-6-astra` with xhigh reasoning. The host did not expose the primary
session's exact model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 20:59:01 -05:00