mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-20 15:09:57 +02:00
Measure native tool context and expand bounded workflow evals (#15218)
## Thinking Path > - Paperclip manages persistent agents and their assigned work. > - Agents receive both fixed instructions and tool definitions. > - Moving a procedure into a tool description still adds model context. > - We need to measure the complete delivered catalog and test real outcomes. > - The tested reductions saved little space and introduced failing outcomes. > - This PR keeps measurement, bounded eval coverage and the original evidence. > - Production prompts, tools and runtime behavior stay unchanged. ## Linked Issues or Issue Description Refs: #15151, #14961, #14948, #14985. This adds the measurement and eval coverage needed to assess further native instruction changes. The attempted hiring and dependency reduction failed qualification and is excluded from the final diff. ## What Changed - Measure the actual standard-mode tool authority, including all 39 tools and their input schemas. Capture scripted native start, resume and continuation payloads and the OpenCode MCP declaration list. - Add OpenCode to the two explicit-only local hiring/reuse and delegation/feedback stories. Preserve the original task requests and independent oracles. - Apply one attempt per selected story and explicit company and lead-agent budget stops. - Select managed hiring credentials from the requested profile, including OpenRouter. - Run the existing Node test files under Node instead of collecting them as Vitest suites. - Retain sanitized comparison reports, original failed grades, source hashes and evidence gaps. ## Verification Final source: `2e7cef78eef7cdfe02265e0dcb03e855b8e50bd8`. Local verification passes: - Full repository `pnpm -r typecheck` and `pnpm build`. - Eval-support unit tests: 1,252 Vitest checks and 128 Node checks. - Eval TypeScript check and six final-source measurement tests. All 36 normalized components across nine scripted deliveries and the OpenCode MCP catalog match the baseline exactly; the complete standing projection is 49,200 bytes. Fresh review of this exact head is [5/5 with no remaining findings](https://github.com/paperclipai/paperclip/pull/15218#issuecomment-5996723170); all three review threads are resolved. [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/37354539126) passes all 47 jobs, including repository typecheck, build, tests, runner checks, browser shards and the canary check. One initial annotation-test timeout is retained in attempt 1; its seven-test suite passed locally, and the affected CI lane passed on one targeted retry. No source or paid eval rerun was needed. Both ready-transition security scans passed with zero annotations. The PR is out of draft and conflict-free. GitHub still requires code-owner approval for the `package.json` test-script change; its requested reviewers are already set. A byte-for-byte comparison against master context `a65ca0950834a85bb93bcc4b4042ecacdebfef53` confirms no production changes under `packages/` or production `server/` paths. The only server addition is a measurement test. No new paid rerun is needed to compare unchanged production bytes. This does not claim that existing product defects have been fixed. The rejected corrected experiment had baseline **5 PASS / 1 FAIL** and candidate **3 PASS / 3 FAIL**, including **two newly failing pairs**. The later readiness experiment had candidate **3 PASS / 3 FAIL** and baseline **3 PASS / 2 FAIL / one setup cell without a behavioral grade**. Its five comparable pairs had two new failures, two new passes and one unchanged pass. The missing baseline Codex cell never reached its provider step because Docker setup timed out. The observed missing behaviors include parent continuation, waiting for the latest child revision, revised ZIP delivery and OpenCode credential persistence. Those failures remain failures. Source review and passing CI do not regrade them. The original reduction saved 460 bytes; its first repair saved only 125 bytes, and the larger unqualified runtime repair increased the full projection. None of those production changes is shipped here. Read the [report](https://github.com/paperclipai/paperclip/blob/2e7cef78eef7cdfe02265e0dcb03e855b8e50bd8/doc/plans/2026-10-05-native-procedure-guidance.md) and its linked sanitized receipts for exact sources, original campaign links, pair-level results and evidence limitations. ## Risks - Full-catalog bytes are not model tokens, invoices, private vendor prompts, lazy-loading behavior or truncation proof. - The two stories are explicit-only and do not prove general coding quality or arbitrary resume behavior. Single trials do not establish causation or performance trends. - The retained OpenCode candidate credential guard failed. Cleanup removed the original provider database, so the precise persistence mechanism remains unknown. The guard is unchanged. - ACPX provider-execution IDs lack proven host-call mapping. No extra-work or feedback-consumption claim is inferred by matching names, order or counts. - PostgreSQL cannot start locally while the host's shared-memory slots are exhausted. Hosted CI must supply the full database checks; the full local database suite is not claimed green. ## Model Used OpenAI Codex, based on GPT-6, with repository tools and code execution. The exact deployment model ID and context-window limit are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
5537d61794
commit
72ff3a9f27
15 files changed
+8117
-22
No files matched your search
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,382 @@
|
||||
# Native hiring and dependency guidance
|
||||
|
||||
Status: **ship measurement and eval coverage only. Do not ship the procedure
|
||||
relocation or the unqualified runtime repairs.**
|
||||
|
||||
The final PR keeps production agent behavior byte-identical to master
|
||||
`a65ca0950834a85bb93bcc4b4042ecacdebfef53`. Its changes are limited to the
|
||||
measurement test, explicit eval selection, correct hiring credential selection,
|
||||
company/lead budget stops, test-runner configuration, and retained reports.
|
||||
No production prompt, tool description, tool schema, session catalog, dependency
|
||||
mutation or status-transition code changes are included in the final diff.
|
||||
|
||||
The experiment did not pass the no-regression gate. The first corrected
|
||||
comparison had baseline 5 PASS / 1 FAIL and candidate 3 PASS / 3 FAIL: two new
|
||||
failures, no new passes, three unchanged passes and one unchanged failure.
|
||||
The attempted repair also had two new failures among five comparable pairs.
|
||||
Moving text out of the fixed prompt did not justify shipping worse outcomes.
|
||||
Single trials do not establish that the changed instructions caused a failure.
|
||||
|
||||
The original candidate reduced the complete standing projection by 460 bytes,
|
||||
about 0.9%. Restoring important instructions reduced the saving to 125 bytes,
|
||||
about 0.25%. A later, unqualified dependency guard made the projection 127 bytes
|
||||
larger than baseline. None of those changes is included in the final production
|
||||
tree. The final measurement records the unchanged full catalog. Bytes are not
|
||||
tokens, billing, or proof of upstream loading or truncation.
|
||||
|
||||
The [original sanitized comparison](2026-10-05-native-procedure-comparison.json)
|
||||
and [readiness comparison v4](2026-10-05-native-procedure-readiness-v4.json)
|
||||
retain exact sources, original grades and evidence hashes. Their old source
|
||||
revisions remain historical experiments; they do not describe production changes
|
||||
in this measurement-only PR. No completed behavioral failure was rerolled to
|
||||
obtain a pass. Earlier infrastructure recovery and missing evidence remain
|
||||
explicit below. This follows the merged completion slice in #15151.
|
||||
|
||||
## Final scope and verification
|
||||
|
||||
The server measurement test instantiates the actual standard-mode tool authority
|
||||
with all advertised tools and input schemas. It captures scripted native start,
|
||||
resume and continuation payloads across Codex, ACPX Claude and OpenCode and the
|
||||
OpenCode MCP declaration list. These are provider-free measurements.
|
||||
|
||||
The two original Everyday Workflows stories remain explicit-only and keep their
|
||||
independent artifact and ordering oracles. OpenCode is added only to the local
|
||||
hiring/reuse and delegation/feedback cells. These stories use one attempt per
|
||||
cell and actual company and lead-agent budget stops. Managed hiring selects
|
||||
the credential provider from the chosen profile, including OpenRouter. Existing
|
||||
Node test files run under Node's test runner instead of being misclassified as
|
||||
empty Vitest suites.
|
||||
|
||||
The final production identity check compares `packages/` and production
|
||||
`server/` files against the fixed master context. A fresh CI and review must
|
||||
qualify the final tree. The full local database suite is unavailable while this
|
||||
host's PostgreSQL shared-memory slots are exhausted; hosted CI provides those
|
||||
database checks. No new paid campaign is needed to compare unchanged production
|
||||
bytes. This does not claim that the existing product is free of the failures
|
||||
observed in the retained trials.
|
||||
|
||||
## Unqualified runtime repairs retained for follow-up
|
||||
|
||||
The local branch `codex/native-procedure-controller-repairs-unqualified` at
|
||||
`7e270a3d3` preserves the attempted controller repairs. It is not part of this
|
||||
PR's final diff and has not passed live qualification. It scopes completion
|
||||
wakes to each committed transition and rejects silent removal of unfinished
|
||||
dependencies, with explicit handling for obsolete dependencies. Its regression
|
||||
tests still require database execution. Source typecheck, catalog generation,
|
||||
and the non-database measurement/session checks passed locally.
|
||||
|
||||
The remaining issues need separate bounded investigation:
|
||||
|
||||
- Reopened child completion can reuse a previously consumed parent wake key.
|
||||
- A parent can discard a still-running required dependency and finish early.
|
||||
- OpenCode's v4 hiring candidate failed the exact-credential persistence guard
|
||||
in its local SQLite database. Cleanup removed that database. The credential
|
||||
identity and exact persistence mechanism are not retained. No cause or fix
|
||||
is claimed, and no credential check is weakened.
|
||||
|
||||
## Readiness comparison v4 — historical failed experiment
|
||||
|
||||
Candidate `aba7ec219b588de02cf323b4ebca559a105e6d01` and baseline
|
||||
`ce31f5efd0d416205c3994c55dd02b645f8b8423` use common master
|
||||
`a65ca0950834a85bb93bcc4b4042ecacdebfef53`. Candidate
|
||||
[campaign 37348723829](https://github.com/paperclipai/paperclip/actions/runs/37348723829)
|
||||
and baseline [campaign 37348764875](https://github.com/paperclipai/paperclip/actions/runs/37348764875)
|
||||
use trusted workflow `59015846ae02f935411afc620e0867ce812378fb`, distinct
|
||||
from both measured trees. Eleven result/API/ledger sets and 99 evidence hashes
|
||||
are retained. Their final ledgers contain 56 started runs, in addition to the
|
||||
earlier cohorts. All eleven cleanup results passed.
|
||||
|
||||
Candidate: 3 PASS / 3 FAIL. Baseline: 3 PASS / 2 FAIL / 1 setup cell without a
|
||||
behavioral grade. Comparable pairs: two new failures, two new passes and one
|
||||
unchanged pass. The remaining Codex delegation pair is uncomparable because
|
||||
baseline Docker setup timed out before its paid step; no behavioral grade or
|
||||
final result artifact exists for that cell.
|
||||
|
||||
| Story and harness | Baseline | Candidate | Observed missing behavior |
|
||||
|---|---|---|---|
|
||||
| Codex hiring/reuse | PASS | PASS | None observed by the original oracle |
|
||||
| Codex delegation/feedback | No behavioral grade | FAIL | Parent remains blocked after the revised child finishes |
|
||||
| Claude hiring/reuse | FAIL | PASS | Baseline did not deliver the revised ZIP |
|
||||
| Claude delegation/feedback | PASS | FAIL | Candidate finishes the parent before the child revision |
|
||||
| OpenCode hiring/reuse | PASS | FAIL | Candidate credential-persistence guard fails |
|
||||
| OpenCode delegation/feedback | FAIL | PASS | Baseline finishes the parent before the child revision |
|
||||
|
||||
The two new passes do not offset the new failures. The `aba7ec2` CI passed on
|
||||
attempt 2 after five infrastructure-interrupted jobs were recovered and its
|
||||
fresh review was 5/5; source checks did not clear these behavioral failures.
|
||||
|
||||
## Historical experiment scope
|
||||
|
||||
The original paid comparison used master
|
||||
`a386a599983519eb1d399f8b770bfccdb2a74762` as common context. It moved
|
||||
hiring/reuse and assigned-worker guidance to `hire_agent` and `create_task`,
|
||||
kept dependency discovery in the fixed prompt, and returned waiting instructions
|
||||
from `set_dependencies`. It advanced prompt and session-catalog revisions.
|
||||
Those production changes have been removed from the final PR. Connection
|
||||
procedures and legacy skill/API instructions remain unchanged.
|
||||
|
||||
## Measurement boundary
|
||||
|
||||
Capture the actual standard-mode server tool authority with API tools enabled,
|
||||
a local workspace, and no assigned external apps. Measure fixed instructions,
|
||||
all advertised tool descriptions and input schemas, and the serialized task
|
||||
input together. Scripted native start/resume/continuation checks cover Codex,
|
||||
ACPX Claude and OpenCode; authenticated MCP tools/list checks the OpenCode
|
||||
bridge separately. Export source identity, file hashes, bytes and tool counts.
|
||||
|
||||
These receipts have zero provider calls. Bytes are not model tokens or invoice
|
||||
costs. This boundary cannot establish the upstream SDK's private instructions,
|
||||
lazy loading, truncation or cognitive use of returned guidance. The previous
|
||||
completion-only capture selected a partial catalog and is not a full-tool
|
||||
baseline. Preserve its historical evidence unchanged.
|
||||
|
||||
## Historical matched live comparison plan
|
||||
|
||||
Use the existing explicit-only `everyday-workflows` suite with original
|
||||
`hire-reuse` and `delegate-feedback` requests and independent artifact oracles.
|
||||
Select only native Codex `gpt-5.6-sol`, ACPX Claude `claude-sonnet-5`, and OpenCode
|
||||
`openrouter/deepseek/deepseek-v4-flash-0731`, in local environments.
|
||||
|
||||
Freeze common measurement, harness and test-runner repairs in both branches
|
||||
before the candidate production changes. The historical branch keeps the
|
||||
current production prompt and tool contracts. Record exact measured source
|
||||
SHAs separately from the trusted master workflow definition and selected
|
||||
runtime/artifact hashes. Never label a later source revision as the measured
|
||||
revision without showing its delta.
|
||||
|
||||
The plan has six cells per variant, one attempt per cell, a 12-minute cell
|
||||
deadline, at most 12 story run records per cell, and a 1,000-cent company and
|
||||
lead-agent hard stop per cell. The original campaign applied the company cap
|
||||
but mistakenly left the lead budget at 0; the corrected common setup verifies
|
||||
both actual API payloads. All worker runs count toward the company budget;
|
||||
individual worker caps are not asserted.
|
||||
Count every actual run, including notifications and failed attempts. No broad
|
||||
matrix, automatic retries or baseline rerolls are part of this comparison.
|
||||
|
||||
Inspect original grades and retained content for persistent teammate identity,
|
||||
reuse on revision, worker ownership, independently tested delivered artifacts,
|
||||
parent review, dependency release, and final ordering. Check per-pair changes;
|
||||
equal totals are insufficient. Preserve every original failure and missing
|
||||
evidence. Model-authored tests cannot replace the existing independent oracle.
|
||||
|
||||
These two stories do not qualify arbitrary live resume, every existing-blocker
|
||||
combination, general coding quality, or cost/speed trends. Explicitly report
|
||||
missing evidence. Do not loosen the original oracle to create a pass.
|
||||
|
||||
## Historical complete-catalog measurement
|
||||
|
||||
The corrected source `ffe847fabaebbf24bc93924a4b6c8922d46a1fb6` is compared with
|
||||
baseline `aecfa3b6a5cab110a9b926e93705311bbcf4db47`. Both include the same
|
||||
measurement fixture and common eval repairs. Exactly 12 source/generated/unit
|
||||
paths differ; 256 common fixture files have identical bytes.
|
||||
|
||||
| Normalized extracted component | Baseline bytes | Corrected candidate bytes | Change |
|
||||
|---|---:|---:|---:|
|
||||
| Fixed instructions, start/resume | 1,809 | 1,256 | −553 |
|
||||
| All 39 tool declarations including schemas | 44,944 | 45,037 | +93 |
|
||||
| Task input, start/resume | 1,993 | 1,993 | 0 |
|
||||
| Combined standing projection, start/resume | 48,781 | 48,321 | −460 |
|
||||
| OpenCode MCP tools/list declarations | 44,249 | 44,342 | +93 |
|
||||
|
||||
Scripted Codex, ACPX Claude and OpenCode deliveries give the same comparative
|
||||
change. The serialization of the extracted components is measured separately
|
||||
from its path-normalized comparison. The fixture's real temporary paths add
|
||||
168 bytes to the combined start/resume projection in each variant. Neither
|
||||
projection is the entire transport request. Returned dependency guidance adds
|
||||
input when used, so this is not an overall token, latency or cost saving claim.
|
||||
The measurement produces zero provider calls and does not prove upstream
|
||||
loading or truncation behavior.
|
||||
|
||||
## Original comparison — failures retained
|
||||
|
||||
Original sources were candidate `eb01d073c35134bf0d373947b55a79f9e443f8ee` and
|
||||
baseline `64cd2194a1d058356679b653c181f16404aefc80`, using trusted master workflow
|
||||
`a386a599983519eb1d399f8b770bfccdb2a74762`. Source identity remains distinct from
|
||||
workflow identity and the later measurement-only correction.
|
||||
|
||||
- [Original candidate campaign](https://github.com/paperclipai/paperclip/actions/runs/37325777217)
|
||||
- [Original baseline campaign](https://github.com/paperclipai/paperclip/actions/runs/37325870695)
|
||||
|
||||
| Profile | Hire/reuse baseline → candidate | Delegate/feedback baseline → candidate |
|
||||
|---|---|---|
|
||||
| Codex | PASS → PASS | PASS → FAIL |
|
||||
| ACPX Claude | PASS → FAIL | PASS → FAIL |
|
||||
| OpenCode | FAIL → FAIL | FAIL → PASS |
|
||||
|
||||
There are three new overall failures, one new pass, one unchanged pass and one
|
||||
unchanged failure. The candidate did not qualify. Final ledgers match the
|
||||
result run-ID sets: 25 started baseline runs and 15 candidate runs, zero retry
|
||||
links. All twelve cell cleanup receipts passed. Missing/zero billing amounts
|
||||
and unmetered host runtime prevent any claim about actual cost savings.
|
||||
|
||||
The candidate Claude hire created Morgan and a child task, then ended with a
|
||||
waiting message without recording dependencies or using a terminal tool. The
|
||||
lead failed with `native_finalization_missing`. The corrected v7 prompt keeps
|
||||
explicit `set_dependencies` and `paperclip_block` discovery. This is a concrete
|
||||
missing behavior; a single trial does not establish causation.
|
||||
|
||||
The candidate Codex delegate failed in Riley's worker turn with the provider
|
||||
error `Selected model is at capacity` / `serverOverloaded`. Its parent had
|
||||
received the dependency guidance. Candidate Claude delegation failed opening
|
||||
an ACPX session (`session_ensure_failed`); its underlying cause is unknown.
|
||||
Both original grades and classifiers remain unchanged.
|
||||
|
||||
Both OpenCode hire cells failed common setup requesting an OpenAI credential
|
||||
although OpenRouter was selected, before any agent run. Shared setup now uses
|
||||
the profile's actual credential. A separate nonce/cell-ID bug left original
|
||||
lead budgets at 0; six nonce-based setup tests now verify company and lead caps
|
||||
using only the selected provider credential. These common control changes are
|
||||
why the corrected campaign uses a newly matched baseline.
|
||||
|
||||
The baseline OpenCode delegation delivered an artifact passing the independent
|
||||
oracle but completed the parent before the child's final revision. Candidate
|
||||
passed both the artifact and ordering checks. Correct output alone did not
|
||||
qualify that baseline outcome.
|
||||
|
||||
## Corrected comparison
|
||||
|
||||
- [Candidate ffe847f](https://github.com/paperclipai/paperclip/actions/runs/37331580879)
|
||||
- [Baseline aecfa3b](https://github.com/paperclipai/paperclip/actions/runs/37331642168)
|
||||
- [Candidate recovery for two interrupted cells, same ffe847f source](https://github.com/paperclipai/paperclip/actions/runs/37334061104)
|
||||
|
||||
These use the same corrected fixtures, original requests and artifact graders,
|
||||
with one attempt per campaign cell. The original campaign is preserved and is
|
||||
not relabeled as this revision. The common suite hash is
|
||||
`acd17b4acbd7051070531f7d7587131139900f54245b992eb7a62faacda8e334`.
|
||||
|
||||
| Profile | Hire/reuse baseline → candidate | Delegate/feedback baseline → candidate |
|
||||
|---|---|---|
|
||||
| Codex | PASS → PASS | PASS → FAIL (recovery) |
|
||||
| ACPX Claude | PASS → PASS | PASS → PASS (recovery) |
|
||||
| OpenCode | PASS → FAIL | FAIL → FAIL |
|
||||
|
||||
There are **two new failures, zero new passes, three unchanged passes, one
|
||||
unchanged failure and zero pending pairs**. Workflow completion is not the
|
||||
behavioral grade. These are the original result grades; no regrading was done.
|
||||
|
||||
### Missing and recovered behavior
|
||||
|
||||
**Codex delegation regressed in the observed pair.** The first parent turn
|
||||
recorded its dependency and blocked correctly. After feedback, the parent
|
||||
reviewed a stale ZIP and requested a replacement. An API comment operation was
|
||||
rejected with `API comments cannot change runner execution state`. The child
|
||||
finished the replacement at 15:45:48.284 UTC. The parent then recorded the
|
||||
dependency at 15:45:51.801–15:45:59.295 UTC; its receipt scheduled no wake. The
|
||||
parent blocked and finalized at 15:46:19.909 UTC. The child was Done, but the
|
||||
parent remained Blocked with no active continuation. All four native runs had
|
||||
successful run status; that does not make the task outcome successful. This
|
||||
sequence does not by itself establish a prompt, race or platform root cause.
|
||||
|
||||
**OpenCode hiring regressed in the observed pair.** It hired the expected
|
||||
teammate and produced an initial artifact that passed the independent checks.
|
||||
It reused the same teammate for the separator revision, then requested further
|
||||
CLI argument-handling changes. After nine started runs, the story reached the
|
||||
12-minute deadline with parent and revision work still in progress. Cleanup
|
||||
cancelled the remaining work. The final revised-ZIP and settled-outcome checks
|
||||
were not completed; they are not assumed to pass. The baseline passed after
|
||||
four runs. This single trial does not establish a speed or cost trend.
|
||||
|
||||
**OpenCode delegation failed in both variants.** The artifacts passed the
|
||||
independent output checks, but the parents finished before the final child
|
||||
revision. Candidate parent/child completion times were 15:33:43.628 /
|
||||
15:35:03.222 UTC; baseline times were 15:35:21.429 / 15:36:03.043 UTC. Both
|
||||
parents lacked review of that final revision. Equal FAIL totals conceal neither
|
||||
this missing behavior nor the passing artifact checks.
|
||||
|
||||
**Claude hiring now passes all 45 original checks.** Retained semantic receipts
|
||||
show dependency recording, returned guidance and an accepted current-track
|
||||
block with the child agent as owner. The parent later resumed and finished.
|
||||
This repairs the original missing dependency/finalization observation in this
|
||||
trial. It does not prove cognitive use of the guidance or every hiring tool
|
||||
path; API hiring remained available.
|
||||
|
||||
### Interruptions and accounting
|
||||
|
||||
Candidate Codex and Claude delegation jobs received runner shutdown signals at
|
||||
15:27:38 UTC and were cancelled at 15:27:50 UTC. Jobs `111838521694` and
|
||||
`111838521502` retained no result, final ledger or cleanup receipt. Their
|
||||
behavioral grades and total actual runs/charges are unknown. Eight distinct
|
||||
run-record IDs were visible in partial logs; this is not a complete started-run
|
||||
ledger. Preserve the workflow failures and missing evidence.
|
||||
|
||||
Only those two missing candidate cells received one bounded recovery each, on
|
||||
the identical measured source. No completed baseline or other candidate cell
|
||||
was repeated. The original workflow revision was `a386a599`; recovery used
|
||||
`a65ca0950834a85bb93bcc4b4042ecacdebfef53` after master advanced. Workflow file
|
||||
bytes were identical, SHA-256
|
||||
`eb570267e66043c1196338bb386b07ab8e8e8306ba04b6223f13b73b04fdc8f4`.
|
||||
The measured source stayed `ffe847f`, distinct from both workflow revisions.
|
||||
|
||||
All twelve completed/recovered cells have matching result/final-ledger run
|
||||
sets and passing cleanup receipts. They contain 27 baseline and 33 candidate
|
||||
started runs. The original campaign contains 40 more: **100 retained started
|
||||
runs, plus incompletely known activity from the two interrupted jobs**. Zero
|
||||
retry links within the completed ledgers do not erase the two infrastructure
|
||||
recoveries. The audits retain 100 original and 108 corrected JSON evidence
|
||||
hashes. Actual charges remain unknown because usage and interruption records
|
||||
are incomplete and host runtime is unmetered. No free-call or savings claim is
|
||||
made.
|
||||
|
||||
ACPX provider execution IDs are not proven mapped to host call IDs. No strict
|
||||
no-extra-work claim or feedback-consumption inference is made from matching
|
||||
names, ordering or counts. These evidence limits remain even for a passing
|
||||
original story grade.
|
||||
|
||||
## Historical integration and repository checks
|
||||
|
||||
After the paid sources were frozen, the PR merged master
|
||||
`a65ca0950834a85bb93bcc4b4042ecacdebfef53` as
|
||||
`ee096aa23fc88a001e7666215bcd338b5863f377`. Three generated contract conflicts
|
||||
were resolved by normal generation from the combined sources. The paired
|
||||
provider-free baseline is `ce31f5efd0d416205c3994c55dd02b645f8b8423`.
|
||||
Their twelve intended differing paths are identical to the paid source delta;
|
||||
all 256 common fixture files remain byte-identical. **No live campaign measured
|
||||
the integrated head.** Later documentation-only commits do not change that.
|
||||
|
||||
The integrated full-catalog projection is 49,200 → 48,740 normalized bytes at
|
||||
start/resume and 47,422 → 46,962 during continuation. The 460-byte reduction is
|
||||
unchanged. Master's project-discovery schema adds 419 bytes to both tool sets.
|
||||
The actual extracted start/resume projection is 49,368 → 48,908 bytes. All 39
|
||||
declarations remain present; this still does not measure private vendor prompts,
|
||||
model tokens, lazy loading or upstream truncation limits.
|
||||
|
||||
Before integration, 74 focused tool/session/measurement checks, 1,252 eval-support
|
||||
Vitest checks, 128 Node checks and the eval TypeScript check passed. Repository
|
||||
typecheck and build passed before the prompt-only v7 correction. The corrected
|
||||
source CI then passed typecheck, build, tests and all eight browser shards. An
|
||||
interrupted canary job was retried once; that is separate from paid eval recovery.
|
||||
|
||||
At integrated `ee096aa23`, local measurement/session checks passed 48 tests;
|
||||
23 authority tests could not start because PostgreSQL initialization failed,
|
||||
and 21 project-tool tests were skipped by the unsupported database helper.
|
||||
The integrated baseline passed six measurement tests. The older full local
|
||||
suite was stopped after source revisions changed during its run. Its failures
|
||||
remain retained; it is not reported as green.
|
||||
|
||||
[Integrated CI server shard 5](https://github.com/paperclipai/paperclip/actions/runs/37335450901/job/111849219339)
|
||||
failed `workspace-git-snapshot-streaming.test.ts`: the four Git filename lanes
|
||||
over 32 MiB exceeded the 180-second timeout (707 tests passed, two skipped,
|
||||
one failed in that shard). The aggregate verify job therefore failed. The cause
|
||||
has not been established; this is not reclassified as an infrastructure pass.
|
||||
The [review summary](https://github.com/paperclipai/paperclip/pull/15218#issuecomment-5996723170)
|
||||
gave integrated `ee096aa23` 5/5 but noted project-discovery authorization work
|
||||
for actors with sparse visibility. Both inline measurement findings are
|
||||
resolved; the summary's separate concern is not erased by that count.
|
||||
|
||||
At this historical checkpoint the failed behavior comparison required keeping
|
||||
the candidate draft. The final scope separates measurement from the rejected
|
||||
reduction. Future behavior changes need their own diagnosis and qualification;
|
||||
completed paid failures must not be rerolled simply to obtain passing grades.
|
||||
|
||||
## Historical behavior-change gates (not met; change removed)
|
||||
|
||||
- [x] Freeze common fixtures and both source revisions.
|
||||
- [x] Export comparable full-catalog delivery projections.
|
||||
- [x] Pass 74 focused checks, 1,252 eval-support Vitest checks and 128 Node checks.
|
||||
- [ ] Complete repository typecheck, tests and build.
|
||||
- [x] Run and inspect the bounded matched campaign, including original failures.
|
||||
- [ ] Meet the no-regression gate: **failed, two newly failing corrected pairs**.
|
||||
- [ ] Verify current-head CI, review and conflicts before readiness.
|
||||
- [x] Preserve original and corrected grades, hashes and incomplete accounting.
|
||||
|
||||
No merge is authorized.
|
||||
File diff suppressed because it is too large.
Load diff
+1
-1
@@ -73,7 +73,7 @@
|
||||
"test:runner-recovery": "vitest run server/src/services/native-runtime/native-replacement-evidence.test.ts server/src/services/native-runtime/stopped-codex-turn.test.ts server/src/services/native-runtime/native-safe-replacement.test.ts",
|
||||
"test:e2e:browser-context": "playwright test --config tests/e2e/playwright-company-context.config.ts",
|
||||
"test:e2e:runner:browser-support": "playwright test --config tests/runner-e2e/playwright-support.config.ts",
|
||||
"test:e2e:runner:unit": "vitest run --config tests/runner-e2e/vitest.config.ts",
|
||||
"test:e2e:runner:unit": "vitest run --config tests/runner-e2e/vitest.config.ts && node --test tests/runner-e2e/native-completion-checks.test.mjs tests/runner-e2e/native-completion-git-source.test.mjs tests/runner-e2e/native-completion-source-contract.test.mjs",
|
||||
"test:e2e:runner:stock-harness": "node tests/runner-e2e/stock-harness-checks.mjs",
|
||||
"test:e2e:runner:typecheck": "tsc -p tests/runner-e2e/tsconfig.json",
|
||||
"test:e2e:runner:report": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/report.ts",
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
import { afterAll, describe, expect, it } from "vitest";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { PaperclipRunnerToolAuthority } from "../services/native-runtime/paperclip-runner-tool-authority.js";
|
||||
import { buildNativeModelEnvelope, parseNativeExecutionInput, type NativeExecutionInput } from "../../../packages/paperclip-runner/src/contracts/native-execution.js";
|
||||
import type { PersistedNativeSession } from "../../../packages/paperclip-runner/src/contracts/native-session-backend.js";
|
||||
import { NATIVE_RUNTIME_ASSET_SCHEMA, PAPERCLIP_EXECUTION_PROMPT, PAPERCLIP_EXECUTION_PROMPT_REVISION, canonicalNativeRuntimeContextDigest, nativeRuntimePromptDigest } from "../../../packages/paperclip-runner/src/contracts/runtime-context.js";
|
||||
import { createRunnerdNativeSessionBackend } from "../../../packages/paperclip-runner/src/backends/codex-native-backend.js";
|
||||
import { resolveQualifiedAcpxProfile } from "../../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.js";
|
||||
import { FakeCodexTransport, WORKSPACE } from "../../../packages/paperclip-runner/src/drivers/codex/codex-app-server-driver.test-support.js";
|
||||
import { startOpenCodeMcpBridge } from "../../../packages/paperclip-runner/src/drivers/opencode/mcp-bridge.js";
|
||||
|
||||
// Production catalog selection, scripted provider transport: this measures what
|
||||
// Paperclip delivers, not the vendor's private prompt, tokenizer or billing.
|
||||
const root = new URL("../../../", import.meta.url);
|
||||
const sha = (value: string | Buffer) => createHash("sha256").update(value).digest("hex");
|
||||
const identity = { companyId: "company", agentId: "agent", issueId: "issue", runId: "run", sessionId: "session" };
|
||||
const tools = new PaperclipRunnerToolAuthority({} as Db, {
|
||||
...identity, apiToolsEnabled: true, workspaceRoot: WORKSPACE,
|
||||
}).definitions();
|
||||
const { qualificationModel: _model, reportedModelId: _reportedModel, permissionPolicy: _policy, ...profile } = resolveQualifiedAcpxProfile("claude", "claude-sonnet-5");
|
||||
const providers: NativeExecutionInput["provider"][] = [
|
||||
{ kind: "codex", model: "gpt-5.6-sol", approvalPolicy: "never" },
|
||||
{ kind: "acpx", agent: "claude", model: "claude-sonnet-5", permissionMode: "approve-all", profile },
|
||||
{ kind: "opencode", model: "openrouter/deepseek/deepseek-v4-flash-0731", permissionMode: "allow" },
|
||||
];
|
||||
const receipts: unknown[] = [];
|
||||
let mcpReceipt: unknown;
|
||||
function measure(value: unknown) {
|
||||
const json = JSON.stringify(value);
|
||||
const normalized = json.replaceAll(WORKSPACE, "/workspace");
|
||||
return {
|
||||
utf8Bytes: Buffer.byteLength(json), characters: json.length, sha256: sha(json),
|
||||
normalizedComparison: { utf8Bytes: Buffer.byteLength(normalized), characters: normalized.length, sha256: sha(normalized) },
|
||||
};
|
||||
}
|
||||
function execution(provider: NativeExecutionInput["provider"]): NativeExecutionInput {
|
||||
const entry = readFileSync(new URL("server/src/onboarding-assets/default/AGENTS.md", root), "utf8");
|
||||
writeFileSync(`${WORKSPACE}/AGENTS.md`, entry);
|
||||
const context = {
|
||||
prompt: { revision: PAPERCLIP_EXECUTION_PROMPT_REVISION, text: PAPERCLIP_EXECUTION_PROMPT, digest: nativeRuntimePromptDigest() },
|
||||
instructions: { entryPath: "AGENTS.md", bundle: { schema: NATIVE_RUNTIME_ASSET_SCHEMA, rootPath: WORKSPACE, digest: sha(entry), manifestDigest: sha(entry), fileCount: 1, totalBytes: Buffer.byteLength(entry) } },
|
||||
skills: [], mcp: { assignmentSetId: "none", digest: "0".repeat(64), bindingId: null },
|
||||
};
|
||||
return parseNativeExecutionInput({
|
||||
schema: "paperclip.native-execution-input.v5", binding: { companyId: identity.companyId, agentId: identity.agentId, issueId: identity.issueId, runId: identity.runId, executionWorkspaceId: "workspace" },
|
||||
task: { identifier: "TASK-1", title: "Review delegated work", description: "Review delegated work", prompt: "Review delegated work.", workMode: "standard" },
|
||||
workspace: { cwd: WORKSPACE, repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: "session", driverKind: provider.kind === "acpx" ? "acpx_runtime" : provider.kind === "opencode" ? "opencode_server" : "codex_app_server", protocolVersion: 1 },
|
||||
provider, executionMode: "default", planningContext: null,
|
||||
completionContract: { id: "contract", sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.completion-contract.v1", contract: { revision: "1", objective: "Review delegated work", criteria: [{ id: "objective", requirement: "Review complete" }] } },
|
||||
interactionResponses: [], credentialBindings: [],
|
||||
runtimeContext: { ...context, aggregateDigest: canonicalNativeRuntimeContextDigest(context) },
|
||||
});
|
||||
}
|
||||
function transport(provider: NativeExecutionInput["provider"]) {
|
||||
return new FakeCodexTransport("thread-1", "provider-session-1", provider.kind === "acpx" ? {
|
||||
kind: "acpx", normalizedSessionId: "session", acpxRecordId: "fixture-record", backendSessionId: "fixture-backend",
|
||||
agentSessionId: "fixture-agent", profileDigest: `sha256:${"a".repeat(64)}`, workspaceDigest: `sha256:${"b".repeat(64)}`,
|
||||
requestedModel: provider.model, effectiveModel: provider.model, permissionMode: "approve-all",
|
||||
providerLifetimeFenceCandidates: [60001, 60002, 60003],
|
||||
} : undefined);
|
||||
}
|
||||
|
||||
describe("production native procedure instruction measurement", () => {
|
||||
it("retains delivered serialized bytes separately from path-normalized comparison", () => {
|
||||
const value = { cwd: WORKSPACE, text: "λ" };
|
||||
expect(measure(value).utf8Bytes).toBe(Buffer.byteLength(JSON.stringify(value)));
|
||||
expect(measure(value).normalizedComparison.utf8Bytes).toBe(Buffer.byteLength(JSON.stringify({ ...value, cwd: "/workspace" })));
|
||||
expect(measure(value).sha256).not.toBe(measure(value).normalizedComparison.sha256);
|
||||
});
|
||||
|
||||
it("uses the real standard-mode authority, including optional and connection tools", () => {
|
||||
expect(tools.map(tool => tool.name)).toEqual(expect.arrayContaining([
|
||||
"hire_agent", "list_agents", "create_task", "set_dependencies", "connections_search", "connection_request", "register_deliverable",
|
||||
]));
|
||||
expect(new Set(tools.map(tool => tool.name)).size).toBe(tools.length);
|
||||
const withoutApi = new PaperclipRunnerToolAuthority({} as Db, { ...identity, apiToolsEnabled: false }).definitions();
|
||||
expect(withoutApi.some(tool => tool.name === "hire_agent")).toBe(false);
|
||||
});
|
||||
|
||||
for (const provider of providers) it(`captures the full ${provider.kind} start/resume/continuation payload`, async () => {
|
||||
const input = execution(provider);
|
||||
let checkpoint: PersistedNativeSession | undefined;
|
||||
for (const phase of ["start", "resume", "continuation"] as const) {
|
||||
const captured = transport(provider);
|
||||
captured.readResponse = { thread: { id: "thread-1", sessionId: "provider-session-1", cwd: WORKSPACE, turns: [] } };
|
||||
const current = phase === "continuation" ? parseNativeExecutionInput({ ...input, continuationPrompt: "Review the teammate's saved result." }) : input;
|
||||
const backend = createRunnerdNativeSessionBackend(current, { transportFactory: () => captured, dynamicTools: tools, environment: { PAPERCLIP_WORKSPACE_CWD: WORKSPACE } });
|
||||
const recovery = checkpoint ? await backend.recoverSession!(checkpoint, { signal: new AbortController().signal }) : null;
|
||||
if (recovery) expect(recovery.recovered).toBe(true);
|
||||
const session = recovery?.session ?? await backend.openSession({ identity, workingDirectory: WORKSPACE });
|
||||
try {
|
||||
checkpoint ??= await session.snapshot();
|
||||
const envelope = buildNativeModelEnvelope(current, phase === "continuation" ? { resumedSession: true } : undefined);
|
||||
if ("requestedSkills" in envelope && backend.preparedTaskConstraints) envelope.constraints = [...backend.preparedTaskConstraints];
|
||||
await session.startTurn({ message: { role: "user", text: JSON.stringify(envelope) } });
|
||||
const setup = captured.calls.find(call => call.method === (phase === "start" ? "thread/start" : "thread/resume"))!.params;
|
||||
const turn = captured.calls.find(call => call.method === "turn/start")!.params;
|
||||
const instructions = setup.developerInstructions ?? setup.baseInstructions;
|
||||
const delivered = setup.dynamicTools as Record<string, unknown>[];
|
||||
expect(delivered.filter(tool => !["paperclip_finish", "paperclip_block"].includes(String(tool.name)))).toEqual(tools);
|
||||
expect(delivered).toHaveLength(tools.length + 2);
|
||||
expect(instructions).toContain(PAPERCLIP_EXECUTION_PROMPT);
|
||||
expect(JSON.parse((turn.input as {text: string}[])[0]!.text)).toEqual(envelope);
|
||||
receipts.push({ provider: provider.kind, phase, instructions: measure(instructions), tools: measure(delivered), input: measure(turn.input),
|
||||
fullProjection: measure({ instructions, tools: delivered, input: turn.input }), deliveredToolCount: delivered.length });
|
||||
} finally { await session.close({ reason: "measurement" }); }
|
||||
}
|
||||
});
|
||||
|
||||
it("preserves the full catalog through the actual OpenCode MCP tools/list bridge", async () => {
|
||||
const bridge = await startOpenCodeMcpBridge({ tools, handler: async () => { throw new Error("No tool execution in measurement"); } });
|
||||
try {
|
||||
const response = await fetch(bridge.url, { method: "POST", headers: { authorization: `Bearer ${bridge.secret}`, "content-type": "application/json" },
|
||||
body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "tools/list", params: {} }) });
|
||||
expect(response.ok).toBe(true);
|
||||
const result = await response.json() as { result: { tools: Record<string, unknown>[] } };
|
||||
expect(result.result.tools.filter(tool => !["paperclip_finish", "paperclip_block"].includes(String(tool.name)))).toEqual(
|
||||
tools.map(({ name, description, inputSchema }) => ({ name, description, inputSchema })),
|
||||
);
|
||||
mcpReceipt = { boundary: "authenticated loopback MCP tools/list; no OpenCode/provider process", serverAnnotationsForwarded: false, tools: measure(result.result.tools), count: result.result.tools.length };
|
||||
} finally { await bridge.close(); }
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
const output = process.env.PAPERCLIP_NATIVE_PROCEDURE_MEASUREMENT;
|
||||
if (!output) return;
|
||||
if (receipts.length !== 9 || !mcpReceipt) throw new Error("Refusing incomplete instruction measurement");
|
||||
const paths = [
|
||||
"server/src/onboarding-assets/default/AGENTS.md",
|
||||
"packages/paperclip-runner/src/contracts/runtime-context.ts",
|
||||
"packages/paperclip-runner/src/protocol-actions/hire-agent.ts",
|
||||
"packages/paperclip-runner/src/protocol-actions/create-task.ts",
|
||||
"packages/paperclip-runner/src/protocol-actions/set-dependencies.ts",
|
||||
"packages/paperclip-runner/src/catalog/semantic-action-catalog.ts",
|
||||
"packages/paperclip-runner/generated/semantic-action-catalog.json",
|
||||
"packages/paperclip-runner/generated/capability/semantic-tool-contracts.json",
|
||||
"packages/paperclip-runner/protocol/fixtures/evals/native-execution-seeded.json",
|
||||
"packages/paperclip-runner/protocol/manifest.json",
|
||||
"packages/paperclip-runner/src/backends/codex-native-backend.ts",
|
||||
"packages/paperclip-runner/src/drivers/opencode/mcp-bridge.ts",
|
||||
"server/src/services/native-runtime/paperclip-runner-tool-authority.ts",
|
||||
"server/src/services/native-runtime/native-session-resume.ts",
|
||||
"server/src/__tests__/native-procedure-measurement.test.ts",
|
||||
];
|
||||
writeFileSync(output, `${JSON.stringify({ schema: "paperclip.native-procedure-measurement.v2",
|
||||
sourceSha: execFileSync("git", ["rev-parse", "HEAD"], { cwd: root, encoding: "utf8" }).trim(),
|
||||
sourceDirty: execFileSync("git", ["status", "--porcelain"], { cwd: root, encoding: "utf8" }).trim().length > 0,
|
||||
measuredInputsDirty: execFileSync("git", ["status", "--porcelain", "--", ...paths], { cwd: root, encoding: "utf8" }).trim().length > 0,
|
||||
sourceHashes: Object.fromEntries(paths.map(path => [path, sha(readFileSync(new URL(path, root)))])),
|
||||
boundary: "scripted runnerd RPC with production standard-mode authority, API tools enabled, local workspace, no assigned external apps",
|
||||
byteAccounting: "utf8Bytes counts the unmodified JSON serialization of each extracted component/projection; normalizedComparison substitutes only the fixture workspace path. Neither is the entire transport request.",
|
||||
providerCalls: 0, tokenCount: null, upstreamLoadingOrTruncation: "unverified", vendorStockPrompt: "unavailable",
|
||||
toolDescriptions: tools.map(tool => ({ name: tool.name, characters: String(tool.description).length, utf8Bytes: Buffer.byteLength(String(tool.description)), inputSchema: measure(tool.inputSchema) })),
|
||||
receipts, mcpReceipt,
|
||||
}, null, 2)}\n`);
|
||||
});
|
||||
@@ -40,7 +40,9 @@ See [controlled recovery tests](../runner-recovery/README.md).
|
||||
The local matrix has fourteen cases on native Codex `gpt-5.6-sol`, native ACPX Claude
|
||||
`claude-sonnet-5`, and native Codex `gpt-5.4-mini`: 42 cells. The two core profiles
|
||||
also declare build/revise, delegation, controller-restart, and skill-creation cases
|
||||
on Daytona: eight cells. Remote runner-process killing is not supported. For remote controller
|
||||
on Daytona: eight cells. OpenCode adds only local hiring/reuse and delegation,
|
||||
for 52 cells total. Hiring/reuse and delegation have one attempt and a
|
||||
1,000-cent company and lead-agent hard stop. Remote runner-process killing is not supported. For remote controller
|
||||
restart, a verified first download supplies the persistence checkpoint; the
|
||||
controller is interrupted during a subsequent revision with another queued
|
||||
requirement.
|
||||
|
||||
@@ -12,6 +12,31 @@ scheduled execution gets
|
||||
a fresh Paperclip home, embedded Postgres database, instance configuration,
|
||||
port, workspace, company, encrypted secrets, environment, and agent.
|
||||
|
||||
## Native procedure guidance comparison (explicit only)
|
||||
|
||||
Select `--suite everyday-workflows --environment local --case hire-reuse --case delegate-feedback
|
||||
--profile runner-codex --profile runner-acpx-claude --profile runner-opencode` for the six comparison
|
||||
cells on native Codex, ACPX Claude, and OpenCode.
|
||||
Both variants use the same Studio Lead persona, original user requests,
|
||||
artifact oracle, lifecycle checks, models and permissions. Each cell permits
|
||||
one attempt, a 12-minute deadline, at most 12 story run records, and a
|
||||
1,000-cent company hard stop; the lead also has a 1,000-cent hard stop. Worker
|
||||
runs count toward the company budget. The suite is excluded from `--all`.
|
||||
|
||||
Compare frozen branches on the same master with identical fixture sources.
|
||||
Retain each original grade, source SHA, harness digest, actual run inventory,
|
||||
downloaded artifacts and partial cost evidence. Check hiring identity/reuse,
|
||||
worker ownership, delivered revisions, dependency release and parent completion
|
||||
ordering independently of aggregate grades. These bounded stories do not
|
||||
qualify every existing-blocker combination or arbitrary provider resume.
|
||||
|
||||
The provider-free `native-procedure-measurement.test.ts` uses the server's real
|
||||
tool authority and captures scripted start/resume/continuation delivery plus
|
||||
the OpenCode MCP catalog. It includes descriptions and argument schemas. Its
|
||||
byte counts are not model token counts or proof of an upstream harness's lazy
|
||||
loading/truncation. The older completion measurement used a partial catalog;
|
||||
do not use it as a full production tool-payload baseline.
|
||||
|
||||
The vocabulary is: a **campaign** is one workflow invocation against one SHA; a
|
||||
**suite** is a durable testing purpose; a **matrix** is that suite's profiles ×
|
||||
environments × cases; an **execution/cell** is one parallel job; and an
|
||||
|
||||
@@ -136,10 +136,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 50, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
|
||||
6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(476);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(476);
|
||||
expect(validateRunnerCatalog()).toHaveLength(478);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(478);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(48);
|
||||
|
||||
@@ -1191,12 +1191,21 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
{
|
||||
id: "everyday-workflows", label: "Everyday Paperclip Work", manualOnly: true,
|
||||
description: "Real user requests, useful downloaded work, and durable continuation using production instructions.",
|
||||
groups: ["native"], profiles: everydayProfiles, environments: [localEnvironment, daytonaWarmEnvironment],
|
||||
tasks: everydayTasks, expectedMatrixSize: 50,
|
||||
groups: ["native"], profiles: [...everydayProfiles,
|
||||
productionStoryProfile(runnerProfiles.find(profile => profile.id === "runner-opencode")!),
|
||||
], environments: [localEnvironment, daytonaWarmEnvironment],
|
||||
tasks: everydayTasks.map(task => ["hire-reuse", "delegate-feedback"].includes(task.id)
|
||||
? { ...task, automaticRetryPolicy: "single_attempt" as const } : task), expectedMatrixSize: 52,
|
||||
excludedExecutionIds: [...everydayProfiles.flatMap(profile => everydayTasks
|
||||
.filter(task => !["build-revise", "delegate-feedback", "recover-controller", "create-skill-studio"].includes(task.id))
|
||||
.map(task => `everyday-workflows.${profile.id}.daytona.${task.id}`))],
|
||||
definitionMetadata: { version: 4, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only" },
|
||||
.map(task => `everyday-workflows.${profile.id}.daytona.${task.id}`)),
|
||||
...everydayTasks.flatMap(task => [
|
||||
`everyday-workflows.runner-opencode.daytona.${task.id}`,
|
||||
...(["hire-reuse", "delegate-feedback"].includes(task.id) ? [] : [`everyday-workflows.runner-opencode.local.${task.id}`]),
|
||||
]),
|
||||
],
|
||||
definitionMetadata: { version: 6, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only",
|
||||
procedureCases: { ids: ["hire-reuse", "delegate-feedback"], maximumAttemptsPerCell: 1, companyAndLeadBudgetCents: 1_000 } },
|
||||
},
|
||||
{
|
||||
id: NATIVE_INSTRUCTION_SUITE, label: "Native completion instruction consolidation", manualOnly: true,
|
||||
|
||||
@@ -212,7 +212,7 @@ describe("manual everyday workflow catalog", () => {
|
||||
const selected = selectRunnerExecutions(
|
||||
parseRunnerSelectors(["--suite", "everyday-workflows"]),
|
||||
);
|
||||
expect(selected).toHaveLength(50);
|
||||
expect(selected).toHaveLength(52);
|
||||
expect(selected.every((e) => e.profile.generation === "native")).toBe(true);
|
||||
expect(
|
||||
selectRunnerExecutions(
|
||||
|
||||
@@ -4,6 +4,29 @@ import { runnerMatrix } from "./catalog.js";
|
||||
import { setupLiveFixtures } from "./live-fixtures.js";
|
||||
|
||||
describe("live runner fixtures", () => {
|
||||
it.each(["runner-codex", "runner-acpx-claude", "runner-opencode"].flatMap(profile =>
|
||||
["hire-reuse", "delegate-feedback"].map(task => [profile, task]))) (
|
||||
"applies %s %s budget stops in actual setup with a nonce and only its selected credential", async (profile, task) => {
|
||||
const execution = runnerMatrix.find(row => row.id === `everyday-workflows.${profile}.local.${task}`)!;
|
||||
let agentBody: any;
|
||||
let companyBody: any;
|
||||
const api = {
|
||||
async get() { return [{ id: "local", driver: "local" }]; },
|
||||
async postSensitive(url: string) { return url.endsWith("/ai-connections") ? { connectionId: "account" } : { id: "secret" }; },
|
||||
async post(url: string, data: any) {
|
||||
if (url === "/api/companies") { companyBody = data; return { id: "company", name: "Test" }; }
|
||||
if (url.endsWith("/agents")) { agentBody = data; return { id: "lead", ...data }; }
|
||||
throw new Error(`Unexpected POST ${url}`);
|
||||
},
|
||||
} as unknown as RunnerApi;
|
||||
const fixtures = await setupLiveFixtures({ api, execution, executionNonce: "random-nonce", workspacePath: "/tmp/test",
|
||||
credentials: { [execution.profile.credential]: "test-value" } });
|
||||
expect(companyBody.budgetMonthlyCents).toBe(1_000);
|
||||
expect(agentBody.budgetMonthlyCents).toBe(1_000);
|
||||
await fixtures.teardown();
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["runner-codex", "legacy-codex", "runner-acpx-claude", "legacy-opencode"])(
|
||||
"creates a production-default %s hire with company and agent budget stops", async (profile) => {
|
||||
const execution = runnerMatrix.find(row => row.suite.id === "stock-harness" && row.profile.id === profile)!;
|
||||
@@ -51,6 +74,7 @@ describe("live runner fixtures", () => {
|
||||
["runner-acpx-claude", "hiring-templates", "hire-coder-template-reuse"],
|
||||
["runner-codex", "everyday-workflows", "hire-reuse"],
|
||||
["runner-acpx-claude", "everyday-workflows", "hire-reuse"],
|
||||
["runner-opencode", "everyday-workflows", "hire-reuse"],
|
||||
["runner-codex", "agent-chat-hardening", "hire-delegate-reuse"],
|
||||
["runner-acpx-claude", "agent-chat-hardening", "hire-delegate-reuse"],
|
||||
])(
|
||||
@@ -64,7 +88,7 @@ describe("live runner fixtures", () => {
|
||||
e.environment.id === "local",
|
||||
)!;
|
||||
const provider =
|
||||
profile === "runner-acpx-claude" ? "anthropic" : "openai";
|
||||
profile === "runner-acpx-claude" ? "anthropic" : profile === "runner-opencode" ? "openrouter" : "openai";
|
||||
let connected = false;
|
||||
let agentBody: any;
|
||||
const api = {
|
||||
@@ -101,8 +125,7 @@ describe("live runner fixtures", () => {
|
||||
executionNonce: "nonce",
|
||||
workspacePath: "/tmp/test",
|
||||
credentials: {
|
||||
OPENAI_API_KEY: "test-value",
|
||||
ANTHROPIC_API_KEY: "test-value",
|
||||
[execution.profile.credential]: "test-value",
|
||||
},
|
||||
});
|
||||
expect(connected).toBe(true);
|
||||
|
||||
@@ -39,7 +39,7 @@ interface AgentRecord {
|
||||
interface ManagedAccountFixture {
|
||||
connectionId: string;
|
||||
binding: {
|
||||
provider: "openai" | "anthropic";
|
||||
provider: "openai" | "anthropic" | "openrouter";
|
||||
method: "api_key";
|
||||
mode: "responsible_user";
|
||||
};
|
||||
@@ -136,7 +136,8 @@ export async function setupLiveFixtures(input: {
|
||||
return api.post<CompanyRecord>("/api/companies", {
|
||||
name: `Runner E2E ${execution.id} ${input.executionNonce}`,
|
||||
description: "Ephemeral paid full-stack runner acceptance fixture",
|
||||
budgetMonthlyCents: ["native-completion", "native-instruction-consolidation"].includes(execution.suite.id) ? NATIVE_COMPLETION_BUDGET_CENTS
|
||||
budgetMonthlyCents: ["native-completion", "native-instruction-consolidation"].includes(execution.suite.id)
|
||||
|| (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id)) ? NATIVE_COMPLETION_BUDGET_CENTS
|
||||
: execution.suite.id === "task-titles" ? TASK_TITLE_BUDGET_CENTS
|
||||
: execution.suite.id === "stock-harness" ? 1_000 : 0,
|
||||
});
|
||||
@@ -244,10 +245,11 @@ export async function setupLiveFixtures(input: {
|
||||
dependencies: ["company"],
|
||||
async setup(resolved) {
|
||||
const company = value<CompanyRecord>(resolved, "company");
|
||||
const provider =
|
||||
execution.profile.provider === "acpx" ? "anthropic" : "openai";
|
||||
const key =
|
||||
provider === "anthropic" ? "ANTHROPIC_API_KEY" : "OPENAI_API_KEY";
|
||||
const key = execution.profile.credential;
|
||||
const provider = key === "ANTHROPIC_API_KEY" ? "anthropic"
|
||||
: key === "OPENROUTER_API_KEY" ? "openrouter"
|
||||
: key === "OPENAI_API_KEY" ? "openai" : null;
|
||||
if (!provider) throw new Error(`Unsupported managed hiring credential ${key}`);
|
||||
const apiKey = input.credentials[key];
|
||||
if (!apiKey) throw new Error(`Missing credential ${key}`);
|
||||
const account = await api.postSensitive<{ connectionId: string }>(
|
||||
@@ -290,7 +292,10 @@ export async function setupLiveFixtures(input: {
|
||||
secretRefs,
|
||||
executionId: input.executionNonce,
|
||||
});
|
||||
if (execution.suite.id === "stock-harness") agent.budgetMonthlyCents = 1_000;
|
||||
if (execution.suite.id === "stock-harness"
|
||||
|| (execution.suite.id === "everyday-workflows" && ["hire-reuse", "delegate-feedback"].includes(execution.task.id))) {
|
||||
agent.budgetMonthlyCents = 1_000;
|
||||
}
|
||||
if (managedHiring) {
|
||||
const account = value<ManagedAccountFixture>(resolved, "ai-connection");
|
||||
const config = agent.adapterConfig as Record<string, unknown>;
|
||||
|
||||
@@ -69,7 +69,10 @@ describe("native instruction comparison admission", () => {
|
||||
const files = Object.keys(NATIVE_INSTRUCTION_VARIANTS.baseline);
|
||||
const baseline = new Map(files.map(file => [file, execFileSync("git", ["show", `${NATIVE_INSTRUCTION_BASE_SHA}:${file}`])]));
|
||||
expect(nativeInstructionVariant(file => baseline.get(file)!)).toBe("baseline");
|
||||
const candidate = new Map(files.map(file => [file, readFileSync(new URL(`../../${file}`, import.meta.url))]));
|
||||
// This historical admission intentionally rejects later production changes.
|
||||
// Exercise the retained qualified tree rather than requiring HEAD to remain
|
||||
// one of this completed comparison's frozen variants forever.
|
||||
const candidate = new Map(files.map(file => [file, execFileSync("git", ["show", `16ef1a5744b23bb043a8b286e6288921da1de98f:${file}`])]));
|
||||
const currentVariant = nativeInstructionVariant(file => candidate.get(file)!);
|
||||
expect(["baseline", "candidate", "corrected", "feedback", "opencodeFeedback", "opencodeFeedbackSettlement", "opencodeFeedbackResponses"]).toContain(currentVariant);
|
||||
candidate.set(files[0]!, currentVariant === "candidate" ? baseline.get(files[0]!)! : Buffer.from("unknown source"));
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { runnerMatrix, runnerSuites } from "./catalog.js";
|
||||
import { everydayTasks } from "./everyday-cases.js";
|
||||
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
|
||||
|
||||
describe("bounded native procedure comparison", () => {
|
||||
const suite = runnerSuites.find(suite => suite.id === "everyday-workflows")!;
|
||||
it("selects only two original journeys on three native profiles, with no automatic retry", () => {
|
||||
expect(suite.manualOnly).toBe(true);
|
||||
const selected = selectRunnerExecutions(parseRunnerSelectors(["--suite", suite.id,
|
||||
"--profile", "runner-codex", "--profile", "runner-acpx-claude", "--profile", "runner-opencode",
|
||||
"--case", "hire-reuse", "--case", "delegate-feedback", "--environment", "local"]));
|
||||
expect(selected).toHaveLength(6);
|
||||
for (const { task } of selected) {
|
||||
const original = everydayTasks.find(candidate => candidate.id === task.id)!;
|
||||
expect(task).toEqual({ ...original, automaticRetryPolicy: "single_attempt" });
|
||||
expect(task.buildPrompt("nonce")).toBe(original.buildPrompt("nonce"));
|
||||
expect(task.attemptTimeoutMs.local).toBe(720_000);
|
||||
}
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"]), runnerMatrix).some(cell => cell.suite.id === suite.id)).toBe(false);
|
||||
});
|
||||
it("keeps procedure and completion directions out of the fixture persona", () => {
|
||||
for (const profile of suite.profiles) {
|
||||
const input = { executionId: "random-nonce", environmentId: "env", environmentFixtureId: "local" as const, workspacePath: "/workspace", secretRefs: {
|
||||
[profile.credential]: { type: "secret_ref" as const, secretId: "00000000-0000-4000-8000-000000000001", version: "latest" as const },
|
||||
} };
|
||||
const agent = profile.buildAgent(input);
|
||||
expect(JSON.stringify(agent.instructionsBundle)).not.toMatch(/paperclip_finish|paperclip_block|set_dependencies|hire_agent/);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -1,10 +1,12 @@
|
||||
import { defineConfig } from "vitest/config";
|
||||
import { configDefaults, defineConfig } from "vitest/config";
|
||||
|
||||
export default defineConfig({
|
||||
root: import.meta.dirname,
|
||||
test: {
|
||||
name: "runner-e2e-support",
|
||||
include: ["**/*.test.ts", "**/*.test.mjs"],
|
||||
// These use node:test and run separately in test:e2e:runner:unit.
|
||||
exclude: [...configDefaults.exclude, "native-completion-{checks,git-source,source-contract}.test.mjs"],
|
||||
testTimeout: 30_000,
|
||||
},
|
||||
});
|
||||
Reference in new issue
Block a user