Commit Graph
532 Commits
Author SHA1 Message Date
Dotta 9ea0c8a61d fix(evals): retain original clock for seven prepared human journeys 2026-10-28 02:31:43 -05:00
Dotta d282ae2eee fix(evals): share original observation clock across selected native phases 2026-10-28 02:01:44 -05:00
Dotta 81abf04b21 fix(evals): qualify owned OpenCode post-stop proxy cleanup 2026-10-28 01:32:20 -05:00
Dotta 15fb2318bf fix(evals): preserve human origin on consumed queue probes 2026-10-28 01:32:20 -05:00
Dotta b70439f387 fix(evals): keep deadline boundary timeout strictly positive 2026-10-28 01:32:20 -05:00
Dotta b0eff403b8 fix(evals): share original budget for instance and confirmation waits 2026-10-28 01:32:19 -05:00
Dotta e232200747 test(e2e): pin configurable auxiliary tests to requested model 2026-10-28 00:55:14 -05:00
Dotta 1e686fa0f3 fix(evals): bind routine failure evidence to selected profile 2026-10-28 00:53:18 -05:00
Dotta 78783dda57 fix(evals): bind routine history grading to observed provider 2026-10-28 00:47:54 -05:00
Dotta 765dd51e0f test(e2e): retain context custody across budget handoff 2026-10-28 00:37:55 -05:00
Dotta fa7304ae5e test(e2e): share original budget across company followups 2026-10-28 00:37:55 -05:00
Dotta 6eca7f6bcc test(e2e): verify returned historical artifact copies 2026-10-28 00:25:04 -05:00
Dotta 2c6d746015 fix(evals): retain contracted delivery negation in date grading 2026-10-28 00:25:04 -05:00
Dotta fefc475470 fix(evals): accept exact natural historical delivery dates 2026-10-28 00:25:04 -05:00
Dotta 8b29ca5489 fix(evals): enable real workspace view before lifecycle journeys 2026-10-28 00:21:59 -05:00
Dotta 81bba1dd0d fix(evals): scope design approval claims to contrast clauses 2026-10-27 23:00:30 -05:00
Dotta 89063699af fix(evals): recognize explicit design revision ordering 2026-10-27 23:00:29 -05:00
Dotta 22dd6a1280 test(evals): report effective project-defaults timeout policy 2026-10-27 22:36:30 -05:00
DottaandPaperclip d1f983e395 test(evals): distinguish team pricing from negated per-user charges
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 9434442348 test(evals): reject waived required skills without confusing unrelated history
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 96dcd15449 test(evals): reject modal contradictions of required plan features
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 021c155682 test(evals): ground team requirements in actual skill identities
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 913ffeec7c test(evals): accept substantive single-page plan paraphrases
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 33f99c6c5a docs(evals): preserve the composed operating section heading
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:18:36 -05:00
DottaandPaperclip b797e1cc29 test(evals): read delivered task HTML with the authenticated fixture actor
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:18:00 -05:00
DottaandPaperclip 1ba267de60 test(evals): use production chat timeout within finite cell budgets
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:18:00 -05:00
DottaandPaperclip 0195fcb016 Merge pinned production for the post-custody chat pilot
Preserve frozen 244-scenario YAML intent and 191-capability coverage. Production baseline: 39e811c9c5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:03:19 -05:00
DottaandPaperclip e4b30163bb fix(evals): preserve the complete trusted controller artifact inventory (#17943)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/103/changes/paperclip/103
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages real agent work.
> - Controlled Runner evaluations record model spend and traces.
> - A trusted build creates one portable controller with a complete file
inventory.
> - GitHub artifact upload omitted one hidden dependency metadata file.
> - The next job correctly refused the incomplete inventory before any
paid worker started.
> - This change preserves all inventoried bytes without relaxing
verification.

## Linked Issues or Issue Description

Refs #17889, #17929.

**What happened?**
[Run
40585138437](https://github.com/paperclipai/paperclip/actions/runs/40585138437)
failed at copied controller verification. Its artifact contains 294
files; its signed inventory expects 295. The only missing file is
`server/node_modules/protobufjs/node_modules/@protobufjs/pool/.npmignore`.
No extra files exist. Every paid worker was skipped and no model request
ran.

**Expected behavior**
The trusted controller artifact preserves its full inventory, including
hidden package metadata. The strict verifier rejects any actual change
or missing file.

**Steps to reproduce**
Build the portable runtime, upload the directory with the action's
default hidden-file exclusion, then verify the downloaded artifact
against the original receipt.

## What Changed

- Enable hidden files only on the fresh trusted controller code upload.
- Download that exact upload and run a real artifact-service round-trip
test before any private preparation. The test fails if the upload drops
hidden files again; it tests the actual transport instead of matching a
YAML flag.
- Keep the strict byte inventory and trusted source/receipt checks.
- Document why this code artifact differs from private attempt evidence,
which still uses the protected archive.

## Verification

- Exact retained artifact comparison: one missing `.npmignore`, zero
extra files.
- `actionlint .github/workflows/runner-protocol-live-evals.yml` passed.
- Protocol workflow security tests: 14 passed.
- The artifact-service integration test is explicit-only locally. The
controlled workflow supplies the real downloaded artifact, trusted
source and receipt digest.
- No product runtime, provider routing or grading code changed. CI runs
broader checks. Fresh controlled workflow qualification follows merge.

## Risks

Hidden files are allowed only in the code-only output of the
provider-credential-free build. Raw private evidence and credentials do
not enter that upload. The artifact verifier still requires every file
and SHA-256 from the trusted receipt.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 20:55:18 -05:00
DottaandPaperclip a8bd012635 fix(evals): apply reviewed lock corrections to reports and qualification images (#17937)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/98/changes/paperclip/98
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages agents and records their work.
> - Product E2E campaigns measure real agent behavior and model spend.
> - A paid hosted run completed both cases and all ten model receipts.
> - Its report job could not install dependencies from the committed
lock.
> - The trusted checkout needs a reviewed dependency correction.
> - This change applies a reviewed lock correction before the frozen
report install.

## Linked Issues or Issue Description

Refs: #17928

**Description**
The report in [run
40576181788](https://github.com/paperclipai/paperclip/actions/runs/40576181788)
failed with `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH`. Both paid cases had
passed. The worker and build jobs already restored the resolved source
lock, but the report did not.

**Steps to reproduce**
Dispatch the bounded daily campaign on the default branch with a stale
committed lock configuration. Workers use the resolved lock. The report
uses the stale lock and fails before report generation.

**Expected behavior**
The report installs frozen dependencies from a verified lock for the
exact trusted source. Evidence can be reported without repeating paid
work.

## What Changed

- Apply a reviewed lock correction from the trusted report checkout
before its frozen install. Use the same correction for the qualification
image build, which also requires a compatible frozen lock.
- Pin the original lock, correction and output SHA-256. The trusted
authorization job rejects unknown inputs before any paid work.
- Apply the same correction to trusted manual history publication.
Qualification image builds apply it only when their target equals the
workflow revision; other targets keep their own lock.
- Update existing Claude ACP and postgres patch identities and link the
declared yaml dependency. Existing package tarball integrity entries
stay unchanged; the declared yaml dependency adds exactly yaml@2.9.1
with pinned integrity.
- Keep runtime registry resolution and worker target locks out of the
credential-bearing report.
- Test valid correction, idempotence and changed-lock, changed-patch and
wrong-output rejection.

## Verification

- Focused workflow and lock-correction tests: 36 passed.
- `actionlint .github/workflows/runner-full-stack-e2e.yml` passed.
- Original worker evidence: 2/2 passed, ten final model receipts, zero
unresolved model requests.
- The original report was regenerated from retained evidence with zero
provider calls and saved privately with replay provenance.
- Broader repository checks run in CI. Prior full Product checks passed
for #17928.

## Risks

The correction is tied to the reviewed committed lock. Dependency input
changes need a reviewed correction update. No new registry version or
lifecycle script can enter the privileged report through this fix. The
original failed workflow stays failed. No root lockfile is committed.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 20:55:13 -05:00
DottaandPaperclip 5787962454 fix(evals): report setup-inclusive model token totals (#17946)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/108/changes/paperclip/108
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip Product E2E evaluations run real agent requests.
> - Model accounting includes setup, canceled requests and earlier
attempts.
> - Heartbeat token totals can omit setup requests.
> - The shared gateway journal retains final normalized usage for those
requests.
> - Reports should expose that usage once, with coverage and conflicts
visible.

## Linked Issues or Issue Description

Refs #17928.

**What happened?** The two-case qualification reports showed complete
model dollars from the shared ledger, while their headline heartbeat
tokens omitted setup. For example, the verified Codex campaign has nine
final requests, including setup requests, whose aggregate is only
available in private receipts.

**Expected behavior** The existing ledger report shows all-request token
totals, a setup subtotal and missing/conflicting usage counts. Heartbeat
and request totals remain separate.

**Steps to reproduce** Run a budgeted native marker case with an
asynchronous setup probe, then compare the heartbeat totals with final
request observations in its retained ledger.

## What Changed

- Aggregate final provider/gateway usage once per model request,
including setup and canceled requests. Ignore provisional observations
and infrastructure reservations.
- Expose uncached input, cache-read input, cache-write input and output
separately. Label reasoning as included in output. Preserve agreeing
optional reasoning from partial final receipts only when it fits the
complete output count.
- Check every final provider/gateway observation for conflicting
supplied fields, including partial receipts. Exclude ambiguous requests
and report their conflicting coverage. Missing token usage remains
visible.
- Preserve retained reports without the optional token field; validate
new counts and totals before dashboard regeneration.
- Keep the journal-bound experiment identity stable when adding derived
token presentation, so old accepted experiments can still be linked
without a reimport. Journal changes still invalidate the receipt.

## Verification

- Focused report and scheduled-operations tests: 54 passed.
- Eval TypeScript check passed after refreshing ignored Runner
declaration outputs.
- Regression covers duplicate sources, provisional usage, setup,
canceled requests, missing usage, conflicting final usage,
infrastructure exclusion and invalid retained totals, including nonzero
totals with zero contributing requests. All 32 partial-field subsets are
checked in both observation orders.
- Regenerated both real retained Claude and Codex reports successfully
with all final request tokens and existing experiment receipts, using
zero model calls.
- Broader checks run in CI. No model calls are needed for report
generation.

## Risks

The subtotal requires final normalized gateway/provider token
observations. Older journals and other telemetry sources can have
incomplete coverage, which remains explicit. Token totals do not
establish invoice reconciliation. The change does not alter paid
selection, routing or budgets.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 18:12:20 -05:00
DottaandPaperclip 64a42d2cf0 fix(evals): honor the Daytona setup attempt deadline (#17947)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/110/changes/paperclip/110
<!-- entire-trail-link-end -->

## Thinking Path

> - Budgeted Product E2E cases must attribute asynchronous hello probes
to setup.
> - Daytona can take longer than local startup to provision an isolated
environment.
> - The setup helper imposed a 45-second local limit on both
environments.
> - A qualified remote marker stopped while the agent was still
verifying.
> - Remote readiness should use its existing finite attempt deadline.

## Linked Issues or Issue Description

Refs #17936.

**What happened?** The bounded Daytona marker campaign stopped at the
setup helper after 45 seconds while the agent was still verifying. The
case permits a 15-minute attempt and the environment allows up to five
minutes to provision. It recorded zero model requests; retained cleanup
passed and a provider-side company-label query confirmed no sandbox
remained.

**Expected behavior** Daytona readiness can finish within the existing
attempt budget. Local setup keeps the current short limit; observed
errors, holds and identity changes still stop either environment
immediately.

**Steps to reproduce** Run a budgeted native marker against a cold
Daytona environment whose readiness takes longer than 45 seconds.

## What Changed

- Pass the execution environment to the budgeted setup helper.
- Use the existing finite attempt deadline for Daytona; retain the
45-second local limit.
- Set a five-minute saved-account hello-probe allowance only for
explicit budgeted Daytona fixtures, matching the environment
provisioning allowance. A later qualification exposed the inner probe’s
separate 90-second timeout.
- Keep the gateway in the setup phase until the same agent is ready.
Model limits, attempt limits, cleanup and automatic retries are
unchanged.
- Update the operations guide with the two timing windows.

## Verification

- Focused setup and operations regression suites: 48 passed.
- Eval TypeScript check passed.
- A fixture regression verifies the five-minute probe allowance is
remote-only and preserves gateway credential routing. The timing
regression advances the observed clock past 45 seconds: Daytona reaches
ready while local setup still stops. Existing tests retain
identity/error/hold and attempt-deadline guards.
- Fresh real Daytona qualification follows this correction with one
attempt and the existing $20 stage cap.

## Risks

A failed remote setup can now consume more of its existing attempt
window. It cannot extend that deadline, increase model allowance, or
automatically retry. This change does not claim the prior failed
campaign passed.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 17:52:46 -05:00
DottaandPaperclip d40f9ac377 fix: enforce first-task completion decisions and retain E2E evidence (#17940)
## Thinking Path

> - Paperclip lets people delegate work to agents and approve proposed
work.
> - The onboarding first task asks for scope, proposes a task, and waits
for a human decision.
> - An answer to its opening question supplies scope. It does not
approve a proposal.
> - Task creation already checks the saved proposal decision.
> - Native completion used the general completion policy and could mark
the onboarding parent Done without that decision.
> - This PR checks the trusted decision before completion feedback and
again when the native status is committed.
> - The first task stays in review until a saved human decision permits
closure. Ordinary completion and acknowledged refusal keep their current
behavior.
> - The Product E2E guard also retains closed scalar evidence so future
failures show which boundary condition was observed.

## Linked Issues or Issue Description

Refs: #17914. Profile compatibility follow-up to #17922.

**What happened?**

A native onboarding run can submit a valid low-risk Done result after
only the opening answer. The native finalizer then marks the first task
Done even though no human has accepted or rejected a proposal.

**Expected behavior**

The opening answer must not permit completion. A trusted saved proposal
decision must permit it. A saved rejection permits acknowledgment and
closure, but does not authorize new work. A later pending proposal must
keep the task in review.

**Steps to reproduce**

Create a server-owned onboarding first task with an answered opening
question, no proposal decision, and a valid low-risk completion
contract. Pass a Done result through native completion feedback, the
control-plane port, and the finalizer. Before this fix, the persisted
status is Done. The regression expects In review. Ordinary completion
and acknowledged rejection pass as controls.

## What Changed

- Share the trusted latest-proposal and human-response checks with
completion admission. Keep task creation acceptance-only.
- Refuse premature native completion feedback. Keep an already saved
premature result in review during finalization.
- Check the decision again under the existing task and run locks before
committing Done.
- Cover accepted decisions, saved refusals, forged context, pending or
changed proposals, and a real database race.
- Retain a closed scalar receipt when the existing E2E pre-acceptance
guard fails. Validate and display it without task content, IDs,
filenames, or raw rows. Preserve the original failure, blocked
dependents, and historical reports.
- Include the evidence module in the journey definition hash. Bind the
receipt to its exact failure code and check its closed keys against the
full type.
- Repair base-branch profile drift found by the full harness check:
include the active Copilot v66 fixture in image hashing and update stale
Pi version/hash expectations from #17922. Keep all older image inputs
and strict hash checks.
- Update the onboarding feature and agent access maps and E2E evidence
documentation.

## Verification

- Semantic regression before implementation: one failure and two passing
controls on PostgreSQL 18. The failed assertion observed Done instead of
In review.
- The local database uses an owned disposable container with an
immutable image digest. Actual migrations, feedback, control-plane port,
finalizer, and status commit run against PostgreSQL. A temporary test
adapter replaces only the embedded database transport because this host
has exhausted its PostgreSQL semaphores.
- Native control-plane suite: 55 tests passed, including 17 new
controls. Existing task-creation admission: 25 passed. Existing
completion feedback: 19 passed.
- Credential-free E2E support: 206 Vitest suites passed, 4,407 tests
passed, and two skipped. All 137 Node support checks passed. Harness
typecheck passed.
- The failure-receipt regression had 11 failed assertions before capture
was implemented. Two failure-code binding assertions fail without their
gate. Eight placement/binding cases first validate the receipt-free
control, then reject the misplaced receipt.
- The untouched base reproduced the three profile compatibility
failures. The narrow repair passed all 83 affected tests before the full
harness run.
- Private workflow preparation and credential-free onboarding preview:
nine compatibility checks passed. Ownership, recovery, evidence,
shared-contract and workflow compatibility: 77 passed. The actual
provider preview also confirms immediate success deletion, earlier
declared failure eligibility, limits, and zero retries without
credential access. Map checks and git diff checks passed.
- Required repository typecheck passed for the product fix. The combined
branch build passed. The local full repository suite was stopped after
repeated PostgreSQL startup failures; a separate initdb probe confirmed
host SysV semaphore exhaustion. No shared IPC resources were changed.
[Hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/40591020873)
passed the full database suites, typecheck, build, runner checks,
serialized suites, and browser shards on final head
`1ac4c23605a03bfd1f1f8a017b14ce29fda85f6b` (55 success, two expected
skips, one advisory neutral).
- Independent source review approved the product guard, atomic race
regression, evidence contract, and maps with no actionable findings.
Both automated reviews are 5/5 on final head
`1ac4c23605a03bfd1f1f8a017b14ce29fda85f6b`; prior findings are fixed and
resolved. Final CI is green and the PR is conflict-free.
- No Cloud workspace, provider call, release publication, or deployment
is part of this verification. These tests do not identify a historical
Cloud failure or qualify live onboarding.
- Local dependency installation uses the existing CI-generated lockfile
from #17811. No lockfile change is included.

## Risks

- Existing onboarding runs without a trusted saved proposal decision
remain in review even when the agent reports success. This is the
intended approval boundary.
- Decision changes can race completion. The status commit checks the
decision under its existing locks and uses the existing status-race
recovery path.
- A saved refusal permits closure only. Creation admission still rejects
it.
- Evidence readers reject malformed receipts and unrelated failure
codes. The version-1 key registry is checked against every receipt field
at compile time. Existing attempts do not gain evidence after the fact.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), ultra reasoning, through Codex. Used
repository inspection, code execution, and collaborating coding agents.
The runtime did not report a context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
ticket or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 17:07:50 -05:00
DottaandPaperclip 2c70d04bf6 feat(evals): account for explicit Daytona model requests (#17936)
## Thinking Path

> - Paperclip evals need complete model-spend evidence.
> - The bounded Product E2E controller already owns request admission
and token receipts.
> - Its loopback endpoints cannot serve a Daytona agent.
> - Explicit remote cases therefore need a scoped transport to the same
controller.
> - This PR exposes only authenticated model and OTLP endpoints through
one owned quick tunnel.
> - Remote requests keep the same budget, request identity, usage
journal and cancellation drain.

## Linked Issues or Issue Description

Refs #17928, #17929, #17523.

**What happened?**
The operations manifest and fixture reject Daytona. The parent gateway
and native trace relay bind loopback, so normal remote agent traffic
cannot use their cost accounting.

**Expected behavior**
An explicit native marker or feedback case on Daytona can route requests
through the bounded parent. Upstream provider and Braintrust keys stay
on that controller. Every paid request keeps its usage receipt and the
attempt retains ordinary feedback and cleanup evidence.

**Steps to reproduce**
Use the explicit `remoteGateway: "cloudflare"` manifest option for a
native Codex or Claude Daytona marker/feedback case. Without that
option, the remote selection remains rejected. The parent verifies its
public health nonce before fixture setup.

## What Changed

- Add a deadline-limited bridge with exact model/OTLP paths, separate
scoped tokens, streamed responses and owned tunnel cleanup.
- Start one HTTPS quick tunnel with a private home and no provider
credentials or ambient Cloudflare configuration. Do not retry creation
or paid work.
- Admit only explicit native Daytona marker/feedback cells and require
the provisioning credential before opening the journal.
- Bind fixture model/OTLP endpoints to the assigned remote origin. Keep
provisioning access in the fixture/server controller.
- Preserve final token usage when a remote client disconnects during
shutdown.
- Document the manual route. Daily and weekly selections remain local.

## Verification

- Full Product E2E unit suite: 4,370 passed and 2 skipped across 206
files, plus 137 Node checks.
- Focused remote/usage/operations/trace checks: 71 tests passed; final
bridge/drain checks: 15 passed.
- Product E2E typecheck passed.
- Tests cover missing tunnel executable, expiry, authentication, blocked
paths, token separation, missing provisioning credentials, endpoint
assignment and listener cleanup.
- The cancellation regression uses the real gateway and ledger with a
synthetic upstream. It retains the final receipt once after remote
disconnect.
- Latest source and architecture/cleanup checks: 167 passed; eval
typecheck passed. Both reviews are 5/5 and exact-head CI is green. Live
qualification follows merge.

## Risks

The explicit route sends synthetic eval requests and scoped attempt
tokens through Cloudflare. Those tokens expire with the existing
campaign deadline and budget. Upstream keys stay on the controller.
Public readiness has a finite wait; failure stops before fixture setup.
The operator still needs the existing immutable Daytona image and
verified native artifacts. Source and deterministic tests do not
establish live remote acceptance.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 16:38:23 -05:00
Dotta d4e8906dc3 test(evals): verify rendered references inside saved task comments 2026-10-27 16:10:58 -05:00
Dotta d6f00a8136 test(evals): use genuine human origin for disabled chat probe 2026-10-27 16:10:07 -05:00
Dotta 945674fffe test(evals): retain ordinary catalog chat screenshots privately 2026-10-27 16:06:44 -05:00
Dotta 43aeb5c141 test(evals): accept concealed expired-session reads with full sign-in proof 2026-10-27 16:00:38 -05:00
Dotta 872988beb1 test(evals): reconcile runtime descriptors before campaign admission 2026-10-27 16:00:03 -05:00
Dotta 50d9ecad91 test(evals): distinguish capability-view advice from target selection 2026-10-27 15:35:02 -05:00
Dotta ba87644fcf test(evals): preserve public effective assignment authority for hire denial 2026-10-27 15:29:20 -05:00
Dotta b41ccbe890 test(evals): reject incompatible target recommendations 2026-10-27 15:28:51 -05:00
Dotta acfd0340db test(evals): separate advertised target capability from SSH health 2026-10-27 15:25:13 -05:00
DottaandPaperclip bf3ef12f3a fix(evals): qualify model spend separately from operating costs (#17928)
## Thinking Path

> - Paperclip manages agent work and its budgets.
> - Product E2E campaigns must explain each model request and attempt.
> - The current completion gate also requires complete operating-cost
evidence.
> - Long-lived observability and storage charges can block a campaign
after its model receipts are complete.
> - An operator can now choose model-request readiness in a retained
allowance.
> - The report keeps full accounting, operating holds and model
readiness separate.

## Linked Issues or Issue Description

Refs #17889, #17523, #17452.

**What happened?**
A hosted campaign could pass both product cases and retain complete
model receipts, yet fail readiness because operating charges were
unresolved.

**Expected behavior**
An explicit model-readiness policy permits complete model receipts to
qualify a run. Unknown operating charges stay visible and reserved.
Missing model receipts and budget violations still block qualification.

**Steps to reproduce**
Use a campaign with four unresolved operating reservations and one
completed model request. The regression checks readiness before and
after the final model receipt, then checks an operating overrun. The
default policy retains the original full-accounting gate.

## What Changed

- Add the explicit `accountingReadiness: "model_requests"` allowance
policy. Bind it into operating-bound configuration when present.
- Derive a model subtotal and receipt-coverage projection from the
retained journal. Show it beside full accounting and retained holds.
- Use the selected readiness policy in scheduled finalization. Keep
product, cleanup, feedback, trace and budget checks.
- Refresh two existing eval prerequisites found by the full unit suite:
the active Copilot image identity and the model-account catalog mapping.
- Document the policy and its limit: it does not certify a
full-lifecycle operating-cost ceiling.

## Verification

- Regression failed on the old implementation with `Invalid operating
overhead fields` and passed after the change.
- Focused accounting/operations/finalization checks: 54 tests passed
before the final policy assertions.
- Existing image/catalog failures reproduced, then their 12 checks
passed after the metadata repairs.
- Product E2E unit suite: 4,360 passed and 2 skipped, plus 137 Node
checks. Product E2E and workspace typechecks and workspace build passed.
- Review regressions: 35 focused accounting/finalization checks passed
on 4aeb8bd65c, including the optional-policy hash and the scheduled
consumer.
- Full local server tests encountered embedded PostgreSQL bootstrap
failures; retained output is not a passing full-suite claim. Hosted CI
remains required.
- Final head 4aeb8bd65c: all 57 check runs pass or are skipped, the
legacy security status passes, both reviews are 5/5 and no review
threads remain open.
- No paid campaign or schedule activation is claimed by this PR.

## Risks

The opt-in policy permits readiness while operating costs remain
unreconciled. The report retains full-accounting status and all holds.
The gateway still reserves requests inside the same total allowance.
This is an admission limit, not a vendor invoice guarantee. The omitted
policy keeps existing behavior. Old journals remain unchanged.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 15:10:26 -05:00
DottaandPaperclip 1c72e2d74e test(evals): retain genuine setup health pass evidence
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 15:08:29 -05:00
Dotta 2f68f4ff3c fix(evals): display existing campaign summary inline 2026-10-27 13:35:34 -05:00
DottaandPaperclip b3c77ff304 test(evals): check remote OpenRouter key before setup hello
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 13:35:33 -05:00
DottaandPaperclip 5b54009a03 test(evals): retain frozen chat scope across merged capability maps
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 13:11:41 -05:00
DottaandPaperclip e0b2fec56f Merge pinned master 50b932a142 into preserved chat campaign candidate
Preserve existing campaign measurements and initial ordinals; incorporate already-merged production behavior only. No live qualification claimed.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 13:07:03 -05:00
DottaandPaperclip 78de49f09d test(evals): use ordinary embedded Postgres on AWS chat
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 12:44:48 -05:00
Dotta e68d1a8044 fix(evals): preserve retained model in discovered suite profiles 2026-10-27 12:28:24 -05:00