Commit Graph
322 Commits
Author SHA1 Message Date
DottaandPaperclip 78d91b0e2f fix(evals): admit verified missing personal secret prerequisite
Prove the controller and worker ready before introducing the missing required current-person secret. Preserve that specific scenario only after exact owner-scoped missing-binding resolution; reject unrelated lifecycle, hold, identity and configuration failures. Keep feature grading, human secure entry, native delivery, clocks and privacy assertions unchanged.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:18:31 -06:00
DottaandPaperclip bb33df5320 test(evals): bound wrong-reviewer observation to its case clock
Use the existing remaining original deadline for the wrong-reviewer native reply before the real human denial probe. Preserve reviewer identity, request state, permission and no-work assertions.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:18:31 -06:00
DottaandPaperclip fce64fe2f2 test(evals): keep CLI guidance inside the original clock
Pass the already established case deadline into the two CLI guidance waits and validate it before and after observation. Preserve command grammar, scoped CLI actor and saved-state oracles.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:18:31 -06:00
DottaandPaperclip 9ab0f951a5 test(evals): retain readonly company deadline typing
Set synthetic invalid context values without assigning to the readonly production contract. Runtime observer behavior is unchanged.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 19:57:25 -06:00
DottaandPaperclip 2387c95569 fix(evals): preserve controller secrets during secure entry fixture
Authenticate only exact bootstrap secret references as the pre-entry baseline. Preserve their company ownership, metadata and versions through human deployment-key creation; reject foreign extras or a precompleted target while retaining secure UI, permission, binding and privacy checks.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 19:55:20 -06:00
DottaandPaperclip a5408a736d test(evals): share the original company settings clock
Observe declared company settings native turns within the existing setup-inclusive case deadline instead of the unrelated 150-second helper cap. Keep the original case limits, model, queries and feature oracles.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 19:55:20 -06:00
DottaandPaperclip b2f86ff593 fix(evals): use public membership for incidental human fixtures
Keep invitation enrollment quarantine for invitation journeys. Establish the same real scoped non-admin identities through the existing public prerequisite helper for feedback, timeline, and primary-agent fixtures.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 19:50:51 -06:00
DottaandPaperclip 192d83e828 Fix chat catalog prerequisite clocks and retained evidence (#18377)
## Thinking Path

> - Paperclip evals must prove actual work and retain inspectable
evidence.
> - Project defaults must reach new work while prior work remains
active.
> - A hidden HTTP deadline invalidated that prerequisite before the case
clock.
> - Report rewriting and missing evidence files obscured real outcomes.
> - Observer assumptions also rejected valid history, reads and lease
bookkeeping.
> - This PR repairs those layers and keeps the original feature
requirements and clocks.

## Linked Issues or Issue Description

Refs #18364 for product repairs. Refs #18170 for protected catalog
evidence.

The held worker's native fetch had a five-minute header timeout.
Successful cases lacked required files. Restart compared stored Markdown
to rendered text. Diagnostic redaction changed trusted Playwright
program files, causing the existing producer guard to reject
failed-report retention. Valid dedicated project reads and automatic
environment leases were also misclassified.

This PR targets **`codex/chat-catalog-master-integration`**, not master.
Its base combines master `51ea624344`, preserved catalog `e2b35be76f`
and all product/test changes from #18364, including the native chat
response-stage contract repair. The frozen preservation branch remains
unchanged. The full catalog stays off master. Shipped subscription
aliases retain their 120-second turn limit.

## What Changed

- Hold the exact worker with native HTTP, a one-use agent and the
original absolute deadline. Keep its capability in an owned 0700
directory / 0600 script outside evidence trees. Put only its path in
task/tool inputs. Register the token with the existing secret scanner.
Remove file/listener after normal execution and failed setup.
- Use the already-selected original native case deadline for
prerequisite readiness. Consume only public lifecycle state and fixed
recovery diagnostics, preserve manual-repair/terminal failures, and
record bounded state transitions and recovery flags. Other lanes retain
their authored 45-second setup cap. Nine tests pass actual server
diagnostics through the public agent normalizer.
- Poll canonical source state with the existing settlement predicate;
collect full domain/browser evidence after settlement.
- Capture required `final-state.png` and actual `api-state.json`. Keep
restart logs inside owned roots and recover abnormal evidence from the
actual capture root.
- Compare attributed rendered history before/after restart.
Independently require exact stored bytes, identity and no duplicate
admissions.
- Authenticate and scan canonical Playwright program files with the
existing validator before preserving their exact bytes. Keep ordinary
redaction and every original path, producer, secret, size and transfer
guard. Trusted protocol files remain unchanged.
- Expose only fixed diagnostics, recognized tool names and bounded
counts. Keep raw provider output private.
- Accept correlated `list_projects` reads only when the same native call
delivered the exact full saved target-project defaults. Cover real
OpenCode and server event producers; reject stale, truncated, foreign
and unpaired reads.
- Version project-defaults as **v4**. Accept natural “newly admitted”
future-scope wording; still reject replies missing the future/existing
distinction. Exclude only automatic lease acquisition/release with the
exact `environment_lease` entity type from settings-change rejection.
Retain rejection of actual budget/environment edits, mismatched types
and unknown events. Keep active-work, inheritance and reply checks
strict.

## Verification

**Qualified: [pilot
41256592937](https://github.com/paperclipai/paperclip/actions/runs/41256592937)
passes 3/3, with all 35 independent checks passing.**

| Case | Checks | Full case duration | Evidence and cleanup |
| --- | --- | --- | --- |
| Instance restart | 9/9 | 140.1 s | complete |
| Project defaults | 11/11 | 204.2 s | complete |
| Project work | 15/15 | 204.0 s | complete |

All three are original-attempt results from source
`c51e079f7f2364305b131ffd549e263ff78c5335`, containing the exact product
files from `13c2b61f31d873a475ea0c3c026b19926ff42301`. Actual saved
conversation contracts were inspected, as were ordinary native worker
contracts where present. Each protected archive includes the actual API
state and final PNG; cleanup completes. Original queries,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, finite case clocks,
concurrency one and zero eval retries are preserved. Earlier passes do
not transfer into this qualification.

Both current heads pass all checks: product **62** (56 success, one
neutral, five skipped), catalog **56** (54 success, one neutral, one
skipped). Both have exact-head Greptile **5/5**, zero unresolved review
threads and clean mergeability. The three trusted private-evidence
protocol files remain byte-identical to the dispatch workflow. Raw
evidence stays protected.

- **202 focused catalog/evidence tests and catalog typecheck pass.**
Nine readiness controls consume actual lifecycle diagnostics through the
public agent normalizer, which omits private operation state. Manual
repair, terminal states and unknown errors still fail.
- Real Node 24 control: old fetch fails at 301,464 ms; native HTTP
completes after release at 310,054 ms under the same 340-second control
deadline. Actual native OpenCode/Runner also completes a separate source
while the held worker stays blocked. Failed-setup regressions verify
resource cleanup.
- All **45 privacy/report controls pass**. The old failed-report trace
rewriting reproduces the producer-guard rejection; authenticated
canonical program bytes now survive unchanged. Contaminated assets still
fail. No secret, producer, path, transfer, expansion or size guard is
weakened.
- Oracle **`paperclip.chat-catalog.project-defaults.v4`** retains
active-work, inheritance, actual mutation/read and reply requirements.
Positive and negative controls cover correlated full saved-default
reads, automatic lease bookkeeping and natural “newly admitted” wording.
Original failed grades remain retained alongside explicit versioned
regrades.
- Product validation in #18364 includes 2,044 discovery tests, 112
real-database/HTTP/native-runner/OpenAPI consumer tests, and passing
typecheck/build. Task-owned Docker PostgreSQL unblocked local DB checks
and was removed afterward.

Retained attempt ledger; no earlier result substitutes for the qualified
run:

| Attempt | Recorded outcome / finding |
| --- | --- |
| 41114636294 / 41119215255 / 41203017450 | Initial report/setup and
unavailable-archive failures. Functional partial success did not qualify
the reports. |
|
[41207155025](https://github.com/paperclipai/paperclip/actions/runs/41207155025)
| 0/3. Rendered-Markdown mismatch; other archives unavailable. |
|
[41211541655](https://github.com/paperclipai/paperclip/actions/runs/41211541655)
| Work-only diagnostic: 15/15 with required files and cleanup. |
|
[41212946325](https://github.com/paperclipai/paperclip/actions/runs/41212946325)
| 2/3. Defaults did not settle; transfer failed. |
|
[41216880087](https://github.com/paperclipai/paperclip/actions/runs/41216880087)
| Defaults failed; counters did not identify finish outcomes. |
|
[41219519639](https://github.com/paperclipai/paperclip/actions/runs/41219519639)
| Native work succeeded but grader did not; report assets and expansion
limits rejected transfer. |
|
[41222196525](https://github.com/paperclipai/paperclip/actions/runs/41222196525)
| 1/3. Restart exposed fabricated empty-response success; defaults
exposed backfill and missing dedicated-read recognition. All archives
retained. |
| 41226224666 / 41226710271 | Cancelled before paid cells to repair
generated outputs, credential custody and native event-shape coverage. |
|
[41227965230](https://github.com/paperclipai/paperclip/actions/runs/41227965230)
| 2/3. Defaults 9/11 correctly rejected existing-task backfill and reply
scope. |
|
[41233723348](https://github.com/paperclipai/paperclip/actions/runs/41233723348)
| Restart 9/9. Defaults 10/11: v2 misclassified automatic lease events.
Work was cancelled because replacement was dispatched prematurely; no
work pass inferred. |
|
[41236679567](https://github.com/paperclipai/paperclip/actions/runs/41236679567)
| 1/3. Defaults 11/11. Restart 4/9: no answer. Work failed and the
unchanged ZIP expansion guard rejected full retention; missing outcome
remains inconclusive. |
|
[41240010122](https://github.com/paperclipai/paperclip/actions/runs/41240010122)
| Restart 4/9 under authority guidance: its literal native criterion
still required the unavailable action. Defaults 10/11: v3 rejected valid
“newly admitted” wording; v4 explicitly regrades 11/11 while retaining
the original grade and privacy findings. Work 10/15: created/repaired a
task without final chat reply. All three archives retained. |
|
[41243833425](https://github.com/paperclipai/paperclip/actions/runs/41243833425)
| Source `6fa0c83674` / product `dddd2d5206`: 2/3. Restart 9/9 and work
15/15; defaults failed prerequisite setup before any canonical
query/native run because its 45-second helper cap cut short a normal
60-second probe. |
| 41247798414 | Cancelled before paid cells after review caught a
private-field assumption in the readiness helper. The repair uses public
lifecycle state and fixed recovery diagnostics. |
| 41248428190 | Cancelled before paid cells after the unresolved-thread
sweep caught a conversation identity edge case. The repair uses
canonical identity and carries the user ID through heartbeat. |
|
[41250502608](https://github.com/paperclipai/paperclip/actions/runs/41250502608)
| Source `7b1de24ce1` / product `c0bf249807`: 1/3. Work 15/15. Restart
4/9 despite the repaired contract actually being admitted; it continued
discovery after board-only 403 and had no reply. Nearby HTTP 200 reads
are not call-correlated to the MCP timeouts. Defaults timed out before
settlement (zero finish proposals), then exceeded the unchanged 256 MiB
original-evidence bound; detailed domain behavior remains inconclusive.
|

## Risks

The prerequisite needs native Node in the worker environment. Trusted
program bytes remain subject to the existing authentication and
scanners. The audit exception rejects unknown events. Oversized ZIPs can
still be rejected, as in the retained work attempt; this PR does not
weaken that guard or claim an unobserved outcome. A passing pilot
qualifies the selected attempts, not full-catalog reliability.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, tool use, code
execution and live eval tooling. Exact hosted model revision and
context-window size are not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

- [x] Final same-source three-cell pilot passes 3/3 with required
evidence and cleanup

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-31 19:52:57 -05:00
Dotta 6c1a5b9e07 test(evals): reconcile preserved catalog with current harness contracts 2026-10-31 08:45:24 -05:00
Dotta b1e3950e6d Merge master into the preserved catalog integration candidate
Preserve the frozen catalog and its historical qualification separately. Keep current supervisor ownership, installed-artifact and recovery controls, the shipped TestOps cell, and the historical OpenCode candidate's exact model. Production source matches master f2db909393. Focused support checks: 218 passed; full catalog typecheck passed.
2026-10-31 08:22:25 -05:00
Dotta e2b35be76f preserve reviewed document audience and email task observers 2026-10-28 23:18:12 -05:00
Dotta 372b192327 Preserve reviewed SDK and task prerequisite eval corrections 2026-10-28 21:28:32 -05:00
DottaandPaperclip 24e236c0f5 Preserve reviewed chat catalog eval source without production candidates
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 20:31:12 -05:00
Dotta ed762ce875 Bind ambiguous-task clarification to visible linked question cards 2026-10-28 14:37:29 -05:00
Dotta bcdcd0b242 test(evals): require fresh notification comment boundary 2026-10-28 13:30:48 -05:00
Dotta 1b7e97ef16 fix(evals): bind public notification events to saved comments 2026-10-28 13:30:48 -05:00
Dotta 7ef235b063 fix(evals): collect bound conversation project creation links 2026-10-28 13:24:43 -05:00
Dotta 104a23be5f fix(evals): retain completion stability window within original handoff deadline 2026-10-28 11:21:06 -05:00
Dotta c455a206a8 fix(evals): preserve original native observation clocks for eligible historical finals 2026-10-28 11:15:52 -05:00
Dotta 3e1c87117e fix(evals): seed active prerequisite tasks with valid owners 2026-10-28 11:03:18 -05:00
Dotta b740827954 test(evals): refresh pricing and artifact valid observations 2026-10-28 10:33:20 -05:00
Dotta 22dca95926 fix(evals): avoid cancelling consumed privacy response bodies 2026-10-28 10:22:02 -05:00
Dotta cb31ebccf0 test(e2e): preserve secret proposal original observation clock 2026-10-28 09:57:44 -05:00
Dotta e386933481 test(e2e): bind secret proposal review to expanded card 2026-10-28 09:46:16 -05:00
Dotta 8bbf313552 test(e2e): keep long-thread measurements within original clock 2026-10-28 09:46:16 -05:00
Dotta 7170ca4b81 test(e2e): preserve original approval and session clocks 2026-10-28 09:12:28 -05:00
Dotta a477b2c5d8 fix(evals): preserve original task and inbox observation budgets 2026-10-28 08:17:14 -05:00
Dotta efa3382b21 test(e2e): fingerprint executed plugin CLI source and loader 2026-10-28 08:15:28 -05:00
Dotta 250ba62791 test(e2e): bind installed team to selected OpenCode runtime 2026-10-28 08:15:27 -05:00
DottaandPaperclip 1c6021a4e6 fix(evals): bind inline secret helper to actual consumer task
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 08:15:27 -05:00
DottaandPaperclip 0e4ebca61c fix(evals): bind approval wait to actual task workspace
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 08:15:27 -05:00
DottaandPaperclip dea44755b0 fix(evals): prepare prerequisite people through public membership routes
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 08:15:26 -05:00
Dotta 8cd9e8bc6b fix(evals): keep fast response initial observation on original clock 2026-10-28 07:26:09 -05:00
DottaandPaperclip 1c7cbdc25b test(evals): preserve artifact outcome without extra task status gate
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 07:20:35 -05:00
DottaandPaperclip 3dab66775d fix(evals): bind answer race completion and ordinary draft evidence
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 07:20:35 -05:00
Dotta af01c4f6d3 fix(evals): await fast response save before model readback 2026-10-28 07:19:59 -05:00
Dotta 9ea0c8a61d fix(evals): retain original clock for seven prepared human journeys 2026-10-28 02:31:43 -05:00
Dotta d282ae2eee fix(evals): share original observation clock across selected native phases 2026-10-28 02:01:44 -05:00
Dotta 81abf04b21 fix(evals): qualify owned OpenCode post-stop proxy cleanup 2026-10-28 01:32:20 -05:00
Dotta 15fb2318bf fix(evals): preserve human origin on consumed queue probes 2026-10-28 01:32:20 -05:00
Dotta b70439f387 fix(evals): keep deadline boundary timeout strictly positive 2026-10-28 01:32:20 -05:00
Dotta b0eff403b8 fix(evals): share original budget for instance and confirmation waits 2026-10-28 01:32:19 -05:00
Dotta e232200747 test(e2e): pin configurable auxiliary tests to requested model 2026-10-28 00:55:14 -05:00
Dotta 1e686fa0f3 fix(evals): bind routine failure evidence to selected profile 2026-10-28 00:53:18 -05:00
Dotta 78783dda57 fix(evals): bind routine history grading to observed provider 2026-10-28 00:47:54 -05:00
Dotta fa7304ae5e test(e2e): share original budget across company followups 2026-10-28 00:37:55 -05:00
Dotta 6eca7f6bcc test(e2e): verify returned historical artifact copies 2026-10-28 00:25:04 -05:00
Dotta 2c6d746015 fix(evals): retain contracted delivery negation in date grading 2026-10-28 00:25:04 -05:00
Dotta fefc475470 fix(evals): accept exact natural historical delivery dates 2026-10-28 00:25:04 -05:00
Dotta 8b29ca5489 fix(evals): enable real workspace view before lifecycle journeys 2026-10-28 00:21:59 -05:00