PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-02 16:58:52 +01:00

Commit Graph

Author	SHA1	Message	Date
DottaandPaperclip	9e1735ec17	test(evals): add a fixed TestOps AWS chat feature route (#18322 ) ## Thinking Path > - Paperclip lets people manage AI agents and inspect their saved work. > - Chat feature evaluations test a real request against saved tasks, audit records, links and reload behavior. > - The current suite has Codex and OpenCode cells but no dedicated runnable DeepSeek 1017 backlog plan in TestOps. > - TestOps must select the real case before the workflow can access a model key. > - This PR adds one fixed, default-branch AWS route for native OpenCode and DeepSeek 1017. > - The route keeps original evidence private and sends only a verified result and approved screenshots to TestOps. ## Linked Issues or Issue Description Subsystem affected Product E2E evaluation definitions and GitHub Actions delivery. Problem or motivation A dedicated TestOps plan cannot run the existing backlog chat journey with native OpenCode. A result must prove saved work and cleanup rather than report that a model call completed. Proposed solution Add one OpenCode backlog cell and one fixed public workflow. TestOps selection validation runs in a token-isolated job before builds or inference. The workflow retains private originals and reports the bound outcome to the selected TestOps case. Alternatives considered A private-repository wrapper would need a new cross-repository GitHub write credential. The new route uses the existing TestOps GitHub integration instead. Existing manual candidate approval paths remain separate. Roadmap alignment This extends the existing feature evaluation infrastructure. It changes no production feature, global runtime default or model default. Current main #18319 supplies the v2 grader, delegation queue accounting and final screenshot capture; these improvements are preserved. Related product work: #16814 and #17126 concern backlog handling. This PR changes neither product implementation and does not claim those changes are required. ## What Changed - Add exactly one native OpenCode backlog cell with DeepSeek 1017 under the separate `runner-opencode-deepseek-1017` profile. Preserve all four existing Codex and production OpenCode cells and their models. - Add a default-branch-only workflow with fixed evaluation, zero-result registration and one plan-launch modes. Require GitHub actor checks and authenticated singleton selection before inference. - Add a separate shipped-main evidence authority. Reuse the existing strict private storage, scanners, inventories and scoped AWS sessions. - Bind sanitized results, generated JUnit and producer-approved screenshots to the exact source, run, attempt and case. Keep failed grades and cleanup failures failed. - Add TestOps API, CLI and result adapters with pure controls and normal unit-test wiring. Keep the reporting token out of the evaluated process. - Verify the current server case result and attachment metadata after upload, without masking an earlier failed upload or evaluation. - Apply only the existing reviewed lock correction, verify unchanged tracked dependency inputs and bind the lock receipt across the compiled-output handoff before frozen installation. - Bind the initial administrator launch to one approved workflow run number and live GitHub source/actor metadata before TestOps authentication. Use bounded REST case reconciliation and only the reviewed existing GitHub integration. - Make two existing CI tests observe their intended state deterministically: a fake clock for copy-feedback lifetime and a completed-pass barrier for retry-cancellation batches. Production behavior and the original assertions remain unchanged. The two test-only fixes are intentionally included because their observation boundaries blocked this delivery’s required normal CI. They preserve every original assertion and change no production behavior. The original failed runs remain recorded, and the exact causes of those observations remain unknown. ## Verification - 157 focused profile, catalog, oracle, filesystem admission, export, private-path and workflow controls passed on the current-main composition. The existing stock OpenCode models and the new exact DeepSeek 1017 selection are checked separately. - 44 API, CLI, filesystem and post-upload readback controls passed, including foreign selection, changed hashes, unsafe files, failed outcomes, expired readback and no-resend behavior. - The new exact private session and trace layout controls produced four genuine parent failures and pass with the selected-path extension. The frozen 244-case candidate approval registry remains unchanged. - The lock guard has seven filesystem controls plus seven existing canonical-owner controls. The unrestricted old workflow failed its actual policy calibration; the corrected frozen-install route passed. - Eval-owning TypeScript checks, workflow lint and Git whitespace checks passed. The separate UI test correction was checked with the focused jsdom unit suite. The server test barrier has pure wrapper and real scheduler controls; its DB-backed route was not run locally. No additional local typecheck/build was run for these test-only changes. - Full native builds, provider execution, app/database behavior and TestOps registration are reserved for GitHub Actions/AWS. They have not run locally. - The initial normal CI run failed the new-workflow policy allowlist and a heartbeat scheduling assertion. The policy fix has an actual parent red/current green control and rejects five unsafe route variants. Heartbeat source is unchanged and its failure cause remains unknown. The original run is retained. - The launch-fence actual caller fails against the published parent and passes the five own-run contradiction controls before authentication. - The prior `a5aa359d` normal CI completed successfully. The complete `f9e1f289` run passed all leaf jobs except an existing copy-feedback UI assertion, plus its dependent verification gate. The four owning UI files were unchanged from the earlier passing head and actual tested merge; the exact scheduling cause remains unknown. - The correction changes only that test to actual React `act` and a scoped fake clock. All eight component tests pass; early-hide and never-clear source variants each fail the intended lifetime assertion. Production clipboard/card behavior is unchanged. - The `2f27f3fc` run passed the corrected UI shard and every leaf job except the existing retry-cancellation intermediate-count assertion, plus its dependent verification gate. The seven relevant owners match the earlier passing run and historical source comparison; the exact CI scheduling cause remains unknown. - The test-only barrier observes a persisted completed worker pass before asserting the unchanged 20-item cap and two-company fairness, then releases it for the original 29-item drain and 180-second idle check. Two extracted-wrapper controls and three actual scheduler controls pass. The real DB-backed test awaits AWS qualification. - On `836e80ac`, both repaired test shards passed in AWS. The original attempt retained a separate E2E cleanup HTTP 500 from a PostgreSQL lock timeout and one runner shutdown. No production repair or precise contention/shutdown cause is claimed. - Exactly one targeted retry of E2E shard 3 and its aggregate gate passed in attempt 2; exactly one recovery of the interrupted server shard and its verification gate passed in attempt 3. Other passed jobs were carried forward. The final normal run is successful. No whole-workflow retry or evaluation/model retry was launched. - Both literal required checks (`ci / verify` and `ci / e2e`) are successful on the exact final source `836e80ac`. Greptile is 5/5 and no review threads remain open. Actual TestOps registration and the OpenCode/DeepSeek feature result remain unqualified until their separate executions. ## Risks - Actual zero-result registration, tenant API shapes and public-repository dispatch access must be checked before the first evaluation. Missing or ambiguous metadata stops the route. - The real OpenCode/DeepSeek backlog result and Allure outcome remain unqualified until that run. No passing result is seeded. - One human query, one accounted work run, 120-second turn, 420-second attempt, 300-second settled-state bound and zero retries remain unchanged. Setup usage is outside work accounting and stays unknown. - The reporting token was configured through the separately reviewed protected credential bootstrap. It is unavailable to AWS evaluation jobs. - Metadata reconciliation is capped at ten pages, 1,000 cases and four parallel detail reads. Missing or inconsistent metadata stops before inference. The initial administrator launch counter must be created once after registration; it does not restrict normal TestOps Run actions. - Existing manual workflows, historical evaluation grades and candidate approval records remain unchanged. The original CI failures are preserved; the test corrections do not establish retrospective causes. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning and tool use. The runtime did not expose its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-30 15:08:16 -05:00
DottaandPaperclip	cd0c86e5e1	fix(evals): support private evidence for the frozen chat catalog (#18170 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E tests need to retain evidence from both successful and failed work. > - Manual private runs use the existing protected AWS evidence route. > - Its fixed eight-case registry blocks other scenarios in the frozen chat catalog. > - Failed Playwright cells also produce static trace files that need exact owner and content checks. > - This pull request admits the fixed catalog, keeps eight cells per run, and retains the existing report identities. > - Exact source and run approval still controls each private operation. ## Linked Issues or Issue Description Refs: #18134, #17888. What happened? The manual private route accepts only its first eight execution IDs. It rejects other IDs from the frozen 244-scenario chat catalog before a run can start. The dotted `area.case` report identity also fails the retained report parser. A failed Playwright cell can produce canonical trace assets that the private evidence owner does not yet recognize. Expected behavior The trusted route must recognize exactly the frozen 244 OpenCode/local execution IDs. Each original run must still select at most eight cells and have exact protected approval. The evidence owners must retain complete scanned originals from the actual pinned report producer. Unknown carriers and credential contamination must still fail closed. Steps to reproduce Use a known frozen chat ID outside the first eight with the current manual classifier. The classifier rejects it. Parse a dotted chat execution ID with the current report parser. The parser rejects it. Generate a synthetic failed Node-only Playwright test and pass its actual trace/report tree through the private evidence owners. This change includes the narrow parser repair from the existing open #17888. It does not merge or close that PR. The related source work was checked for overlap before this branch was created. ## What Changed - Replace the small private registry with the exact frozen 244 execution IDs. Preserve the two existing undotted standalone task IDs. - Keep the private run limit at eight through classification, approval rows, cell binding and transfer sessions. Preserve nonprivate selector behavior. - Bind private cell report assets to the actual installed pinned producer, selected cell, canonical path and exact bytes before the existing scans and archive checks. Pass that verified owner into both real launcher packaging calls. - Exclude installed build staging from the private evidence walk. Retain the real standard-file cap and reject unknown nested carriers. - Add the existing dotted chat report grammar. Preserve other report families and legacy task IDs. - Use the existing public announcement dismissal control in one external-link/upload E2E test so a real announcement cannot cover its composer button. Preserve the open menu and existing timeout. - Document source compatibility and operator gates. No workflow, product, dependency, lockfile, case callback, query, clock or budget changes. ## Verification - 132 distinct focused controls are closed. The current affected owner suite passes 39 controls under a nondefault temporary directory. The other 93 unchanged controls retain their previous passes. The initial registry, map-order and test-tooling failures remain recorded. - Two actual launcher call-expression controls fail on the prior source because canonical trace files are dropped. Both pass after the private-owner wiring. These controls do not claim full launcher lifecycle or actual E2E qualification. - Temporary paths use the real operating-system temporary directory, including the handoff check. - The controls check all 244 IDs through real private routing, cell binding and original-run approval. They reject unknown IDs, aliases, wrong profiles, traversal, private duplicates and excess private cells. - The controls preserve actual nonprivate 33-ID and duplicate selector behavior. - A real synthetic Node-only failed Playwright test produces the report and automatic trace. The control uses actual package, seal, archive, receipt and restored validation paths with mock AWS transport only. - Owning TypeScript `--noEmit` check passes. No local product server, browser, model, database, SDK build or evaluation was run. - Prior-head CI run 40906120388 failed one E2E test because the real announcement footer intercepted its Add to composer click. The other 28 tests in that shard passed. The narrow fixture uses the existing UI dismissal pattern. The added handler passes an isolated noEmit check against the real Playwright Page type. An additional strict check of the whole browser test fails on five unchanged shared-fixture type errors; checking that unchanged shared file alone gives identical diagnostics. The initial shell exit was misreported and corrected in the validation record. Browser verification is deferred to the next normal CI run; no local browser or server was started. - Repository-wide local build and test commands were not run because this is a scoped source change. Normal required PR checks will provide the repository CI result. ## Risks A registry entry is not proof that a target or feature is ready. A future target must use the exact matching private helper and producer contract, retain any stronger target scanners, and produce fresh source fingerprints. Existing running jobs keep their pinned workflow and target sources. All original attempt limits, declared inputs, private storage checks, fatal scanners and exact administrator-controlled run approval remain in force. The new cell report contract accepts only canonical assets from the actual pinned Playwright installation. Other assets and layouts still fail closed. No private evidence or generated report is included in this PR. ## Model Used OpenAI GPT-6 through Codex, with tool use and code execution. The runtime does not expose a finer model deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 11:42:35 -05:00
DottaandPaperclip	018c4c9c9b	Retain approved manual E2E evidence in private history (#18134 ) ## Thinking Path > - Paperclip lets people manage AI agents and their work. > - The manual AWS E2E workflow records browser and model evidence. > - Public repository Actions artifacts are not a restricted evidence store. > - Some chat evidence needs the existing private-history store. > - A target flag must not grant archive access or enable a public fallback. > - This pull request binds restricted runs to protected exact-source approval and retains their standard evidence with trusted transfer code. > - Operators can retain complete evidence without changing scanner verdicts or scenario outcomes. ## Linked Issues or Issue Description What existing behavior does this improve? Manual AWS E2E evidence retention for exact reviewed chat-feature selections. Subsystem affected The runner E2E manual workflow, standard evidence handoff and private-history transfer. Current behavior Manual cell and merged evidence use Actions artifacts. In a public repository, repository readers can download those files. Scheduled private history does not authorize a manual target to use its role. Proposed behavior Classify sensitive selections in the trusted workflow. Require one protected original-run approval. Verify existing private storage before target setup or model secrets. Retain scanned standard originals and final packaged results through a workflow-pinned runtime. Deny public artifact and history fallback on failure. Reason and benefit Preserve inspectable standard evidence in restricted storage without treating a private bucket as permission to retain credentials. Additional context Related source work is separate: #18067 changes paid runner routing, #18100 changes scoped input delivery, and #17888 changes report catalog parsing. This pull request does not include or revert those changes. ## What Changed - Add a trusted manual classification and exact run, attempt, workflow, target, selection and launcher approval contract. - Add a protected manual private job. Prepare a second workflow-pinned checkout and isolated publication dependencies before archive credentials. - Seal standard originals and final result bytes with exact inventories. Reject credential contamination, unscanned carriers and changed files. - Use existing immutable private-history archives and transfer receipts for cells and merged reports. Keep the existing bounds. - Use per-cell exact-key read sessions and a separate report writer. Check every inline policy against the 2,048-character STS limit. - Scan and seal fresh generated report logs and files with the actual report-step credential roots before transfer. - Require a positive nonprivate route for Actions uploads and public history. Private failures have no public fallback. Preserve normal catalog semantics for nonprivate selections with more than 32 IDs or duplicate IDs. - Authenticate generated-report fonts and every reserved trace-viewer static asset against the exact trusted producer path and bytes, including HTML and JavaScript. Keep standard cell evidence and unknown carriers closed. - Document the first eight supported IDs, deployment gates and the operator trust boundary. ## Verification - 57 new pure, actual-helper and actual-workflow-shell controls pass. - 50 affected existing workflow, routing, target and private-history controls pass. The final committed workflow security controls also pass. - Targeted helper/test and actual launcher noEmit checks pass. - `actionlint -shellcheck= -pyflakes= .github/workflows/runner-full-stack-e2e.yml` passes. - Parent `2f46336d7` required CI is green. The server 11/12 shard first failed on database cleanup/test timing and passed one bounded recheck with unchanged source. The original failure remains recorded. - The preserved docs-only parent `d721c170f` had 59 terminal, nonblocking checks and fresh Greptile 5/5. Three later review comments exposed a genuine static-report asset failure and nonprivate selector regression; their parent counterexamples remain recorded. - The reviewed parent passes 76 focused controls across nine files. The final dependency preflight successor passes all six affected controls, including two new actual `prepare` subprocess controls, and targeted helper/test noEmit. Unchanged suites were not repeated. A synthetic Node-only Playwright HTML/blob/JUnit report and the real dashboard renderer/brand stager pass the actual sealer, strict archive and immutable receipt path; only AWS transport is synthetic. No browser or product setup runs in that test. - Default cell fonts/SVGs, unknown report files, forged paths, altered static bytes, recomputed inventory tampering and registered credential matches remain rejected. Eligibility and remaining-byte checks run before body reads in both sealer and transfer validator; both independent parent read-order failures are preserved. - The pre-credential target comparison also binds the new generated-report asset helper byte for byte. A genuine tampered-dependency parent failure is preserved. - Six additional actual sealer/validator controls expose the standard-extension trace-owner bypass on the parent. All 34 affected controls pass on its correction, including the real producer/archive/receipt test; targeted noEmit passes. Ordinary dynamic report HTML outside the reserved trace directory is unchanged. - Final head `43683c4e0` passes both required gates, `ci / verify` and `ci / e2e`, in normal CI run `40849916302` attempt 1. All 59 rollup checks/statuses are terminal and nonblocking (55 success, two neutral, two skipped). - Fresh exact-head Greptile is 5/5; automated review is terminal and all technical threads are resolved. Human CODEOWNER approval remains a separate gate. Earlier source, genuine counterexamples and CI history remain preserved. - Full local build and live E2E were not run. This task prohibits local SDK/native/server builds and paid or AWS operations. Normal pull request CI must pass before merge readiness. - No existing artifact bodies, archives, media or provider reasoning were inspected. Synthetic owned fixtures prove the privacy boundaries. ## Risks - Deployment still needs the existing private bucket, role and protected environments. Both environments need the exact original-run approval variable. - Targets need the compatible handoff protocol and a separately reviewed exact launcher digest. Existing target branches do not become ready automatically. - Complete duplicated originals and packaged evidence can exceed existing archive limits. Such a transfer fails; files are not silently removed. - Required evidence omissions remain fatal to PASS. Safe failed evidence is retained with an explicit missing list and incomplete required-evidence verdict. Credential contamination still blocks all original uploads. Restricted storage does not waive PNG, HTML, blob or scanner checks. - This is operator-approved source execution. Target processes can inherit the job's OIDC request environment and may re-assume a role. Inline session attenuation is not isolation from malicious approved source. - Operator authorization must bind the actual exact source and original run. A proven administrator-controlled record may supply it without an environment reviewer UI. No new approval UI or environment change is required by this PR. - Required repository CODEOWNER review is still pending. Commented reviews and review requests are not approvals. This PR stays open and unmerged. - This PR does not deploy infrastructure, change IAM, dispatch workflows, publish report payloads, change grades or transfer prior scenario passes. ## Model Used OpenAI `gpt-6.1-sol` in Codex. Used code execution, source analysis and tool calls. The host does not expose the context window size. Independent peer review used the same model family in a separate agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 08:51:40 -05:00

Author

SHA1

Message

Date

DottaandPaperclip

9e1735ec17

test(evals): add a fixed TestOps AWS chat feature route (#18322 )

## Thinking Path

> - Paperclip lets people manage AI agents and inspect their saved work.
> - Chat feature evaluations test a real request against saved tasks,
audit records, links and reload behavior.
> - The current suite has Codex and OpenCode cells but no dedicated
runnable DeepSeek 1017 backlog plan in TestOps.
> - TestOps must select the real case before the workflow can access a
model key.
> - This PR adds one fixed, default-branch AWS route for native OpenCode
and DeepSeek 1017.
> - The route keeps original evidence private and sends only a verified
result and approved screenshots to TestOps.

## Linked Issues or Issue Description

**Subsystem affected**

Product E2E evaluation definitions and GitHub Actions delivery.

**Problem or motivation**

A dedicated TestOps plan cannot run the existing backlog chat journey
with native OpenCode. A result must prove saved work and cleanup rather
than report that a model call completed.

**Proposed solution**

Add one OpenCode backlog cell and one fixed public workflow. TestOps
selection validation runs in a token-isolated job before builds or
inference. The workflow retains private originals and reports the bound
outcome to the selected TestOps case.

**Alternatives considered**

A private-repository wrapper would need a new cross-repository GitHub
write credential. The new route uses the existing TestOps GitHub
integration instead. Existing manual candidate approval paths remain
separate.

**Roadmap alignment**

This extends the existing feature evaluation infrastructure. It changes
no production feature, global runtime default or model default.

Current main #18319 supplies the v2 grader, delegation queue accounting
and final screenshot capture; these improvements are preserved. Related
product work: #16814 and #17126 concern backlog handling. This PR
changes neither product implementation and does not claim those changes
are required.

## What Changed

- Add exactly one native OpenCode backlog cell with DeepSeek 1017 under
the separate `runner-opencode-deepseek-1017` profile. Preserve all four
existing Codex and production OpenCode cells and their models.
- Add a default-branch-only workflow with fixed evaluation, zero-result
registration and one plan-launch modes. Require GitHub actor checks and
authenticated singleton selection before inference.
- Add a separate shipped-main evidence authority. Reuse the existing
strict private storage, scanners, inventories and scoped AWS sessions.
- Bind sanitized results, generated JUnit and producer-approved
screenshots to the exact source, run, attempt and case. Keep failed
grades and cleanup failures failed.
- Add TestOps API, CLI and result adapters with pure controls and normal
unit-test wiring. Keep the reporting token out of the evaluated process.
- Verify the current server case result and attachment metadata after
upload, without masking an earlier failed upload or evaluation.
- Apply only the existing reviewed lock correction, verify unchanged
tracked dependency inputs and bind the lock receipt across the
compiled-output handoff before frozen installation.
- Bind the initial administrator launch to one approved workflow run
number and live GitHub source/actor metadata before TestOps
authentication. Use bounded REST case reconciliation and only the
reviewed existing GitHub integration.
- Make two existing CI tests observe their intended state
deterministically: a fake clock for copy-feedback lifetime and a
completed-pass barrier for retry-cancellation batches. Production
behavior and the original assertions remain unchanged.

The two test-only fixes are intentionally included because their
observation boundaries blocked this delivery’s required normal CI. They
preserve every original assertion and change no production behavior. The
original failed runs remain recorded, and the exact causes of those
observations remain unknown.

## Verification

- 157 focused profile, catalog, oracle, filesystem admission, export,
private-path and workflow controls passed on the current-main
composition. The existing stock OpenCode models and the new exact
DeepSeek 1017 selection are checked separately.
- 44 API, CLI, filesystem and post-upload readback controls passed,
including foreign selection, changed hashes, unsafe files, failed
outcomes, expired readback and no-resend behavior.
- The new exact private session and trace layout controls produced four
genuine parent failures and pass with the selected-path extension. The
frozen 244-case candidate approval registry remains unchanged.
- The lock guard has seven filesystem controls plus seven existing
canonical-owner controls. The unrestricted old workflow failed its
actual policy calibration; the corrected frozen-install route passed.
- Eval-owning TypeScript checks, workflow lint and Git whitespace checks
passed. The separate UI test correction was checked with the focused
jsdom unit suite. The server test barrier has pure wrapper and real
scheduler controls; its DB-backed route was not run locally. No
additional local typecheck/build was run for these test-only changes.
- Full native builds, provider execution, app/database behavior and
TestOps registration are reserved for GitHub Actions/AWS. They have not
run locally.
- The initial normal CI run failed the new-workflow policy allowlist and
a heartbeat scheduling assertion. The policy fix has an actual parent
red/current green control and rejects five unsafe route variants.
Heartbeat source is unchanged and its failure cause remains unknown. The
original run is retained.
- The launch-fence actual caller fails against the published parent and
passes the five own-run contradiction controls before authentication.
- The prior `a5aa359d` normal CI completed successfully. The complete
`f9e1f289` run passed all leaf jobs except an existing copy-feedback UI
assertion, plus its dependent verification gate. The four owning UI
files were unchanged from the earlier passing head and actual tested
merge; the exact scheduling cause remains unknown.
- The correction changes only that test to actual React `act` and a
scoped fake clock. All eight component tests pass; early-hide and
never-clear source variants each fail the intended lifetime assertion.
Production clipboard/card behavior is unchanged.
- The `2f27f3fc` run passed the corrected UI shard and every leaf job
except the existing retry-cancellation intermediate-count assertion,
plus its dependent verification gate. The seven relevant owners match
the earlier passing run and historical source comparison; the exact CI
scheduling cause remains unknown.
- The test-only barrier observes a persisted completed worker pass
before asserting the unchanged 20-item cap and two-company fairness,
then releases it for the original 29-item drain and 180-second idle
check. Two extracted-wrapper controls and three actual scheduler
controls pass. The real DB-backed test awaits AWS qualification.
- On `836e80ac`, both repaired test shards passed in AWS. The original
attempt retained a separate E2E cleanup HTTP 500 from a PostgreSQL lock
timeout and one runner shutdown. No production repair or precise
contention/shutdown cause is claimed.
- Exactly one targeted retry of E2E shard 3 and its aggregate gate
passed in attempt 2; exactly one recovery of the interrupted server
shard and its verification gate passed in attempt 3. Other passed jobs
were carried forward. The final normal run is successful. No
whole-workflow retry or evaluation/model retry was launched.
- Both literal required checks (`ci / verify` and `ci / e2e`) are
successful on the exact final source `836e80ac`. Greptile is 5/5 and no
review threads remain open. Actual TestOps registration and the
OpenCode/DeepSeek feature result remain unqualified until their separate
executions.

## Risks

- Actual zero-result registration, tenant API shapes and
public-repository dispatch access must be checked before the first
evaluation. Missing or ambiguous metadata stops the route.
- The real OpenCode/DeepSeek backlog result and Allure outcome remain
unqualified until that run. No passing result is seeded.
- One human query, one accounted work run, 120-second turn, 420-second
attempt, 300-second settled-state bound and zero retries remain
unchanged. Setup usage is outside work accounting and stays unknown.
- The reporting token was configured through the separately reviewed
protected credential bootstrap. It is unavailable to AWS evaluation
jobs.
- Metadata reconciliation is capped at ten pages, 1,000 cases and four
parallel detail reads. Missing or inconsistent metadata stops before
inference. The initial administrator launch counter must be created once
after registration; it does not restrict normal TestOps Run actions.
- Existing manual workflows, historical evaluation grades and candidate
approval records remain unchanged. The original CI failures are
preserved; the test corrections do not establish retrospective causes.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning and tool use. The
runtime did not expose its context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-30 15:08:16 -05:00

DottaandPaperclip

cd0c86e5e1

fix(evals): support private evidence for the frozen chat catalog (#18170 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Product E2E tests need to retain evidence from both successful and
failed work.
> - Manual private runs use the existing protected AWS evidence route.
> - Its fixed eight-case registry blocks other scenarios in the frozen
chat catalog.
> - Failed Playwright cells also produce static trace files that need
exact owner and content checks.
> - This pull request admits the fixed catalog, keeps eight cells per
run, and retains the existing report identities.
> - Exact source and run approval still controls each private operation.

## Linked Issues or Issue Description

Refs: #18134, #17888.

**What happened?**

The manual private route accepts only its first eight execution IDs. It
rejects other IDs from the frozen 244-scenario chat catalog before a run
can start. The dotted `area.case` report identity also fails the
retained report parser. A failed Playwright cell can produce canonical
trace assets that the private evidence owner does not yet recognize.

**Expected behavior**

The trusted route must recognize exactly the frozen 244 OpenCode/local
execution IDs. Each original run must still select at most eight cells
and have exact protected approval. The evidence owners must retain
complete scanned originals from the actual pinned report producer.
Unknown carriers and credential contamination must still fail closed.

**Steps to reproduce**

Use a known frozen chat ID outside the first eight with the current
manual classifier. The classifier rejects it. Parse a dotted chat
execution ID with the current report parser. The parser rejects it.
Generate a synthetic failed Node-only Playwright test and pass its
actual trace/report tree through the private evidence owners.

This change includes the narrow parser repair from the existing open
#17888. It does not merge or close that PR. The related source work was
checked for overlap before this branch was created.

## What Changed

- Replace the small private registry with the exact frozen 244 execution
IDs. Preserve the two existing undotted standalone task IDs.
- Keep the private run limit at eight through classification, approval
rows, cell binding and transfer sessions. Preserve nonprivate selector
behavior.
- Bind private cell report assets to the actual installed pinned
producer, selected cell, canonical path and exact bytes before the
existing scans and archive checks. Pass that verified owner into both
real launcher packaging calls.
- Exclude installed build staging from the private evidence walk. Retain
the real standard-file cap and reject unknown nested carriers.
- Add the existing dotted chat report grammar. Preserve other report
families and legacy task IDs.
- Use the existing public announcement dismissal control in one
external-link/upload E2E test so a real announcement cannot cover its
composer button. Preserve the open menu and existing timeout.
- Document source compatibility and operator gates. No workflow,
product, dependency, lockfile, case callback, query, clock or budget
changes.

## Verification

- 132 distinct focused controls are closed. The current affected owner
suite passes 39 controls under a nondefault temporary directory. The
other 93 unchanged controls retain their previous passes. The initial
registry, map-order and test-tooling failures remain recorded.
- Two actual launcher call-expression controls fail on the prior source
because canonical trace files are dropped. Both pass after the
private-owner wiring. These controls do not claim full launcher
lifecycle or actual E2E qualification.
- Temporary paths use the real operating-system temporary directory,
including the handoff check.
- The controls check all 244 IDs through real private routing, cell
binding and original-run approval. They reject unknown IDs, aliases,
wrong profiles, traversal, private duplicates and excess private cells.
- The controls preserve actual nonprivate 33-ID and duplicate selector
behavior.
- A real synthetic Node-only failed Playwright test produces the report
and automatic trace. The control uses actual package, seal, archive,
receipt and restored validation paths with mock AWS transport only.
- Owning TypeScript `--noEmit` check passes. No local product server,
browser, model, database, SDK build or evaluation was run.
- Prior-head CI run 40906120388 failed one E2E test because the real
announcement footer intercepted its Add to composer click. The other 28
tests in that shard passed. The narrow fixture uses the existing UI
dismissal pattern. The added handler passes an isolated noEmit check
against the real Playwright Page type. An additional strict check of the
whole browser test fails on five unchanged shared-fixture type errors;
checking that unchanged shared file alone gives identical diagnostics.
The initial shell exit was misreported and corrected in the validation
record. Browser verification is deferred to the next normal CI run; no
local browser or server was started.
- Repository-wide local build and test commands were not run because
this is a scoped source change. Normal required PR checks will provide
the repository CI result.

## Risks

A registry entry is not proof that a target or feature is ready. A
future target must use the exact matching private helper and producer
contract, retain any stronger target scanners, and produce fresh source
fingerprints. Existing running jobs keep their pinned workflow and
target sources. All original attempt limits, declared inputs, private
storage checks, fatal scanners and exact administrator-controlled run
approval remain in force.

The new cell report contract accepts only canonical assets from the
actual pinned Playwright installation. Other assets and layouts still
fail closed. No private evidence or generated report is included in this
PR.

## Model Used

OpenAI GPT-6 through Codex, with tool use and code execution. The
runtime does not expose a finer model deployment ID or context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-29 11:42:35 -05:00

DottaandPaperclip

018c4c9c9b

Retain approved manual E2E evidence in private history (#18134 )

## Thinking Path

> - Paperclip lets people manage AI agents and their work.
> - The manual AWS E2E workflow records browser and model evidence.
> - Public repository Actions artifacts are not a restricted evidence
store.
> - Some chat evidence needs the existing private-history store.
> - A target flag must not grant archive access or enable a public
fallback.
> - This pull request binds restricted runs to protected exact-source
approval and retains their standard evidence with trusted transfer code.
> - Operators can retain complete evidence without changing scanner
verdicts or scenario outcomes.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Manual AWS E2E evidence retention for exact reviewed chat-feature
selections.

**Subsystem affected**

The runner E2E manual workflow, standard evidence handoff and
private-history transfer.

**Current behavior**

Manual cell and merged evidence use Actions artifacts. In a public
repository, repository readers can download those files. Scheduled
private history does not authorize a manual target to use its role.

**Proposed behavior**

Classify sensitive selections in the trusted workflow. Require one
protected original-run approval. Verify existing private storage before
target setup or model secrets. Retain scanned standard originals and
final packaged results through a workflow-pinned runtime. Deny public
artifact and history fallback on failure.

**Reason and benefit**

Preserve inspectable standard evidence in restricted storage without
treating a private bucket as permission to retain credentials.

**Additional context**

Related source work is separate: #18067 changes paid runner routing,
#18100 changes scoped input delivery, and #17888 changes report catalog
parsing. This pull request does not include or revert those changes.

## What Changed

- Add a trusted manual classification and exact run, attempt, workflow,
target, selection and launcher approval contract.
- Add a protected manual private job. Prepare a second workflow-pinned
checkout and isolated publication dependencies before archive
credentials.
- Seal standard originals and final result bytes with exact inventories.
Reject credential contamination, unscanned carriers and changed files.
- Use existing immutable private-history archives and transfer receipts
for cells and merged reports. Keep the existing bounds.
- Use per-cell exact-key read sessions and a separate report writer.
Check every inline policy against the 2,048-character STS limit.
- Scan and seal fresh generated report logs and files with the actual
report-step credential roots before transfer.
- Require a positive nonprivate route for Actions uploads and public
history. Private failures have no public fallback. Preserve normal
catalog semantics for nonprivate selections with more than 32 IDs or
duplicate IDs.
- Authenticate generated-report fonts and every reserved trace-viewer
static asset against the exact trusted producer path and bytes,
including HTML and JavaScript. Keep standard cell evidence and unknown
carriers closed.
- Document the first eight supported IDs, deployment gates and the
operator trust boundary.

## Verification

- 57 new pure, actual-helper and actual-workflow-shell controls pass.
- 50 affected existing workflow, routing, target and private-history
controls pass. The final committed workflow security controls also pass.
- Targeted helper/test and actual launcher noEmit checks pass.
- `actionlint -shellcheck= -pyflakes=
.github/workflows/runner-full-stack-e2e.yml` passes.
- Parent `2f46336d7` required CI is green. The server 11/12 shard first
failed on database cleanup/test timing and passed one bounded recheck
with unchanged source. The original failure remains recorded.
- The preserved docs-only parent `d721c170f` had 59 terminal,
nonblocking checks and fresh Greptile 5/5. Three later review comments
exposed a genuine static-report asset failure and nonprivate selector
regression; their parent counterexamples remain recorded.
- The reviewed parent passes 76 focused controls across nine files. The
final dependency preflight successor passes all six affected controls,
including two new actual `prepare` subprocess controls, and targeted
helper/test noEmit. Unchanged suites were not repeated. A synthetic
Node-only Playwright HTML/blob/JUnit report and the real dashboard
renderer/brand stager pass the actual sealer, strict archive and
immutable receipt path; only AWS transport is synthetic. No browser or
product setup runs in that test.
- Default cell fonts/SVGs, unknown report files, forged paths, altered
static bytes, recomputed inventory tampering and registered credential
matches remain rejected. Eligibility and remaining-byte checks run
before body reads in both sealer and transfer validator; both
independent parent read-order failures are preserved.
- The pre-credential target comparison also binds the new
generated-report asset helper byte for byte. A genuine
tampered-dependency parent failure is preserved.
- Six additional actual sealer/validator controls expose the
standard-extension trace-owner bypass on the parent. All 34 affected
controls pass on its correction, including the real
producer/archive/receipt test; targeted noEmit passes. Ordinary dynamic
report HTML outside the reserved trace directory is unchanged.
- Final head `43683c4e0` passes both required gates, `ci / verify` and
`ci / e2e`, in normal CI run `40849916302` attempt 1. All 59 rollup
checks/statuses are terminal and nonblocking (55 success, two neutral,
two skipped).
- Fresh exact-head Greptile is 5/5; automated review is terminal and all
technical threads are resolved. Human CODEOWNER approval remains a
separate gate. Earlier source, genuine counterexamples and CI history
remain preserved.
- Full local build and live E2E were not run. This task prohibits local
SDK/native/server builds and paid or AWS operations. Normal pull request
CI must pass before merge readiness.
- No existing artifact bodies, archives, media or provider reasoning
were inspected. Synthetic owned fixtures prove the privacy boundaries.

## Risks

- Deployment still needs the existing private bucket, role and protected
environments. Both environments need the exact original-run approval
variable.
- Targets need the compatible handoff protocol and a separately reviewed
exact launcher digest. Existing target branches do not become ready
automatically.
- Complete duplicated originals and packaged evidence can exceed
existing archive limits. Such a transfer fails; files are not silently
removed.
- Required evidence omissions remain fatal to PASS. Safe failed evidence
is retained with an explicit missing list and incomplete
required-evidence verdict. Credential contamination still blocks all
original uploads. Restricted storage does not waive PNG, HTML, blob or
scanner checks.
- This is operator-approved source execution. Target processes can
inherit the job's OIDC request environment and may re-assume a role.
Inline session attenuation is not isolation from malicious approved
source.
- Operator authorization must bind the actual exact source and original
run. A proven administrator-controlled record may supply it without an
environment reviewer UI. No new approval UI or environment change is
required by this PR.
- Required repository CODEOWNER review is still pending. Commented
reviews and review requests are not approvals. This PR stays open and
unmerged.
- This PR does not deploy infrastructure, change IAM, dispatch
workflows, publish report payloads, change grades or transfer prior
scenario passes.

## Model Used

OpenAI `gpt-6.1-sol` in Codex. Used code execution, source analysis and
tool calls. The host does not expose the context window size.
Independent peer review used the same model family in a separate agent.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-29 08:51:40 -05:00