PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-12 17:00:37 +02:00

Commit Graph

Author	SHA1	Message	Date
DottaandPaperclip	b5342febe5	fix(runner): require current-turn completion after connection continuations (#15514 ) ## Thinking Path > - Paperclip manages agents, tasks, permissions, and execution budgets. > - Native tasks can resume after a connection decision in the same provider conversation. > - Each turn still needs an accepted completion report. > - Compact continuation messages did not explain that reports from earlier turns cannot finish the new turn. > - The connection evaluator could also grade before the final reply was stored or reject valid unavailable-access wording. > - This PR clarifies the current-turn report requirement and fixes those observation boundaries. > - The original connection instructions and strict native completion gate stay in place. ## Linked Issues or Issue Description Refs #15489. The reduction remains draft while this separate repair is qualified. Refs #15471 for the earlier connection continuation work. ## What Changed - Add a current-turn completion reminder to compact continuation inputs. - Keep final prose insufficient for completion. Preserve permissions and retry policy. - Wait for the final successful task run's saved, attributed decline reply within the existing deadline. - Use one bounded explanation matcher for both decline checks. - Wait for a recorded tool-action rejection to dispatch its bound continuation, with strict company, task, agent and source-run checks. - Retain the exact grading input before later API refreshes. - Add failure and delay regressions and update the Runner and evaluator docs. ## Verification The fresh comparison has 15/15 original passes on each variant: 15 unchanged pass pairs, zero new failures, and no pending pair. There are 30 case attempts and 65 actual agent runs (baseline 33; candidate 32). All runs succeeded. All 30 cleanups passed. No model attempt was retried. \| Profile \| Baseline \| Candidate \| \| --- \| --- \| --- \| \| Native Codex `gpt-5.6-sol` \| 5/5 \| 5/5 \| \| ACPX Claude `claude-sonnet-5` \| 5/5 \| 5/5 \| \| OpenCode `openrouter/deepseek/deepseek-v4-flash-0731` \| 5/5 \| 5/5 \| Each profile covers service approval, service decline, connection decline, provider decline, and selection of the second provider. The saved replies, approved briefings, decisions, fixture observations, final task states, and native completion records were inspected. All 18 saved decline-grade snapshots match their original captured inputs and checks. Result, API snapshot, and final ledger run sets agree. - [Candidate campaign](https://github.com/paperclipai/paperclip/actions/runs/37711658378) · [public candidate report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37711658378-1/index.html) - [Baseline campaign](https://github.com/paperclipai/paperclip/actions/runs/37711675579) · [public baseline report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37711675579-1/index.html) - Candidate source: `79905343bba280d462765faad19a26e7f179259e`. Baseline source: `7c5e120158f1385a1fc5f20be41f66c58a605534`. Both use master context `fc6304dfe5f2e446e09bd052a7b45f51e930f250`. The trusted workflow source is separately frozen at `dd777f4b7343305c4e6f44c422f44a1d78e12e4f`. - Both variants use the same evaluator, fixtures, models, permissions, 720-second cell deadline, and 1,000-cent company and agent hard stops. The only production difference is the compact continuation reminder. - Suite hash: `aee30b74b4d38ada08777798db0932fbc64e368bb427d29caa8cfd86c7f59747`. Definition hash: `ea9e17f54fe0af1acbb2337ddfab8a7e92db8488bd0f59510a63095ca0229b60`. - Provider-free transport capture: startup/resume input stays at 55,726 bytes. Compact continuation input grows from 53,334 to 53,535 bytes. The 42-tool catalog stays unchanged. These are Paperclip input bytes, not complete vendor prompt tokens. - Focused evaluator tests: 31 pass. Native contract, transport delivery, and session tests: 194 pass. Evaluator support: 1,819 TypeScript tests and 128 Node tests pass, with one intentional skip. - Full build, workspace typecheck, and evaluator typecheck pass. Current-head CI passes all required gates. The current rollup has 51 successful check runs, two intentional Storybook skips, and a successful Snyk status. Review is 5/5 with zero unresolved threads. - CI attempt 1 had one initial runtime-fixture health timeout. The exact test and its full 164-test file pass locally. One CI shard retry passed. The original CI failure, its dependent verify failure, and the retry remain visible in [CI history](https://github.com/paperclipai/paperclip/actions/runs/37711235060). - The broad local `pnpm test:run` attempt was interrupted after about 49 minutes (exit 130). It recorded one failure in the unchanged Zep memory-connector disabled-setting test. That test and the full 388-test tool-access file pass in separate local checks; current-head CI also passes. The local cause is not established, and this broad local attempt is not claimed as passing. Three earlier local failures also pass in their isolated checks; their original logs remain retained. - The first two setup admissions were cancelled before provider jobs to include the review correction. They made no provider calls. The completed campaigns above are the first and only model attempts for these corrected variants. Cost evidence stays separate from behavior. Original result summaries report only OpenCode amounts: baseline $0.039192096 and candidate $0.063221620. Final run ledgers also retain estimates for Claude (baseline $1.232439000; candidate $1.333611200) and Codex (baseline $2.340324400; candidate $2.129335600). These estimates do not replace the original summaries. Local and GitHub runtime are unmetered here. Invoices are unknown. This is not a cheaper or faster claim. ## Risks - A single matched trial cannot prove general equivalence or causation. The reminder is an instruction change, not a new completion enforcement rule. - Candidate OpenCode service-decline finished within one continuous run; its baseline used two. That pair passed the task outcome, but it does not qualify the reminder on a resumed decline turn. No extra paid run was used to replace it. - The text matcher is bounded evidence of an explanation. It does not prove reasoning or consumption of feedback. Bounded stdout excerpts do not prove that every extra attempted tool call is absent. - Missing saved replies or continuations still fail at the original deadline. Failed native completion remains a failure even when final prose is correct. - The unresolved local-suite discrepancy above remains a validation limit. Full remote CI and both focused local reproductions pass. - The original connection instructions stay in place. These results do not qualify the reduction in #15489. Its original 11/15 versus 12/15 grades and two new failing pairs remain unchanged. ## Model Used OpenAI Codex, based on GPT-6. The exact deployment ID and context window are not exposed in this session. Capabilities used: reasoning, repository editing, code execution, test inspection and eval analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the focused local tests listed above and they pass; the interrupted broad local run is disclosed above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-08 04:52:12 -05:00
DottaandPaperclip	d0db8820db	fix(connections): repair native baseline and approval continuations (#15420 ) ## Thinking Path > - Paperclip manages AI agents and the tools they may use. > - Connection setup separates provider preference from permission to use a tool. > - The first native connection baseline could not exercise its intended decisions. > - The browser used mutable task titles, and the provider fixture already granted access. > - Native provider-choice instructions also disagreed with the preferred question format. Schema rejection gave no field guidance. > - This pull request repairs those test preconditions and native guidance, then fixes restart/approval defects exposed by the corrected baseline. It also restores missing OpenCode tool-error evidence. > - The benefit is an inspectable baseline before any further instruction reduction. ## Linked Issues or Issue Description Refs #15407. The original 15-cell baseline remains 0 PASS / 15 FAIL. Ten cells stopped on stale titles, two Codex cells had schema denials, two OpenCode cells used already-granted tools, and one Claude cell returned no native result. No intended user decisions were submitted. The exact invalid Codex field and underlying Claude failure cause remain unknown. [Original campaign](https://github.com/paperclipai/paperclip/actions/runs/37562577199) · [Original report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37562577199-1/index.html) ## What Changed - Match the browser's task route and visible identifier instead of a title the agent can change. - Start native provider-choice fixtures with no agent tool access. Verify the public effective-access records. - In the positive case, select Arcade, then grant its exact HubSpot tool through the real access card. Require both saved decisions and exactly one observed call. - Return a canonical `providerQuestionSet` for native input and retain the equivalent legacy `providerQuestion`. - Keep invalid input rejected. Return bounded schema locations and required field names without submitted values. - Preserve Claude's exact session/content identity while allowing authenticated registered instruction-copy paths to rotate on a new run. - Reject duplicate approval reports for an existing exact tool-action card before they create another human review. - Wait for a recorded service-approval continuation within the existing deadline; retain missing or failed continuation grades. - Forward OpenCode tool activity through the runner facade, preserving bounded errors and execution-part identity without inventing host-call joins or exposing arguments. - Preserve original grades, costs, scope limits and diagnoses in the dated repair report. ## Verification - Eval typecheck passes. Support suite: 1,801 PASS, one intentional skip; Node checks: 128 PASS. - Connection/schema tests: 51 PASS. Real-server public fixture setup: one PASS with zero providers. - Browser support regression: five PASS, including renamed and wrong tasks. - Focused Rust safe-feedback test: one PASS. - Repository typecheck and build pass before the latest master replay. Post-replay connection/shared/real-server fixture checks: 52 PASS; eval typecheck passes. The browser review fix additionally passes all five browser checks and seven suite checks. - The full local repository run was interrupted incomplete after about 45 minutes, with five integration failures retained. All five pass in a separate targeted invocation (1,250 unrelated tests skipped). No full local-suite pass or root cause for the initial local failures is claimed. - Corrected frozen source `162cc90fdabe7f505b88ae095044531b82784c92`: 10 PASS / 5 FAIL across the [passing Codex canary](https://github.com/paperclipai/paperclip/actions/runs/37575158761) and [remaining 14 cells](https://github.com/paperclipai/paperclip/actions/runs/37576261807). The canary passes all 17 checks. Claude's two provider-choice continuations fail on restart, Claude service approval exposes an early evaluator rejection, Codex service approval creates a duplicate approval, and OpenCode provider-second times out after both decisions with no HubSpot call. No original result is regraded. - Final ledgers count 31 actual runs: 27 succeeded, two failed, two cancelled during cleanup. All 15 cleanup/budget checks pass. The late Claude continuation is absent from its earlier workflow snapshot; it remains in the result/API/final ledger. Original evidence retains 279 hashes. Recorded LLM subtotal $0.04553787 is incomplete billing, not actual total cost; local runtime is unmetered. - New repair regressions reproduce the Claude attach failure, duplicate approval acceptance and dropped OpenCode tool events before their respective fixes. Nine Rust attachment checks, 127 ACPX host/adapter tests, 33 completion/control-plane checks, nine eval deadline tests, 59 OpenCode proxy/driver tests, one Rust tool-error/redaction check, and TypeScript/Rust composer parity pass. Eval typecheck, repository typecheck and build pass. Existing support coverage is 1,802 PASS plus 128 Node PASS, one intentional support skip; two additional deadline tests also pass. - New-source full CI/review and live canaries are pending. The next bounded selection is Claude provider-decline, Codex service-approve and one OpenCode provider-second diagnostic with repaired event evidence. No broader campaign or instruction-reduction qualification is claimed. - Initial corrected campaign [37574251834](https://github.com/paperclipai/paperclip/actions/runs/37574251834) was cancelled during shared build after review found the breadcrumb whitespace assumption. Its matrix job has zero steps and no provider execution. The real adjacent-span browser regression now reproduces the old failure and passes after the fix. ## Risks - The corrected baseline remains 10/15. The new restart/approval fixes require live qualification; OpenCode evidence forwarding does not itself establish or fix its prior behavioral failure. - The positive provider case now expects three runs, including separate access approval. Its new results are distinct from the original invalid fixture. - The old Claude missing-result cause and rejected Codex field are unknown. These repairs do not retroactively explain or erase either failure. - Path rotation must preserve prompt, custom instruction, skill/content identity and protected provider settings; regression checks reject stale or changed content. No connection authorization, JSON schema, budget, cleanup, or final-result requirement is relaxed. Historical Everyday prompts and gateway setup remain unchanged. ## Model Used OpenAI Codex, GPT-6. The exact deployment variant and context window are not exposed in this session. Used repository inspection, code editing, test execution and retained-evidence analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-07 06:18:51 -05:00
DottaandPaperclip	caf120105c	test: prepare neutral native connection guidance evals (#15407 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents need to discover connections, obtain consent, and continue from saved decisions. > - We want to reduce repeated instructions only when measured behavior supports the change. > - The existing decline tasks tell the model not to retry. One provider-decline check can pass without an explanation or an observed service counter. > - This PR adds neutral tasks and stricter saved-evidence checks before any connection instruction reduction. > - Production instructions remain unchanged. The new cells are configured, not live-qualified. ## Linked Issues or Issue Description Refs #15218. Refs #15389. What existing behavior does this improve? The Product E2E connection workflow evaluation and its instruction measurement provenance. Current behavior Some decline prompts supply the policy they intend to test. The provider-decline workflow does not require a saved post-decision explanation. Its old no-call check can use a missing fixture counter as zero. OpenCode has no connection cases in the original Everyday matrix. Proposed behavior Add an explicit-only suite with five connection stories on native Codex, ACPX Claude, and OpenCode. Require an explanation attributed by exact run ID after a saved decline. Observe the provider fixture counter. Preserve the original cases and grades. ## What Changed - Add fifteen configured cells with one attempt, twelve-minute deadlines, and verified 1,000-cent company and agent budget stops. - Remove procedure hints from the three new decline prompts. Keep a user-permitted explanation fallback and the existing positive controls. - Require saved decline state, one decision, unchanged connections, observed zero service calls where applicable, and a post-decision explanation from a successful run on the same task. - Add negative grader calibration and test the actual fixture budget payloads. Exclude the suite from default and generic selection. - Extend the existing full-catalog measurement source manifest with connection descriptions and schemas. Add an audit of fixed text, tool descriptions, returned instructions, and unqualified behavior. - Rebase on master `a6306ba606eb87c89b9ef0344e9fe8e0025580f9` and preserve its new Cursor suites. No production, credential, workflow, or lockfile change. ## Verification - Before rebase: Product E2E support passed 1,424 TypeScript tests and 128 Node checks. Six catalog measurement tests, repository typecheck/build, Product E2E typecheck, and exact fifteen-cell discovery passed. - The full pre-rebase repository test run was stopped when master advanced. Its partial result is not a pass. - After rebase and the review correction: repository build/typecheck, Product E2E typecheck, 1,799 TypeScript support tests (one skipped), 128 Node checks, six measurement tests, and exact fifteen-cell discovery pass. The duplicate local full-suite run was stopped incomplete after about 20 minutes once complete CI passed; no local full-suite pass is claimed. - Review found that the initial grader read `runId` instead of public `createdByRunId`. A regression calibration reproduced both rejection of valid public comments and acceptance of the wrong alias. The fix uses the actual field and binds the evidence type to the shared `IssueComment` contract. A subsequent type-only import path correction passes Product E2E typecheck. - Final source `0de306b9664bfbdebb6709ddb54c95152740d1ad` passes [complete CI](https://github.com/paperclipai/paperclip/actions/runs/37560250545): 51 successful checks and two intentional Storybook skips, plus separate Snyk success. Fresh Greptile review is 5/5 with the single review thread resolved and no new findings. The PR is clean and mergeable. - Local commands: `pnpm build`, `pnpm -r typecheck`, `pnpm test:e2e:runner:unit`, `pnpm test:e2e:runner:typecheck`, and `pnpm test:e2e:runner -- --list --suite native-connection-guidance`. The measurement uses `PAPERCLIP_NATIVE_PROCEDURE_MEASUREMENT=/tmp/connection-measurement.json pnpm exec vitest run --project @paperclipai/server server/src/__tests__/native-procedure-measurement.test.ts`. Validation used pinned pnpm 9.15.4. - No paid provider campaign was started. There is no baseline/candidate behavior result for these new cells. - The audit records 654 UTF-8 bytes of fixed connection guidance. A clean capture at `a04b8c6a452315625014888335d45670a2094fb6` confirms 41 supplied tools, 53,341 normalized bytes at start/resume, 50,949 at compact continuation, and a 48,195-byte authenticated OpenCode MCP catalog. These are byte counts, not tokens, bills, vendor-private prompt sizes, or savings from this PR. ## Risks - This is eval preparation. Passing support tests do not establish live model behavior or qualify an instruction reduction. - The explanation oracle checks attributed saved output. It does not prove cognition or arbitrary prose truthfulness. One saved interaction also does not prove the absence of repeated idempotent tool calls. - Successful new authentication and tool refresh, existing-connection agent grants, independent work while waiting, explicit retry after decline, and blocking when mandatory work remains still need separate coverage. - Notion setup decline does not execute a real Notion service. Positive service approval uses an already installed deterministic service; it does not qualify new connection creation. - The original historical failures remain unchanged. Future comparisons must freeze source, fixture, model, input, and grading controls and retain every actual attempt. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, and code execution. The exact serving model ID and context-window size were not exposed in this session; they are not inferred. No model provider was invoked by the eval suite in this PR preparation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-06 21:28:16 -05:00

Author

SHA1

Message

Date

DottaandPaperclip

b5342febe5

fix(runner): require current-turn completion after connection continuations (#15514 )

## Thinking Path

> - Paperclip manages agents, tasks, permissions, and execution budgets.
> - Native tasks can resume after a connection decision in the same
provider conversation.
> - Each turn still needs an accepted completion report.
> - Compact continuation messages did not explain that reports from
earlier turns cannot finish the new turn.
> - The connection evaluator could also grade before the final reply was
stored or reject valid unavailable-access wording.
> - This PR clarifies the current-turn report requirement and fixes
those observation boundaries.
> - The original connection instructions and strict native completion
gate stay in place.

## Linked Issues or Issue Description

Refs #15489. The reduction remains draft while this separate repair is
qualified. Refs #15471 for the earlier connection continuation work.

## What Changed

- Add a current-turn completion reminder to compact continuation inputs.
- Keep final prose insufficient for completion. Preserve permissions and
retry policy.
- Wait for the final successful task run's saved, attributed decline
reply within the existing deadline.
- Use one bounded explanation matcher for both decline checks.
- Wait for a recorded tool-action rejection to dispatch its bound
continuation, with strict company, task, agent and source-run checks.
- Retain the exact grading input before later API refreshes.
- Add failure and delay regressions and update the Runner and evaluator
docs.

## Verification

The fresh comparison has **15/15 original passes on each variant**: 15
unchanged pass pairs, zero new failures, and no pending pair. There are
30 case attempts and **65 actual agent runs** (baseline 33; candidate
32). All runs succeeded. All 30 cleanups passed. No model attempt was
retried.

| Profile | Baseline | Candidate |
| --- | --- | --- |
| Native Codex `gpt-5.6-sol` | 5/5 | 5/5 |
| ACPX Claude `claude-sonnet-5` | 5/5 | 5/5 |
| OpenCode `openrouter/deepseek/deepseek-v4-flash-0731` | 5/5 | 5/5 |

Each profile covers service approval, service decline, connection
decline, provider decline, and selection of the second provider. The
saved replies, approved briefings, decisions, fixture observations,
final task states, and native completion records were inspected. All 18
saved decline-grade snapshots match their original captured inputs and
checks. Result, API snapshot, and final ledger run sets agree.

- [Candidate
campaign](https://github.com/paperclipai/paperclip/actions/runs/37711658378)
· [public candidate
report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37711658378-1/index.html)
- [Baseline
campaign](https://github.com/paperclipai/paperclip/actions/runs/37711675579)
· [public baseline
report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37711675579-1/index.html)
- Candidate source: `79905343bba280d462765faad19a26e7f179259e`. Baseline
source: `7c5e120158f1385a1fc5f20be41f66c58a605534`. Both use master
context `fc6304dfe5f2e446e09bd052a7b45f51e930f250`. The trusted workflow
source is separately frozen at
`dd777f4b7343305c4e6f44c422f44a1d78e12e4f`.
- Both variants use the same evaluator, fixtures, models, permissions,
720-second cell deadline, and 1,000-cent company and agent hard stops.
The only production difference is the compact continuation reminder.
- Suite hash:
`aee30b74b4d38ada08777798db0932fbc64e368bb427d29caa8cfd86c7f59747`.
Definition hash:
`ea9e17f54fe0af1acbb2337ddfab8a7e92db8488bd0f59510a63095ca0229b60`.
- Provider-free transport capture: startup/resume input stays at 55,726
bytes. Compact continuation input grows from 53,334 to 53,535 bytes. The
42-tool catalog stays unchanged. These are Paperclip input bytes, not
complete vendor prompt tokens.
- Focused evaluator tests: 31 pass. Native contract, transport delivery,
and session tests: 194 pass. Evaluator support: 1,819 TypeScript tests
and 128 Node tests pass, with one intentional skip.
- Full build, workspace typecheck, and evaluator typecheck pass.
Current-head CI passes all required gates. The current rollup has 51
successful check runs, two intentional Storybook skips, and a successful
Snyk status. Review is 5/5 with zero unresolved threads.
- CI attempt 1 had one initial runtime-fixture health timeout. The exact
test and its full 164-test file pass locally. One CI shard retry passed.
The original CI failure, its dependent verify failure, and the retry
remain visible in [CI
history](https://github.com/paperclipai/paperclip/actions/runs/37711235060).
- The broad local `pnpm test:run` attempt was interrupted after about 49
minutes (exit 130). It recorded one failure in the unchanged Zep
memory-connector disabled-setting test. That test and the full 388-test
tool-access file pass in separate local checks; current-head CI also
passes. The local cause is not established, and this broad local attempt
is **not** claimed as passing. Three earlier local failures also pass in
their isolated checks; their original logs remain retained.
- The first two setup admissions were cancelled before provider jobs to
include the review correction. They made no provider calls. The
completed campaigns above are the first and only model attempts for
these corrected variants.

Cost evidence stays separate from behavior. Original result summaries
report only OpenCode amounts: baseline $0.039192096 and candidate
$0.063221620. Final run ledgers also retain estimates for Claude
(baseline $1.232439000; candidate $1.333611200) and Codex (baseline
$2.340324400; candidate $2.129335600). These estimates do not replace
the original summaries. Local and GitHub runtime are unmetered here.
Invoices are unknown. This is not a cheaper or faster claim.

## Risks

- A single matched trial cannot prove general equivalence or causation.
The reminder is an instruction change, not a new completion enforcement
rule.
- Candidate OpenCode service-decline finished within one continuous run;
its baseline used two. That pair passed the task outcome, but it does
not qualify the reminder on a resumed decline turn. No extra paid run
was used to replace it.
- The text matcher is bounded evidence of an explanation. It does not
prove reasoning or consumption of feedback. Bounded stdout excerpts do
not prove that every extra attempted tool call is absent.
- Missing saved replies or continuations still fail at the original
deadline. Failed native completion remains a failure even when final
prose is correct.
- The unresolved local-suite discrepancy above remains a validation
limit. Full remote CI and both focused local reproductions pass.
- The original connection instructions stay in place. These results do
not qualify the reduction in #15489. Its original 11/15 versus 12/15
grades and two new failing pairs remain unchanged.

## Model Used

OpenAI Codex, based on GPT-6. The exact deployment ID and context window
are not exposed in this session. Capabilities used: reasoning,
repository editing, code execution, test inspection and eval analysis.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run the focused local tests listed above and they pass; the
interrupted broad local run is disclosed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-08 04:52:12 -05:00

DottaandPaperclip

d0db8820db

fix(connections): repair native baseline and approval continuations (#15420 )

## Thinking Path

> - Paperclip manages AI agents and the tools they may use.
> - Connection setup separates provider preference from permission to
use a tool.
> - The first native connection baseline could not exercise its intended
decisions.
> - The browser used mutable task titles, and the provider fixture
already granted access.
> - Native provider-choice instructions also disagreed with the
preferred question format. Schema rejection gave no field guidance.
> - This pull request repairs those test preconditions and native
guidance, then fixes restart/approval defects exposed by the corrected
baseline. It also restores missing OpenCode tool-error evidence.
> - The benefit is an inspectable baseline before any further
instruction reduction.

## Linked Issues or Issue Description

Refs #15407.

The original 15-cell baseline remains 0 PASS / 15 FAIL. Ten cells
stopped on stale titles, two Codex cells had schema denials, two
OpenCode cells used already-granted tools, and one Claude cell returned
no native result. No intended user decisions were submitted. The exact
invalid Codex field and underlying Claude failure cause remain unknown.

[Original
campaign](https://github.com/paperclipai/paperclip/actions/runs/37562577199)
· [Original
report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37562577199-1/index.html)

## What Changed

- Match the browser's task route and visible identifier instead of a
title the agent can change.
- Start native provider-choice fixtures with no agent tool access.
Verify the public effective-access records.
- In the positive case, select Arcade, then grant its exact HubSpot tool
through the real access card. Require both saved decisions and exactly
one observed call.
- Return a canonical `providerQuestionSet` for native input and retain
the equivalent legacy `providerQuestion`.
- Keep invalid input rejected. Return bounded schema locations and
required field names without submitted values.
- Preserve Claude's exact session/content identity while allowing
authenticated registered instruction-copy paths to rotate on a new run.
- Reject duplicate approval reports for an existing exact tool-action
card before they create another human review.
- Wait for a recorded service-approval continuation within the existing
deadline; retain missing or failed continuation grades.
- Forward OpenCode tool activity through the runner facade, preserving
bounded errors and execution-part identity without inventing host-call
joins or exposing arguments.
- Preserve original grades, costs, scope limits and diagnoses in the
dated repair report.

## Verification

- Eval typecheck passes. Support suite: 1,801 PASS, one intentional
skip; Node checks: 128 PASS.
- Connection/schema tests: 51 PASS. Real-server public fixture setup:
one PASS with zero providers.
- Browser support regression: five PASS, including renamed and wrong
tasks.
- Focused Rust safe-feedback test: one PASS.
- Repository typecheck and build pass before the latest master replay.
Post-replay connection/shared/real-server fixture checks: 52 PASS; eval
typecheck passes. The browser review fix additionally passes all five
browser checks and seven suite checks.
- The full local repository run was interrupted incomplete after about
45 minutes, with five integration failures retained. All five pass in a
separate targeted invocation (1,250 unrelated tests skipped). No full
local-suite pass or root cause for the initial local failures is
claimed.
- Corrected frozen source `162cc90fdabe7f505b88ae095044531b82784c92`:
**10 PASS / 5 FAIL** across the [passing Codex
canary](https://github.com/paperclipai/paperclip/actions/runs/37575158761)
and [remaining 14
cells](https://github.com/paperclipai/paperclip/actions/runs/37576261807).
The canary passes all 17 checks. Claude's two provider-choice
continuations fail on restart, Claude service approval exposes an early
evaluator rejection, Codex service approval creates a duplicate
approval, and OpenCode provider-second times out after both decisions
with no HubSpot call. No original result is regraded.
- Final ledgers count 31 actual runs: 27 succeeded, two failed, two
cancelled during cleanup. All 15 cleanup/budget checks pass. The late
Claude continuation is absent from its earlier workflow snapshot; it
remains in the result/API/final ledger. Original evidence retains 279
hashes. Recorded LLM subtotal $0.04553787 is incomplete billing, not
actual total cost; local runtime is unmetered.
- New repair regressions reproduce the Claude attach failure, duplicate
approval acceptance and dropped OpenCode tool events before their
respective fixes. Nine Rust attachment checks, 127 ACPX host/adapter
tests, 33 completion/control-plane checks, nine eval deadline tests, 59
OpenCode proxy/driver tests, one Rust tool-error/redaction check, and
TypeScript/Rust composer parity pass. Eval typecheck, repository
typecheck and build pass. Existing support coverage is 1,802 PASS plus
128 Node PASS, one intentional support skip; two additional deadline
tests also pass.
- New-source full CI/review and live canaries are pending. The next
bounded selection is Claude provider-decline, Codex service-approve and
one OpenCode provider-second diagnostic with repaired event evidence. No
broader campaign or instruction-reduction qualification is claimed.
- Initial corrected campaign
[37574251834](https://github.com/paperclipai/paperclip/actions/runs/37574251834)
was cancelled during shared build after review found the breadcrumb
whitespace assumption. Its matrix job has zero steps and no provider
execution. The real adjacent-span browser regression now reproduces the
old failure and passes after the fix.

## Risks

- The corrected baseline remains 10/15. The new restart/approval fixes
require live qualification; OpenCode evidence forwarding does not itself
establish or fix its prior behavioral failure.
- The positive provider case now expects three runs, including separate
access approval. Its new results are distinct from the original invalid
fixture.
- The old Claude missing-result cause and rejected Codex field are
unknown. These repairs do not retroactively explain or erase either
failure.
- Path rotation must preserve prompt, custom instruction, skill/content
identity and protected provider settings; regression checks reject stale
or changed content. No connection authorization, JSON schema, budget,
cleanup, or final-result requirement is relaxed. Historical Everyday
prompts and gateway setup remain unchanged.

## Model Used

OpenAI Codex, GPT-6. The exact deployment variant and context window are
not exposed in this session. Used repository inspection, code editing,
test execution and retained-evidence analysis.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-07 06:18:51 -05:00

DottaandPaperclip

caf120105c

test: prepare neutral native connection guidance evals (#15407 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents need to discover connections, obtain consent, and continue
from saved decisions.
> - We want to reduce repeated instructions only when measured behavior
supports the change.
> - The existing decline tasks tell the model not to retry. One
provider-decline check can pass without an explanation or an observed
service counter.
> - This PR adds neutral tasks and stricter saved-evidence checks before
any connection instruction reduction.
> - Production instructions remain unchanged. The new cells are
configured, not live-qualified.

## Linked Issues or Issue Description

Refs #15218. Refs #15389.

**What existing behavior does this improve?**

The Product E2E connection workflow evaluation and its instruction
measurement provenance.

**Current behavior**

Some decline prompts supply the policy they intend to test. The
provider-decline workflow does not require a saved post-decision
explanation. Its old no-call check can use a missing fixture counter as
zero. OpenCode has no connection cases in the original Everyday matrix.

**Proposed behavior**

Add an explicit-only suite with five connection stories on native Codex,
ACPX Claude, and OpenCode. Require an explanation attributed by exact
run ID after a saved decline. Observe the provider fixture counter.
Preserve the original cases and grades.

## What Changed

- Add fifteen configured cells with one attempt, twelve-minute
deadlines, and verified 1,000-cent company and agent budget stops.
- Remove procedure hints from the three new decline prompts. Keep a
user-permitted explanation fallback and the existing positive controls.
- Require saved decline state, one decision, unchanged connections,
observed zero service calls where applicable, and a post-decision
explanation from a successful run on the same task.
- Add negative grader calibration and test the actual fixture budget
payloads. Exclude the suite from default and generic selection.
- Extend the existing full-catalog measurement source manifest with
connection descriptions and schemas. Add an audit of fixed text, tool
descriptions, returned instructions, and unqualified behavior.
- Rebase on master `a6306ba606eb87c89b9ef0344e9fe8e0025580f9` and
preserve its new Cursor suites. No production, credential, workflow, or
lockfile change.

## Verification

- Before rebase: Product E2E support passed 1,424 TypeScript tests and
128 Node checks. Six catalog measurement tests, repository
typecheck/build, Product E2E typecheck, and exact fifteen-cell discovery
passed.
- The full pre-rebase repository test run was stopped when master
advanced. Its partial result is not a pass.
- After rebase and the review correction: repository build/typecheck,
Product E2E typecheck, 1,799 TypeScript support tests (one skipped), 128
Node checks, six measurement tests, and exact fifteen-cell discovery
pass. The duplicate local full-suite run was stopped incomplete after
about 20 minutes once complete CI passed; no local full-suite pass is
claimed.
- Review found that the initial grader read `runId` instead of public
`createdByRunId`. A regression calibration reproduced both rejection of
valid public comments and acceptance of the wrong alias. The fix uses
the actual field and binds the evidence type to the shared
`IssueComment` contract. A subsequent type-only import path correction
passes Product E2E typecheck.
- Final source `0de306b9664bfbdebb6709ddb54c95152740d1ad` passes
[complete
CI](https://github.com/paperclipai/paperclip/actions/runs/37560250545):
51 successful checks and two intentional Storybook skips, plus separate
Snyk success. Fresh Greptile review is 5/5 with the single review thread
resolved and no new findings. The PR is clean and mergeable.
- Local commands: `pnpm build`, `pnpm -r typecheck`, `pnpm
test:e2e:runner:unit`, `pnpm test:e2e:runner:typecheck`, and `pnpm
test:e2e:runner -- --list --suite native-connection-guidance`. The
measurement uses
`PAPERCLIP_NATIVE_PROCEDURE_MEASUREMENT=/tmp/connection-measurement.json
pnpm exec vitest run --project @paperclipai/server
server/src/__tests__/native-procedure-measurement.test.ts`. Validation
used pinned pnpm 9.15.4.
- No paid provider campaign was started. There is no baseline/candidate
behavior result for these new cells.
- The audit records 654 UTF-8 bytes of fixed connection guidance. A
clean capture at `a04b8c6a452315625014888335d45670a2094fb6` confirms 41
supplied tools, 53,341 normalized bytes at start/resume, 50,949 at
compact continuation, and a 48,195-byte authenticated OpenCode MCP
catalog. These are byte counts, not tokens, bills, vendor-private prompt
sizes, or savings from this PR.

## Risks

- This is eval preparation. Passing support tests do not establish live
model behavior or qualify an instruction reduction.
- The explanation oracle checks attributed saved output. It does not
prove cognition or arbitrary prose truthfulness. One saved interaction
also does not prove the absence of repeated idempotent tool calls.
- Successful new authentication and tool refresh, existing-connection
agent grants, independent work while waiting, explicit retry after
decline, and blocking when mandatory work remains still need separate
coverage.
- Notion setup decline does not execute a real Notion service. Positive
service approval uses an already installed deterministic service; it
does not qualify new connection creation.
- The original historical failures remain unchanged. Future comparisons
must freeze source, fixture, model, input, and grading controls and
retain every actual attempt.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, and code
execution. The exact serving model ID and context-window size were not
exposed in this session; they are not inferred. No model provider was
invoked by the eval suite in this PR preparation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-06 21:28:16 -05:00