Commit Graph
6556 Commits
Author SHA1 Message Date
Dotta 40f4215186 test(evals): synchronize source5 fixture observer declarations 2026-11-03 00:30:57 -06:00
DottaandPaperclip 36ec1aa4fb test(evals): inspect authored checklist as rich text blocks
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 00:30:23 -06:00
DottaandPaperclip 166815c13a test(evals): bind multipart source observation to original deadline
Preserve the original case budget and require the saved question and canonical completed source run before human form actions.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 00:25:30 -06:00
DottaandPaperclip 075651b15c test(evals): match canonical controller secret keys
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 00:20:14 -06:00
Dotta a2d4980eb5 fix(chat): bound proposal metadata before trimming in candidate 2026-11-03 00:16:27 -06:00
DottaandPaperclip 0a76dbde9b test(evals): inspect announcements on normal company surfaces
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 00:16:27 -06:00
Dotta 51ef450754 test(evals): synchronize repaired current case declarations 2026-11-02 23:43:49 -06:00
Dotta 48cfd4a349 fix(evals): observe supported owned CLI profile guidance 2026-11-02 23:43:07 -06:00
DottaandPaperclip 80d7992590 fix(evals): bind SDK public proof to reviewed runtime lock
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 23:42:40 -06:00
Dotta fd69dde707 fix(chat): expose reviewed redacted tool proposal in candidate 2026-11-02 23:38:41 -06:00
DottaandPaperclip 332431de44 test(evals): inspect bound full app proposals before decisions
Pair the ordinary shared proposal disclosure with exact pending invocation arguments and actual provider dispatch. Keep the short preview valid, reject uninspectable or changed content, and retain the original three human/native permission phases.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 23:38:26 -06:00
DottaandPaperclip baa5479225 test(evals): read owned timeline bytes with fixture authority
Keep worker credentials limited to task validation and use the existing trusted fixture client for the fixed board-only file route after a live scope/checkout join. Preserve actual bounded command and receipts; reject changed custody before publishing the exit result.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 23:38:26 -06:00
DottaandPaperclip 636dec32fd test(evals): observe primary guidance within its original deadline
Bound only the initial reply wait to the existing remaining 300-second case clock, checking before and after. Preserve actual terminal reply, personal permission controls and all settled Chat final verification.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 23:38:26 -06:00
DottaandPaperclip 459c5d784f test(evals): bind app test evidence to normalized MCP result
Reconstruct the documented gateway result independently, retaining raw provider and source attribution checks. Preserve the original failure and mark the oracle v3 changed condition.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 23:38:26 -06:00
DottaandPaperclip 37f27fe999 test(evals): synchronize reviewed price and plan source4 declarations
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 22:17:05 -06:00
DottaandPaperclip d272fcff5a test(evals): bind plan decisions to actual source deliveries
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 22:16:25 -06:00
DottaandPaperclip 64da56f835 test(evals): use a genuinely pricier hire proposal fixture
Use the public Pro configuration as the incidental starting proposal, verified against current official prices and every selected 1017 rate variant. Keep all comparison, catalog, identity, configuration and pending-approval guards unchanged. Record the starting-model change as definition v3, never a historical regrade.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 22:16:24 -06:00
Dotta 43c44ccc82 test(evals): bind app team and announcement repair declarations 2026-11-02 21:53:05 -06:00
DottaandPaperclip 6743055493 test(evals): reject explicit missing target access reports
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 21:49:34 -06:00
DottaandPaperclip a74bbc949b test(evals): reject contradictory app test reports
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 21:49:34 -06:00
DottaandPaperclip b23439902a test(evals): qualify authorized target-scoped app tests
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 21:49:34 -06:00
DottaandPaperclip 4837c96544 test(evals): scope announcement dismissal staging input
Enable only the singleton dismissal journey against a reviewed public synthetic staging feed. Require its real server publication before any chat and preserve actual human-only personal dismissal and audience checks. Publication is a separate explicit operator action.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 21:49:34 -06:00
DottaandPaperclip d12be2755b test(evals): recognize exact team account probe refusal
Accept the documented OpenCode auth-required warning as a missing-account prerequisite while keeping a distinct member strictly ready. Retain only the existing closed probe projection before setup refusal; no new probes, model calls or clock changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 21:49:33 -06:00
Dotta 7ff7bacadf test(evals): synchronize next repair declarations and Studio accounting 2026-11-02 21:35:30 -06:00
Dotta 43813e06d0 test(evals): bind SDK public proof to the evaluated source tree 2026-11-02 21:33:04 -06:00
DottaandPaperclip 9ce8b0d274 test(evals): synchronize SDK public source pins offline
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 21:28:36 -06:00
DottaandPaperclip af3b925d04 test(evals): account for actual Studio result delivery
Read immutable pinned version detail and require the source-bound native completion turn. Preserve the original query, model, 540-second attempt and actual failing-sample output requirements.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 21:28:36 -06:00
DottaandPaperclip a5510186ae test(evals): retain closed packaging budget diagnostics
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 21:16:38 -06:00
DottaandPaperclip d432bc425b test(evals): qualify skill update reads before model admission
Require the actual pinned update-status prerequisite before the canonical request. Add an exact human-session GET reader that bounds error-body parsing and preserves only three source-owned reason codes and finite provider statuses; generic response-body withholding is unchanged. Fail closed on unknown, oversized, malformed or private error content. Stop retrying typed update-read refusals while retaining all real source, review, update, version and fork assertions. This improves diagnosis and eligibility, not feature grading.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 21:16:38 -06:00
DottaandPaperclip e55b3b025f test(evals): respect app review preview content
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 21:16:38 -06:00
DottaandPaperclip 481722349e fix(evals): open scoped app review in composer takeover
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 21:16:38 -06:00
Dotta 486cde8748 fix(candidate): clarify authored skills versus source imports 2026-11-02 21:14:43 -06:00
Dotta 36dff2eab3 fix(evals): synchronize repaired declarations before catalog admission 2026-11-02 21:02:29 -06:00
DottaandPaperclip 65a3e8eef8 fix(evals): retain SDK scan audit after bootstrap rejection
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:58:49 -06:00
DottaandPaperclip 462af87e84 test(evals): wait for settled primary Chat evidence
Require the actual recent CEO conversation, original visible human and native reply, and editable composer after navigation and final profile proof. Re-read the persisted Morgan preference and exact original chat/run set after readiness. Use only remaining original case budget and preserve all existing primary/privacy grading. Calibrate the actual final verifier with loading, late side-effect, wrong identity and exhausted-budget negatives.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:58:49 -06:00
DottaandPaperclip b7dfdbbefd test(evals): preserve consumed commentary identity response
Dispose only unread browser-session response bodies. Exercise the actual callback with real Node Responses so successful JSON is not followed by locked-stream cancellation. Preserve real reader and owner identity checks, restoration, denied-body disposal, malformed JSON failures, feedback privacy oracles, and original clocks.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:58:49 -06:00
DottaandPaperclip 0ecd43035a fix(evals): bound status-card setup by original case clock
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:58:49 -06:00
DottaandPaperclip fa644b9b20 test(evals): bind deferred question delivery to its actual successor
Accept a null-target durable wake receipt only with the unique succeeded answer-bound continuation from the server response path. Keep actual human answer, company/task/agent, ordering, duplicate conflict, saved output and full run accounting checks. Version the observer and retain failed history.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:58:49 -06:00
DottaandPaperclip 41743cb0ae test(evals): observe required question validation on Next
Replace the disabled-button assumption with real invalid-click validation, no page advance, no saved answer and no new work. Keep all subsequent exact answers, page retention and continuation checks. Calibrate the real QuestionForm with a pure React test; no product UI behavior changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:58:49 -06:00
DottaandPaperclip 12e6def200 test(evals): mutate readonly context only through fixture setup
Keep the exhausted-budget negative while respecting the readonly public context type. No production observer or grader changes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:58:49 -06:00
DottaandPaperclip bd4dc1f10b test(evals): inspect authored skills through normal Studio navigation
For the reusable-skill request, prove ordinary library discovery, editable Studio checklist content and reload with the same saved version instead of prescribing a chat-link format. Keep current-version/content/attribution/assignment checks and all other skill contracts unchanged; record the explicit authoring v2 conditions.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:58:49 -06:00
DottaandPaperclip d2826233b3 test(evals): align goal link calibration with UI evidence
Keep the aggregate no-chat-link expectation consistent with the reviewed v2 normal Goals UI/readback oracle. Missing hierarchy inspection remains a failing negative control.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:20:29 -06:00
DottaandPaperclip b65a906c87 test(evals): name the requested Fast response feature
Clarify the feature-specific request without injecting evaluator context or changing the fixture, service oracle, permissions, model or clocks. Retain a regression that rejects an instruction-only acknowledgement change.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:18:36 -06:00
DottaandPaperclip 750c9f517b test(evals): grade inspectable goal hierarchies with optional owners
Version the goal-tree oracle against the product contract: ownership is optional unless requested, assigned owners must belong to the company, and normal Goals navigation plus public readback proves inspectability. Keep exact hierarchy, attribution, native work, explicit retry identity and progress assertions; keep original failed history.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:18:31 -06:00
DottaandPaperclip 615b8e6a9a fix(evals): copy readonly skill source inventory
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:18:31 -06:00
DottaandPaperclip cbecbfc788 fix(evals): read immutable skill version details for content evidence
Version listings contain inventory metadata without file bytes. Fetch source-scoped immutable version details and preserve version, author and inventory identity for content and history observations. Keep link, import, semantic and native feature graders unchanged.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:18:31 -06:00
DottaandPaperclip 0697887371 fix(evals): establish and preserve independent tree budget holds
Use the existing positive-cap and explicitly synthetic accounting-import fixture pattern instead of zero, which disables enforcement. Preserve the held branch policy while independently verifying company, CEO and active branches remain funded; retain all real tree stop/pause/hold oracles.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:18:31 -06:00
DottaandPaperclip 6e5a430118 fix(evals): preserve assignment permission in team fixture request
Use the complete public permissions schema and retain the independently observed same-company CEO assignment capability. Reject foreign or unverified actor metadata before writing, and verify the effective capability remains unchanged.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:18:31 -06:00
DottaandPaperclip 78d91b0e2f fix(evals): admit verified missing personal secret prerequisite
Prove the controller and worker ready before introducing the missing required current-person secret. Preserve that specific scenario only after exact owner-scoped missing-binding resolution; reject unrelated lifecycle, hold, identity and configuration failures. Keep feature grading, human secure entry, native delivery, clocks and privacy assertions unchanged.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:18:31 -06:00
DottaandPaperclip bb33df5320 test(evals): bound wrong-reviewer observation to its case clock
Use the existing remaining original deadline for the wrong-reviewer native reply before the real human denial probe. Preserve reviewer identity, request state, permission and no-work assertions.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 20:18:31 -06:00