PaperClipAI/tests/runner-e2e/QUALIFICATION-2026-09-21.md

Native onboarding and Agent Chat qualification — 21 September 2026

This record distinguishes live behavior from the existence of an eval. No native onboarding default or production prompt was changed. All paid work ran on isolated GitHub Actions workers with real providers, Chromium, Paperclip, and public APIs.

Native onboarding

Campaign 35656761484 on master 846336e5a0d3003e1b938e8c983aa5a65bda81ff: 26/26 passed, with 26/26 cleanup passes. The 13 cases ran on both native Codex and native Claude: interviews, clear/ambiguous requests, plain messages, explicit plans, ordinary-task isolation, plan acceptance, card/reply acceptance, acceptance during active work, clarification, revision, and rejection. The observed completed models were gpt-5.6-sol and claude-sonnet-5; some Codex first-response cases stop on an answerable native question before model metadata is emitted.

The real wizard creates the agent. The existing fixture switches its runtime through the public API before the first provider run, preserving the wizard's model choice, persona, skills, and task. This qualifies the pre-default native first-task process. It does not qualify native selection in the onboarding UI or production rollout, neither of which is enabled by this change.

Initial new-case campaign

Campaign 35657128077 on 3ca6d25e0d20dd36e74c0868872802856e5503fc retained all six failures:

• Active reassignment, both providers: the fixture sent an instructions bundle to the general agent PATCH endpoint. It did not update the worker's managed file, so the worker finished before the intended wait. The active boundary was never exercised. Use the instructions-file endpoint and verify its persisted content.
• Answer quality, both providers: the prompt did not specify whether the blocker field should contain only its reference or also its description, and did not explicitly exclude the current chat from the active-run count. Codex returned the reference plus its correct description; Claude correctly counted this chat and explained that no other work was active. Claude also put the explanation outside the JSON. These attempts do not establish a factual reasoning failure. The fixture now specifies the JSON shape, reference-only field, and task scope.
• Worker crash, both providers: after the verified worker loss, Retry was visible and accepted. The second run immediately failed with native_session_cleanup_quarantined. The saved plan and original message survived, but no usable recovered answer was produced. This is a real recovery boundary, not a model failure. Reported billing is incomplete for these crashed runs; absent usage must not be described as zero cost.

The fault helper now uses a Linux pidfd, checks the command's exact run ID and start ticks, and signals the owned handle. Tests reject changed identity and dead handles. Linux CI also exercises a real disposable child. macOS does not run this fault case; the supported qualification host is Linux.

Recovery correction

A quarantined failure can precede creation of its coordinator row. The execution projection now preserves recovery_needed even without that row or with a stale retryable coordinator. The generic failed-run retry endpoint refuses this exact native quarantine instead of admitting another doomed run. An explicitly reconciled successor still clears the historical projection correctly.

The worker-crash eval remains non-passing when recovery is unavailable. It records that saved work survives, verifies the unavailable Retry UI and HTTP 409, and reports the missing usable recovery. Passing a guard test is not recovery qualification. Selecting verified cleanup plus a fresh attempt versus exact-session resume remains a product decision; neither is silently enabled here.

Answer-quality review

Review method: inspect retained user comments, agent comments, question/confirmation cards, and durable task outcomes. This is an agent-authored semantic review, not an independent LLM-judge score and not a claim about all possible questions.

The 26 onboarding recordings show focused questions for ambiguous requests, concrete proposals for clear requests, preservation of the supplied day/fee/audience, and respect for revision and rejection. The first status answers in the new campaign correctly identify the resolved budget issue, current venue blocker, deferred printing, and unknown attendance count. The second-turn counterfactual questions were not reached in that initial campaign.

Two user-facing limitations remain worth deciding:

• One Claude acceptance reply exposed internal wording: “per the confirmation proposal mode.” This conflicts with the existing skill's wording preference.
• Five of the six Claude acceptance journeys left a future-tense handoff as the latest parent-thread reply, even though the child later completed. The saved child output and task status passed. Codex posted completion follow-ups in the corresponding journeys. Decide whether a final parent-thread completion notice should be a required outcome; the current onboarding oracle does not require it.

Production instructions were preserved. These observations must not be hidden by changing prompts solely to make the benchmark green.

Grounded status answers: corrected live proof

Campaign 35658262695 on 4a26f10be7dd6aeabf2ca7b44b3d6b2ece7817e0: 2/2 passed, both cleanup passes. Each provider answered both turns, preserved both source tasks, and started no execution on either task. The original failed attempts above are retained; this is a new campaign with an explicit output contract.

Semantic review of all four retained replies against the task descriptions and chronological comments found:

• Both identified venue confirmation as the current blocker, printing as deferred, no task execution, and unknown attendance. Both gave the useful next step of confirming the venue before printing.
• Both rejected the obsolete budget claim and unsupported printing claim on the follow-up, and distinguished a planned Friday from a guaranteed calendar date. Neither invented a venue, date, or attendance count.
• Codex's explanations were compact and clear. Claude's second explanation was longer but readable and grounded in named task records. Its phrase “no venue has been confirmed” is slightly stronger than “no confirmation is recorded”; its immediately following quotation and null fact make the evidence limitation clear.

This qualifies these two-turn grounding stories, not general answer quality, statistical reliability, multilingual behavior, or arbitrary long conversations. The five review dimensions remain factual grounding, stale-premise correction, honest uncertainty, useful next step, and clear prose. No production prompt changed.

Active reassignment: corrected live proof

Campaign 35659014397 on cf6d4ae3a8576d822861bc916e7f536d0a0b7bc7: 2/2 passed, both cleanup passes, using gpt-5.6-sol and claude-sonnet-5.

The boundary snapshot proves the original worker was running with a saved draft. Chat then reassigns the existing task to a second agent. The original stops with issue_reassigned before the successor starts; exactly one successor completes the same task. The plan, scope, and original draft revision survive. The successor may revise the canonical document, and its final contribution must be attributed to that successor. The audit records the native reassignment tool.

The preceding campaign 35657945095 on 6039b02ed remains failed: both handoffs actually completed, but the draft oracle incorrectly required the latest document to remain frozen. Both successors legitimately revised that document. The new oracle requires the exact original revision to remain retrievable through the history API while allowing progress. Negative calibration rejects deletion or alteration of that revision. Both crash cases in the preceding campaign independently reproduced cleanup quarantine.

Measurement limits

All selected environments here are local Linux GitHub Actions workers, not Daytona-hosted execution or the user's staging company. The result JSON retains source SHA, suite definition hash, profile/model, attempt, timing, token usage, and billing coverage; initial failures are never regraded as passes.

Onboarding reports complete billing for only 4/26 results; active handoff reports incomplete billing for both interrupted workers; the two answer-quality results report complete coverage. Provider-reported monetary cost is zero while billing type is unknown, and local runtime is not metered. This does not establish zero actual spend. Recorded total tokens (including cached input) are 9,812,045 for onboarding, 825,131 for final handoffs, and 1,273,098 for grounded answers.

Public campaign viewers use the standard Product E2E history with the gha-<workflow-run-id>-<attempt> campaign identifier. These are bounded story qualifications, not a claim that all native-runner reliability is solved.

Crash guard follow-up

Campaign 35658772755 did not reach provider cases: GitHub artifact finalization returned HTTP 403. A replacement campaign 35659580100 on a11bd236e33833dc081cc3702baa3d3f98d8d12f retained two failed recovery attempts, both with successful disposable cleanup. The API correctly refused Retry with 409 and created no second run, but the eval then waited for a run that had never been admitted.

Retained network evidence explains the transition: provider_transport_failed first schedules a same-run retry; that attempt then reaches cleanup quarantine. The fixture now waits for this recovery classification before requesting a user retry. Screenshots also exposed a separate Retry button in the task-recovery banner. That banner previously recognized only native continuation reconciliation; it now also recognizes cleanup quarantine and links to Inspect run. Its regression test failed before the fix and passes afterward. The live guard must verify this link is rendered, no Retry remains, the API refuses retry, and saved work survives.

These corrections do not supply a successful post-crash continuation. Worker-crash recovery remains unqualified until an agreed recovery policy is implemented and its successful outcome passes the eval.