diff --git a/.agents/skills/diagnose-why-work-stopped/SKILL.md b/.agents/skills/diagnose-why-work-stopped/SKILL.md index 91caa62fa8b..d5a14886d94 100644 --- a/.agents/skills/diagnose-why-work-stopped/SKILL.md +++ b/.agents/skills/diagnose-why-work-stopped/SKILL.md @@ -1,157 +1,34 @@ --- name: diagnose-why-work-stopped -description: > - Diagnose stalled, looping, or over-recovered Paperclip issue trees and propose - a no-code product-rule plan. Use when asked why work stopped, why it looped, or - how to prevent a tree from going too deep. +description: Compatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope. --- # Diagnose Why Work Stopped -A repeatable procedure for the recurring class of issues where the user (or a manager) points at a stalled / looping / over-recovered issue tree and asks "why did this stop / why is this looping / how do we make sure this doesn't happen again?" +Use [Fix Failed Paperclip Run](../fix-failed-paperclip-run/SKILL.md). It owns the current +workflow: find the first causal failure, reproduce it red, repair the shared +contract, test second-order effects, verify the real workflow, and deliver a PR +with passing checks. A normal request to fix stopped work continues through that +workflow without a separate plan-approval gate. -This skill is **diagnostic + product-design**, not engineering. The output is a written root cause and an approved plan. No code changes leave this skill. +If the user or calling workflow explicitly asks for diagnosis only or requires +approval before implementation, perform the investigation and safety analysis +within that scope. For example, a Terminal-Bench diagnosis child does not override +the loop's board approval boundary. Keep the next action and its owner explicit. -Canonical execution model: read `doc/execution-semantics.md` before diagnosing or proposing a new liveness/recovery rule. Use that document as the source of truth for status, action-path, post-run disposition, bounded continuation, productivity review, pause-hold, watchdog, and explicit recovery semantics. If the investigation finds a true product-rule gap, the plan should say whether `doc/execution-semantics.md` needs a matching update. +For issue-tree diagnosis, read the current `doc/execution-semantics.md`, inspect the +relevant subtree and recent repairs, and classify non-progressing nodes: -## When to use +- **Genuine wait:** a named human or external dependency must act. +- **Agent-actionable but unrouted:** authorized useful work lacks a live path. +- **Already covered:** existing work owns the next action; do not duplicate it. -Trigger on an assignment whose title or body matches any of: +Preserve these three invariants together: -- "why did this work stop", "why did this stall", "why did this just stop" -- "infinite loop", "looping", "spinning", "going too deep", "recovery went too deep" -- "liveness — what happened here", "this tree stopped working", "stuck" -- "approach it from a product perspective", "general product principle / rule" -- An attached link to a specific stalled / looping / over-recovered issue tree +1. Productive work has a real next action and owner. +2. Only genuine blockers stop work, including approval, credentials, and budget. +3. Recovery is bounded and cannot recursively recover itself. -Also use when the user asks for forensics, root cause, or a write-up *before* any product change. - -## When NOT to use - -- The assignment asks you to ship a code change directly. Use normal engineering flow. -- The assignment is a normal bug report against a specific feature. Use normal investigation. -- You are the original implementer being asked to fix your own bug. Use normal debugging. - -## Three invariants you must preserve - -Every diagnosis and every proposed rule must hold these three invariants together. The user has restated them on at least four issues; treat them as load-bearing: - -1. **Productive work continues.** Agents that have a clear next action must keep working without needing the user to wake them. ([PAP-2674](/PAP/issues/PAP-2674), [PAP-2708](/PAP/issues/PAP-2708)) -2. **Only real blockers stop work.** Stops happen when something genuinely cannot proceed (missing approval, missing dependency, human owner). Pseudo-stops (in_review with no action path, cancelled leaves, malformed metadata) must be detected and routed, not left silent. ([PAP-2335](/PAP/issues/PAP-2335), [PAP-2674](/PAP/issues/PAP-2674)) -3. **No infinite loops.** Stranded-work recovery and continuation loops must be bounded and distinguishable from genuinely productive continuation. ([PAP-2602](/PAP/issues/PAP-2602), [PAP-2486](/PAP/issues/PAP-2486)) - -If a proposed rule violates any of the three, drop it or rework it. State explicitly in the plan how each invariant is held. - -## Procedure - -### 0. Read the current execution contract - -Before walking the tree, read `doc/execution-semantics.md` and keep its terms intact: - -- live path / waiting path / recovery path -- post-run disposition: terminal, explicitly live, explicitly waiting, invalid -- bounded `run_liveness_continuation` -- productivity review vs liveness recovery -- active subtree pause holds -- silent active-run watchdog - -Do not invent a new rule until you can state how it differs from the current execution semantics document. - -### 1. Forensics on the named tree — before anything else - -Do this in the same heartbeat. Do not propose a rule until you have a concrete stop point. - -- Open the linked issue (and its blocker chain, parents, recovery siblings, recent runs). -- Walk the tree node-by-node and find the exact issue + state combination that stops the world. Common shapes seen in the company so far: - - `in_review` with no typed execution participant, no active run, no pending interaction, no recovery issue ([PAP-2335](/PAP/issues/PAP-2335), [PAP-2674](/PAP/issues/PAP-2674)). - - `in_progress` after a successful run with no future action path queued ([PAP-2674](/PAP/issues/PAP-2674)). - - Blocker chain whose leaf is `cancelled` / malformed / cross-company-inaccessible ([PAP-2602](/PAP/issues/PAP-2602)). - - `issue.continuation_recovery` waking the same issue >N times after successful runs ([PAP-2602](/PAP/issues/PAP-2602)). - - Stranded-work recovery treating its own recovery issues as more recoverable source work ([PAP-2486](/PAP/issues/PAP-2486)). -- Quote the evidence: run ids, comment timestamps, status transitions. "Inferred" is acceptable only when an API boundary blocks direct evidence — say so explicitly and mark the claim provisional ([PAP-2631](/PAP/issues/PAP-2631)). - -Respect the API boundary. If the linked issue is in another company and your agent token returns 403, do not bypass scoping. Either request a board-approved diagnostic path or proceed from inferred PAP-side evidence and label it. - -### 2. Survey recent related work - -Before proposing a new product rule, read what already shipped this week in the same area. The user has explicitly called this out: ([PAP-2602](/PAP/issues/PAP-2602)) "review our recent work on liveness that we shipped in the last couple of days." A new rule that contradicts code merged 48 hours ago is rework, not improvement. - -Quick survey: -- Recent merged PRs in the affected area. -- Recent done issues whose title mentions liveness, recovery, productivity, continuation, or the affected subsystem. -- Any active plan documents on parent issues. The fix may belong as a revision to an existing plan, not as a new top-level proposal. - -State in the forensics: "I reviewed X, Y, Z. The new gap is …" - -### 3. Classify each non-progressing issue in the tree - -For every issue in the affected tree that is not `done` / `cancelled` / actively running, decide: - -- **Truly needs human or board intervention** — name the owner and the action. -- **Agent-actionable but not currently routed** — name the rule that would have routed it, and the agent that should have been waked. -- **Already covered** — point at the active run, queued wake, recovery issue, or pending interaction. - -This is the table the user has asked for repeatedly ([PAP-2335](/PAP/issues/PAP-2335)). Without it the plan is abstract. - -### 4. Frame as a general product rule - -The user does not want a one-off patch on the named tree. They want the rule. Two checks: - -- The rule is **stated as a contract**, not as an if/else patch. Example contract: "every agent-owned non-terminal issue must finish each heartbeat with a terminal state, an explicit waiting path, or an explicit live path" ([PAP-2674](/PAP/issues/PAP-2674)). -- The rule is reconciled against `doc/execution-semantics.md`. Prefer citing and applying the existing contract; propose a document change only when the current doc is incomplete or contradicted by accepted/implemented behavior. -- The rule **explicitly preserves the three invariants** above. Show the work. - -If the rule would have blocked a recent productive run from succeeding, drop or narrow it. - -### 5. Plan, do not code - -Write the plan into the issue's `plan` document. Cover: - -- Forensics summary (root cause + evidence). -- The general product rule, stated as a contract. -- Whether the existing `doc/execution-semantics.md` contract already covers the case, or what exact documentation update is needed. -- Phased subtasks: typically `Phase 0` resolves the named live tree (carefully, not destructively), `Phase 1` codifies the contract in docs, then implementation phases for detection, recovery, UI surfacing, security review, QA, and CTO review. -- Explicit assignees per phase; favor team specialty (CodexCoder for server, ClaudeCoder for FE, UXDesigner for visible state, SecurityEngineer for ownership/permissions, QA for validation). -- Blocking dependencies wired with `blockedByIssueIds`, parallel branches identified. - -Do not create the child issues yet. Do not push code. - -### 6. Request approval, then decompose - -- Open a `request_confirmation` interaction targeting the latest plan revision. Idempotency key `confirmation:{issueId}:plan:{revisionId}`. -- Wait for board/CTO acceptance. If the user posts a new comment that supersedes the plan, the prior confirmation is invalidated — open a fresh confirmation tied to the new revision ([PAP-2602](/PAP/issues/PAP-2602) cycled three revisions; that is fine). -- Only after acceptance: create the phased child issues with the right assignees and dependencies, then block this parent on the final QA / CTO review issue so the parent only wakes when the chain finishes. - -### 7. Phase 0 hygiene on the named tree - -Phase 0 cleans up the live tree without papering over evidence: - -- Move stalled `in_review` leaves with no participant to `todo` with a precise next action and named owner ([PAP-2335](/PAP/issues/PAP-2335)). -- Detach cancelled/dead blockers from chains they were holding hostage; do not silently mark issues `done` to clear backlog. -- Leave a comment on the original named issue summarizing what changed and why; never hide the recovery chain history. - -### 8. Final close-out - -When the phase chain is complete, post a board-level summary comment on the parent issue: what changed, what the new contract is, what the rollout step is (e.g. "restart the control-plane to pick up the new response shape"), and the live state of the originally-named tree. Then close the parent. - -## Pitfalls - -- **Coding before approval.** The user has said "make a plan first" on every recent diagnostic issue. Producing code in the forensic phase wastes the round-trip. -- **Restating one invariant at the cost of another.** Bound continuation too tightly and productive work stalls; loosen recovery and infinite loops return. Always check all three. -- **Skipping the recent-work survey.** Proposing a contract that contradicts what shipped 24 hours ago is the easiest way to get the plan rejected. -- **Letting "in_review" mean done.** A leaf assigned to another agent with no participant or active run is not progress; treat it as a stop. -- **Bypassing company scoping.** Cross-company forensics needs a board-approved diagnostic path, not a database read. -- **Recursive recovery.** Stranded-work recovery that recovers its own recovery issues is the canonical infinite loop ([PAP-2486](/PAP/issues/PAP-2486)). Detect it and refuse to deepen. -- **Hiding the chain.** Don't silently delete or hide the symptomatic recovery issues — the operator needs the audit trail. - -## Verification checklist (before posting the plan) - -- [ ] The exact stop point in the named tree is identified with run ids / comment ids. -- [ ] Recent shipped work in the same area was surveyed and is referenced. -- [ ] Every non-progressing issue is classified human-needed / agent-actionable / already-covered. -- [ ] The proposed rule is stated as a contract, not a patch. -- [ ] All three invariants are explicitly preserved. -- [ ] No code change has landed in this heartbeat. -- [ ] A `request_confirmation` against the latest plan revision is open. -- [ ] Phase 0 of the plan addresses the live named tree without destroying evidence. -- [ ] Implementation phases name specialty-appropriate assignees and `blockedByIssueIds` dependencies. +Frame any repair as a general behavioral contract, check it against a productive +counterexample, and follow the new skill through the authorized finish line. +Historical task links and private incident access are not required. diff --git a/.agents/skills/fix-failed-paperclip-run/SKILL.md b/.agents/skills/fix-failed-paperclip-run/SKILL.md new file mode 100644 index 00000000000..3d97e50f8ce --- /dev/null +++ b/.agents/skills/fix-failed-paperclip-run/SKILL.md @@ -0,0 +1,204 @@ +--- +name: fix-failed-paperclip-run +description: Diagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks. +--- + +# Fix Failed Paperclip Run + +Own the failure from evidence through a tested repair and a green PR. A diagnosis, +a proposed plan, a successful provider turn, and a passing unit test are intermediate +results. The finish line is a verified user workflow and a PR whose current commit +passes the required checks and review. + +Honor an explicit diagnosis-only request or a calling workflow's approval boundary. +Otherwise proceed through implementation without a separate plan-approval pause. +Carry forward existing authorization for merge, rollout, and recovery; do not ask +again. When the assignment ends at a PR, distinguish local verification from +deployment and recovery of the original task. This skill does not itself authorize +merging or deploying. + +## 1. Establish the incident and the finish line + +Start from the supplied task/run URL or failure report. Resolve these facts from +available tools and records before asking the user for missing information: + +- Instance, company, task, affected run(s), time window, and intended useful result. +- Current state versus state at failure; parent/child tasks and pending interactions + only where they control progress. Multiple failed runs can have different causes. +- The code actually served, runner/provider pack versions, operating system and + architecture, adapter, model/effort, execution workspace, trust mode, and credential + source. Trace effective values back through task overrides, inherited settings, + defaults, and saved session state. Redact credential values. +- Which endpoint is authorized: verified fix and green PR, or also rollout and + recovery. Preserve the requested work and success criteria throughout. + +Use authorized diagnostic access. If company scoping blocks a read, use an +already-authorized diagnostic path or label the missing evidence; do not bypass +the boundary. + +Read repository instructions and the relevant current execution contract. In this +repository, use `doc/execution-semantics.md` for ownership, continuation, waiting, +finalization, and recovery. Inspect recent changes and open/merged repairs in the +affected area. A correct fix that has not reached the target is a deployment gap; +do not create a second patch without evidence that the first is insufficient. +Choose code repair, configuration repair, or recovery from that evidence. If an +existing fix is sufficient, verify it and continue to the authorized endpoint +using its PR; do not manufacture code changes for an already-fixed defect. + +Keep a compact evidence record in the working notes or driving issue: environment, +revision, causal timeline, reproduction, safety assumptions, and acceptance result. +Preserve useful logs and saved work before retries or cleanup. Do not put secrets +or private incident links in public code, fixtures, skills, or PR descriptions. + +## 2. Find the first causal failure + +Reconstruct the run from dispatch through preparation, provider execution, tool +calls, result persistence, finalization, and task disposition. For chat/email work, +follow outbound delivery and the user-visible result too. Align run events, logs, +controller/provider records, task activity, and traces by time and identity. + +Separate these questions: + +1. What first prevented useful work or its delivery? +2. Why did the responsible component make that decision or emit that value? +3. Why did recovery or continuation fail to complete the already-authorized work? +4. Which later errors are consequences, misleading summaries, or independent bugs? + +Trace the causal value or transition to its producer and shared contract. Verify +the hypothesis against a counterexample or a reproduction. Check logs before and +after the apparent error: a recovered authorization warning may not be fatal, and +“final response,” HTTP 200, or “run succeeded” may not mean the task is complete. +Missing error detail is an evidence gap, not proof of a provider outage. When +evidence cannot distinguish hypotheses, name the uncertainty and add the smallest +bounded, redacted observation or fault injection that will resolve it. + +For a stalled tree, classify each relevant non-progressing node as genuinely +waiting on a named human/external dependency, agent-actionable but unrouted, or +already covered by existing work. Preserve all three invariants: + +- Productive work has a real next action and owner. +- Genuine approval, credential, budget, and dependency blockers remain effective. +- Recovery and continuation are bounded and cannot recursively recover themselves. + +Consult [failure patterns](references/failure-patterns.md) for matching mechanisms +and test oracles. Treat them as hypotheses to check, not a catalog of explanations. + +## 3. Prove the bug before changing production code + +Write the smallest behavioral regression at the boundary that owns the defect. +Run it against the unfixed code and capture the expected failure. A setup error, +missing dependency, or assertion about a mock's own return value is not the red. + +- Use the real policy/resolver/serializer and realistic persisted input. Mock only + boundaries outside the behavior under test. If a query duplicates rows, exercise + that query with the relational shape that produces duplicates. +- For recovery or races, control the interruption point and assert the resulting + ownership, command receipts, side effects, and continuation. Restarting between + completed turns does not reproduce a restart during an in-flight side effect. +- For packaging, process startup, platform, or cross-language defects, reproduce + using the actual affected artifact or protocol. A host binary or mocked provider + cannot prove that a sandbox image contains an executable compatible binary. +- For latency, measure the user's full interval and the suspected stage under the + relevant cold/warm conditions. Use a stable controlled regression where possible; + keep noisy wall-clock comparisons as measured acceptance evidence. + +Keep this failing regression as the starting point for the repair. If the defect +requires a real integration environment, capture a +repeatable failing integration probe before the fix and repeat it afterward; +add a durable regression at the closest reliable boundary. If no meaningful red +can be observed, report that gap and keep working on reproduction. Do not claim +red-green proof from a test that was first run after the fix. + +## 4. Repair the cause and prove the safety assumptions + +Fix the responsible producer or shared transition at the narrowest correct scope. +Check sibling callers/adapters that share that contract; generalize only as far as +the evidence requires. Avoid historical string allowlists, swallowed errors, +unbounded retries, or relaxing an integrity check to accommodate malformed input. +Preserve the requested model, trust, isolation, and performance behavior unless a +change or explicit fallback policy is part of the assignment. + +Before implementation, state the behavioral rule and the strongest plausible +second-order failure. Trace callers, downstream consumers, persisted state, and +runtime variants affected by the changed contract. Turn the important safety +assumptions into observable regression cases, not just assurances in the PR: + +| Changed boundary | Safety evidence to consider | +| --- | --- | +| Ownership, cancellation, retry, or continuation | No second executor or repeated external side effect; legitimate queued work still runs; cancellation remains effective. An unchanged assignee field alone is insufficient. | +| Recovery or finalization | Saved files, results, and sessions survive; known-complete work is not rerun; interrupted work continues once; indeterminate outcomes reconcile before retry. | +| Waiting or task disposition | Real approval/budget/credential gates still stop work; actionable work has a durable wake; recovery is bounded; no recursive recovery tree. | +| Input, tool, or credential plumbing | Company and trust boundaries hold; intentional configuration reaches execution without ambient secret forwarding; distinct tools still dispatch correctly; sensitive logging remains redacted. | +| Packaging, compatibility, or session identity | Relevant local/sandbox, platform, fresh/resumed, and old/new artifact combinations work; obsolete saved values are handled by a contract rather than a list of incident exceptions. | +| Performance or lifecycle | Warm reuse remains warm where intended; queueing and finalization do not regress; configured deadlines retain their meaning. | + +Select cases that exercise the actual blast radius. Do not create an unrelated +platform matrix. Test the sibling success path and the negative case that the old +guard protected. Keep reproduction and safety tests through later revisions. + +With the red established and the safety assumptions identified, make the repair +and run the same regression green. Run the relevant safety cases and existing +tests. Refactor only while they stay green. Investigate unexpected failures +instead of weakening an assertion or deleting the regression to finish. + +Routine infrastructure facts belong to the harness when it can determine them +under existing authority. Reconcile process ownership, saved results, or eligible +continuations in code instead of asking a user to certify those facts in a form. +Keep UI choices for genuine user intent, authorization, or unavailable information. + +## 5. Verify the real workflow at the fixed revision + +Apply the repository's `verify-paperclip` skill and affected map recipes when +available. Its essential requirement here is a real run through the affected user +entry point, using the fixed source or package and the relevant runtime configuration. +Run the narrow regression first so each hypothesis gets fast feedback; use a local +server with the real remote provider when that reproduces the failure. Do not wait +for a full deployment for every experiment when a faithful shorter loop exists. + +Exercise the original failure mechanism with a representative task: actual adapter, +provider, tools, trust boundary, persisted state, artifact, and interruption point +where each matters. A mock, Storybook, provider-free benchmark, or generic “hello” +smoke is supporting evidence only when the incident depends on a different path. + +Verify the useful output, persistence after reload/reconnect, finalization, and +correct task state with no stranded lock or pending operation. For resume and +conversation defects, run a second turn and check retained context/files. For +delivery defects, check the destination and user-visible outcome, not just receipt +of an HTTP request. For slowness, compare the end-to-end interval with the baseline. + +Record the tested revision and runtime versions. Re-run affected acceptance after +changes that invalidate that evidence. Report any missing credentials, access, or +environment precisely; an unavailable live check remains unverified. Continue +independent repair and PR work instead of substituting an unsupported “works” claim. + +## 6. Deliver the PR and close the incident to the authorized endpoint + +Use `prepare-paperclip-pr` and `prcheckloop` when available. Follow repository +instructions for checks, branch isolation, PR template, and review. Open the PR, +address review and CI failures, and inspect required checks and review on the +current head SHA. Pending, skipped, and passing are distinct. Retry a failed check +only with a reason; retain its evidence and fix repeatable failures. Resolve +companion repository/artifact changes needed by the fix and state their ordering. + +The PR must explain the causal defect, behavioral change, failing-before/passing- +after evidence, safety regressions, actual acceptance environment, and limitations. +Do not declare it ready while required checks or review remain unresolved. If a +required service or permission prevents completion, name the exact remaining gate. + +When rollout/recovery is already authorized, continue: confirm the served revision +and required artifact versions, re-read the original task, and recover it through +the normal product path. Reconcile existing work before starting another executor. +Observe the original useful result or a genuine, named remaining blocker. Do not +cancel, replace, or shrink the user's original task merely to make a test pass. + +Finish with a concise report: + +- Root cause and what changed, with unresolved hypotheses labeled. +- Red/green regression and second-order behavior checked. +- Real workflow exercised, revision/environment, and observed result. +- PR URL, current check/review status, and companion changes if any. +- Whether the original task is recovered, still awaiting rollout, or blocked on a + specific remaining dependency. Separate code readiness from incident recovery. + +Do not stop at a plan when a fix is authorized. Do not call the incident resolved +because its error message improved or a successful run badge appeared. diff --git a/.agents/skills/fix-failed-paperclip-run/references/failure-patterns.md b/.agents/skills/fix-failed-paperclip-run/references/failure-patterns.md new file mode 100644 index 00000000000..58874fd239b --- /dev/null +++ b/.agents/skills/fix-failed-paperclip-run/references/failure-patterns.md @@ -0,0 +1,138 @@ +# Failure patterns and useful regression oracles + +These are self-contained examples distilled from task investigations. They are +mechanisms to test, not assumptions about the current incident. Read only the +sections that match the evidence. Resolve current contracts and implementation +from the repository; no historical task or private conversation is required. + +## An interrupted turn becomes a permanent task failure + +**Mechanism:** A server restart restores session history, but the old active turn +is now failed or indeterminate. Recovery treats that provider status as a final +task outcome, or validates saved input against today's defaults. Useful work is +left behind a manual gate even though continuation was authorized. + +**Prove:** Interrupt at the observed point, including an in-flight tool call when +relevant. Exercise both a surviving process and a lost process. Reconcile command +receipts and preserve the session's input identity. Assert one owner, no duplicate +side effect, retained files/context, bounded continuation, and correct disposition. +Repeat an interruption to prove the recovery budget and genuine stops still work. + +**Trap:** A reconnect between completed turns proves less. A list of old prompt +strings does not define compatibility. Disabling warm reuse may hide the failure +while imposing a new startup cost on every turn. + +## A successful result gets stranded during finalization + +**Mechanism:** The provider completed useful work, but result persistence, artifact +copyback, checkpoint reconciliation, outbound delivery, or disposition failed. +The UI then shows either success without the deliverable or failure that invites +rerunning an already-complete external action. + +**Prove:** Retain the original result and side-effect receipt, fail the relevant +finalization stage, then recover. Assert the result is delivered/persisted once, +no provider work is repeated unnecessarily, the lock is released correctly, and +reload shows the final output. A known result can be recovered even if the original +task has since moved on; re-read current state before applying it. + +**Trap:** A successful provider turn, request acknowledgement, or status badge is +not proof that a user received the report, document, chat reply, or voice answer. + +## A routing repair permits a second executor + +**Mechanism:** A mention or chat follow-up targets someone other than the assignee. +A shortcut lets that agent execute without changing the assignee field. A unit +test proves the field stayed constant while both agents can now perform the work. + +**Prove:** Attempt the conflicting wake while the real owner is active. Assert +execution and external side effects cannot start twice, and the legitimate owner's +queued work still proceeds. Test cancellation and follow-up routing through the +real shared policy, including identity, company, and trust boundaries as affected. + +**Trap:** Database ownership alone does not prove execution ownership. Broadening +credentials or bypassing checkout is not a routing fix. + +## Runtime inputs are malformed before the model starts + +**Mechanism:** A relational join duplicates tools; names exceed a provider limit +after namespacing; dispatch binds a tool to the wrong connection; an explicit +environment variable is lost at an allowlist; execution and logging hash different +versions of a redacted payload. Later errors obscure the first invalid input. + +**Prove:** Exercise the real query, final name builder, environment filters, or +cross-language serializer with the failing input shape. Assert both the rejection +case and the intended valid case. Preserve distinct connection bindings, full +authorized execution input, redacted logs, and integrity checks. Where config is +cached, test added, changed, and removed values in fresh and resumed sessions. + +**Trap:** Deduplicating every same-named tool, disabling digest validation, or +forwarding all ambient variables can turn a startup fix into a correctness or +credential leak. Retrying the same invalid tool definition is not recovery. + +## The deployed artifact differs from the tested artifact + +**Mechanism:** A dependency exists in source but not the image actually served; a +test checks an obsolete image target; a native package skipped installation; a host +binary works while the sandbox platform cannot run it; a fix is merged but absent +from the tenant's served revision. Separately built artifacts may differ despite +sharing a source revision. + +**Prove:** Inspect the target revision and artifact identity, actual image target, +installed executable, OS/architecture, and launch protocol. Start the real packaged +provider in that environment. Exercise the relevant fallback/install path and a +second turn if session reuse is involved. Keep model/account compatibility separate +from process launch failures. + +**Trap:** Adding a dependency to a manifest or obtaining a 200 from a provider +probe does not prove the launched runner can execute it. A new patch is unnecessary +when the existing repair only needs to reach the target. + +## Recovery confuses useful waiting with a blocker + +**Mechanism:** Work awaiting CI, an eligible monitor, or a durable continuation is +marked blocked or abandoned. The opposite error removes a real approval, budget, +credential, or uncertainty gate to force progress. Recovery issues can recursively +create more recovery issues. + +**Prove:** Cover actionable work with a durable next wake, a genuine human wait, +an exhausted budget, already-covered work, and repeated recovery. Assert the right +next actor and bounded attempts. Let the harness determine facts it can observe; +reserve human input for decisions or facts it cannot establish. + +**Trap:** Changing a status label without scheduling useful work is not liveness. +An operator form asking whether an internal controller is dead is not a substitute +for ownership reconciliation. + +## “It works” hides a slow or incomplete user journey + +**Mechanism:** The model is quick, but preparation, shared repository locks, +database queries, remote startup, or finalization dominates latency. A local mock +benchmark removes the stage responsible for most of the delay. Separately, an +integration test may never reach real authentication or the destination output. + +**Prove:** Measure submit-to-usable-output and the critical stage with the affected +configuration. Compare cold and warm cases where relevant. Use the real remote +provider from a local harness when it shortens iteration without removing the +failure. Check actual output and persisted state, including the next turn. If +optimizing a query, verify equivalent results and authorization as well as speed. + +**Trap:** Extra trace spans only improve observability until a measured repair +improves the path. A fresh greeting or an OAuth setup screen does not establish that +the original task completed. Do not quietly change the model or effort to get a +faster number. + +## Error evidence is incomplete or misleading + +**Mechanism:** The UI reports a generic native failure, an earlier recovered +warning looks fatal, or an exception disappears between a tenant and an error +collector. An enabled reporting flag and successful requests do not prove a +particular event arrived. + +**Prove:** Preserve the original structured cause across the boundary with bounded +redaction. Distinguish event creation, transport acceptance, and recorded receipt. +Verify the concrete error is shown to the user when actionable. Test the failure +path itself, not only the happy-path logger configuration. + +**Trap:** Better future evidence cannot establish an uncaptured historical cause. +Report what remains unknown; do not turn an instrumentation PR into a claim that +the original task failure was fixed.