docs(skills): add fix-failed-paperclip-run workflow (#16830)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Failed or stalled tasks need a repair that restores useful work
without weakening execution safety.
> - The existing diagnosis skill stops at a plan and relies on private
historical task links.
> - A repair can pass a narrow test while still failing in the deployed
runtime or permitting duplicate work.
> - This PR adds a task repair workflow with
failing-before/passing-after evidence, safety tests, real workflow
verification, and PR checks.
> - The old skill name remains usable and preserves explicit
diagnosis-only or caller approval limits.

## Linked Issues or Issue Description

**Issue type**

Outdated documentation.

**Where is the issue?**

`.agents/skills/diagnose-why-work-stopped/SKILL.md`.

**What's wrong?**

The skill requires a plan approval even when implementation is
authorized. Its private task links do not provide usable guidance to
other contributors. It does not carry a repair through regression proof,
second-order checks, real workflow verification, and a green PR.

**Suggested fix**

Add `fix-failed-paperclip-run` and route the old name to it. Keep the
failure patterns self-contained. Keep explicit diagnosis-only requests
and the Terminal-Bench caller's approval boundary effective.

Related: #7658 established the maintainer skill location. A search for
the old and new skill names found no competing task-repair skill PR.

## What Changed

- Added a workflow from incident evidence through a verified repair and
current-head PR checks.
- Added eight standalone failure patterns with behavioral test oracles
and common verification traps.
- Replaced the old plan-only workflow with a compatibility entry point.
Preserved issue-tree classification and the three liveness invariants.
- Distinguished PR readiness, deployment, and recovery of the original
task. Reuse an existing sufficient fix instead of creating a duplicate
patch.

## Verification

- Rename commit `b979b632ee8f1ef57f1e7d6c37c711d1c83f5844`: both skill
validators, local reference checks, and whitespace checks pass. CI and
review are pending for this commit.

- The standard skill validator passes for both skill entry points.
- All Markdown references resolve. Referenced repository workflows
exist. The new files contain no historical task IDs, private URLs, or
machine-specific paths.
- `git diff --check` passes.
- Manually reviewed scenarios for an existing merged fix, unknown
historical cause, unsafe duplicate execution, live-only failure,
explicit diagnosis-only scope, and the Terminal-Bench approval boundary.
This is instruction review, not a live agent evaluation.
- Attempted `pnpm -r typecheck`, `pnpm test:run`, and `pnpm build`. All
stopped before execution because pnpm could not fetch and verify the
pinned version in the sandbox. This worktree also has no installed
repository dependencies. No runtime pass is claimed from these attempts.
- No product code changed. At head
`90ee31270420d2dac20dc3a41b12cddb5134d6ec`, 55 PR checks passed and 2
were skipped. Typecheck, tests, build, and the aggregate verification
gate passed in CI. Greptile is 5/5 with no inline findings or unresolved
review threads.

## Risks

- The old skill now continues into implementation when a fix is
authorized. Its explicit scope rule preserves diagnosis-only requests
and approval requirements from callers.
- This change is workflow guidance. Structural validation and scenario
review do not prove how a future agent will execute it on a live
incident.

## Model Used

OpenAI Codex, `gpt-6-astra`, with tool-assisted history inspection,
repository review, and Markdown authoring. The context window size is
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (skill validation;
repository-wide commands could not start, as noted above)
- [x] I have added or updated tests where applicable (no runtime
behavior changed)
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

.agents/skills/diagnose-why-work-stopped/SKILL.md

@@ -1,157 +1,34 @@
 ---
 name: diagnose-why-work-stopped
-description: >
-  Diagnose stalled, looping, or over-recovered Paperclip issue trees and propose
-  a no-code product-rule plan. Use when asked why work stopped, why it looped, or
-  how to prevent a tree from going too deep.
+description: Compatibility entry point for failed, stalled, looping, or over-recovered Paperclip work. Use fix-failed-paperclip-run for the full diagnosis, regression, repair, real verification, and green-PR workflow; honor an explicitly requested diagnosis-only scope.
 ---
 
 # Diagnose Why Work Stopped
 
-A repeatable procedure for the recurring class of issues where the user (or a manager) points at a stalled / looping / over-recovered issue tree and asks "why did this stop / why is this looping / how do we make sure this doesn't happen again?"
+Use [Fix Failed Paperclip Run](../fix-failed-paperclip-run/SKILL.md). It owns the current
+workflow: find the first causal failure, reproduce it red, repair the shared
+contract, test second-order effects, verify the real workflow, and deliver a PR
+with passing checks. A normal request to fix stopped work continues through that
+workflow without a separate plan-approval gate.
 
-This skill is **diagnostic + product-design**, not engineering. The output is a written root cause and an approved plan. No code changes leave this skill.
+If the user or calling workflow explicitly asks for diagnosis only or requires
+approval before implementation, perform the investigation and safety analysis
+within that scope. For example, a Terminal-Bench diagnosis child does not override
+the loop's board approval boundary. Keep the next action and its owner explicit.
 
-Canonical execution model: read `doc/execution-semantics.md` before diagnosing or proposing a new liveness/recovery rule. Use that document as the source of truth for status, action-path, post-run disposition, bounded continuation, productivity review, pause-hold, watchdog, and explicit recovery semantics. If the investigation finds a true product-rule gap, the plan should say whether `doc/execution-semantics.md` needs a matching update.
+For issue-tree diagnosis, read the current `doc/execution-semantics.md`, inspect the
+relevant subtree and recent repairs, and classify non-progressing nodes:
 
-## When to use
+- **Genuine wait:** a named human or external dependency must act.
+- **Agent-actionable but unrouted:** authorized useful work lacks a live path.
+- **Already covered:** existing work owns the next action; do not duplicate it.
 
-Trigger on an assignment whose title or body matches any of:
+Preserve these three invariants together:
 
-- "why did this work stop", "why did this stall", "why did this just stop"
-- "infinite loop", "looping", "spinning", "going too deep", "recovery went too deep"
-- "liveness — what happened here", "this tree stopped working", "stuck"
-- "approach it from a product perspective", "general product principle / rule"
-- An attached link to a specific stalled / looping / over-recovered issue tree
+1. Productive work has a real next action and owner.
+2. Only genuine blockers stop work, including approval, credentials, and budget.
+3. Recovery is bounded and cannot recursively recover itself.
 
-Also use when the user asks for forensics, root cause, or a write-up *before* any product change.
-
-## When NOT to use
-
-- The assignment asks you to ship a code change directly. Use normal engineering flow.
-- The assignment is a normal bug report against a specific feature. Use normal investigation.
-- You are the original implementer being asked to fix your own bug. Use normal debugging.
-
-## Three invariants you must preserve
-
-Every diagnosis and every proposed rule must hold these three invariants together. The user has restated them on at least four issues; treat them as load-bearing:
-
-1. **Productive work continues.** Agents that have a clear next action must keep working without needing the user to wake them. ([PAP-2674](/PAP/issues/PAP-2674), [PAP-2708](/PAP/issues/PAP-2708))
-2. **Only real blockers stop work.** Stops happen when something genuinely cannot proceed (missing approval, missing dependency, human owner). Pseudo-stops (in_review with no action path, cancelled leaves, malformed metadata) must be detected and routed, not left silent. ([PAP-2335](/PAP/issues/PAP-2335), [PAP-2674](/PAP/issues/PAP-2674))
-3. **No infinite loops.** Stranded-work recovery and continuation loops must be bounded and distinguishable from genuinely productive continuation. ([PAP-2602](/PAP/issues/PAP-2602), [PAP-2486](/PAP/issues/PAP-2486))
-
-If a proposed rule violates any of the three, drop it or rework it. State explicitly in the plan how each invariant is held.
-
-## Procedure
-
-### 0. Read the current execution contract
-
-Before walking the tree, read `doc/execution-semantics.md` and keep its terms intact:
-
-- live path / waiting path / recovery path
-- post-run disposition: terminal, explicitly live, explicitly waiting, invalid
-- bounded `run_liveness_continuation`
-- productivity review vs liveness recovery
-- active subtree pause holds
-- silent active-run watchdog
-
-Do not invent a new rule until you can state how it differs from the current execution semantics document.
-
-### 1. Forensics on the named tree — before anything else
-
-Do this in the same heartbeat. Do not propose a rule until you have a concrete stop point.
-
-- Open the linked issue (and its blocker chain, parents, recovery siblings, recent runs).
-- Walk the tree node-by-node and find the exact issue + state combination that stops the world. Common shapes seen in the company so far:
-  - `in_review` with no typed execution participant, no active run, no pending interaction, no recovery issue ([PAP-2335](/PAP/issues/PAP-2335), [PAP-2674](/PAP/issues/PAP-2674)).
-  - `in_progress` after a successful run with no future action path queued ([PAP-2674](/PAP/issues/PAP-2674)).
-  - Blocker chain whose leaf is `cancelled` / malformed / cross-company-inaccessible ([PAP-2602](/PAP/issues/PAP-2602)).
-  - `issue.continuation_recovery` waking the same issue >N times after successful runs ([PAP-2602](/PAP/issues/PAP-2602)).
-  - Stranded-work recovery treating its own recovery issues as more recoverable source work ([PAP-2486](/PAP/issues/PAP-2486)).
-- Quote the evidence: run ids, comment timestamps, status transitions. "Inferred" is acceptable only when an API boundary blocks direct evidence — say so explicitly and mark the claim provisional ([PAP-2631](/PAP/issues/PAP-2631)).
-
-Respect the API boundary. If the linked issue is in another company and your agent token returns 403, do not bypass scoping. Either request a board-approved diagnostic path or proceed from inferred PAP-side evidence and label it.
-
-### 2. Survey recent related work
-
-Before proposing a new product rule, read what already shipped this week in the same area. The user has explicitly called this out: ([PAP-2602](/PAP/issues/PAP-2602)) "review our recent work on liveness that we shipped in the last couple of days." A new rule that contradicts code merged 48 hours ago is rework, not improvement.
-
-Quick survey:
-- Recent merged PRs in the affected area.
-- Recent done issues whose title mentions liveness, recovery, productivity, continuation, or the affected subsystem.
-- Any active plan documents on parent issues. The fix may belong as a revision to an existing plan, not as a new top-level proposal.
-
-State in the forensics: "I reviewed X, Y, Z. The new gap is …"
-
-### 3. Classify each non-progressing issue in the tree
-
-For every issue in the affected tree that is not `done` / `cancelled` / actively running, decide:
-
-- **Truly needs human or board intervention** — name the owner and the action.
-- **Agent-actionable but not currently routed** — name the rule that would have routed it, and the agent that should have been waked.
-- **Already covered** — point at the active run, queued wake, recovery issue, or pending interaction.
-
-This is the table the user has asked for repeatedly ([PAP-2335](/PAP/issues/PAP-2335)). Without it the plan is abstract.
-
-### 4. Frame as a general product rule
-
-The user does not want a one-off patch on the named tree. They want the rule. Two checks:
-
-- The rule is **stated as a contract**, not as an if/else patch. Example contract: "every agent-owned non-terminal issue must finish each heartbeat with a terminal state, an explicit waiting path, or an explicit live path" ([PAP-2674](/PAP/issues/PAP-2674)).
-- The rule is reconciled against `doc/execution-semantics.md`. Prefer citing and applying the existing contract; propose a document change only when the current doc is incomplete or contradicted by accepted/implemented behavior.
-- The rule **explicitly preserves the three invariants** above. Show the work.
-
-If the rule would have blocked a recent productive run from succeeding, drop or narrow it.
-
-### 5. Plan, do not code
-
-Write the plan into the issue's `plan` document. Cover:
-
-- Forensics summary (root cause + evidence).
-- The general product rule, stated as a contract.
-- Whether the existing `doc/execution-semantics.md` contract already covers the case, or what exact documentation update is needed.
-- Phased subtasks: typically `Phase 0` resolves the named live tree (carefully, not destructively), `Phase 1` codifies the contract in docs, then implementation phases for detection, recovery, UI surfacing, security review, QA, and CTO review.
-- Explicit assignees per phase; favor team specialty (CodexCoder for server, ClaudeCoder for FE, UXDesigner for visible state, SecurityEngineer for ownership/permissions, QA for validation).
-- Blocking dependencies wired with `blockedByIssueIds`, parallel branches identified.
-
-Do not create the child issues yet. Do not push code.
-
-### 6. Request approval, then decompose
-
-- Open a `request_confirmation` interaction targeting the latest plan revision. Idempotency key `confirmation:{issueId}:plan:{revisionId}`.
-- Wait for board/CTO acceptance. If the user posts a new comment that supersedes the plan, the prior confirmation is invalidated — open a fresh confirmation tied to the new revision ([PAP-2602](/PAP/issues/PAP-2602) cycled three revisions; that is fine).
-- Only after acceptance: create the phased child issues with the right assignees and dependencies, then block this parent on the final QA / CTO review issue so the parent only wakes when the chain finishes.
-
-### 7. Phase 0 hygiene on the named tree
-
-Phase 0 cleans up the live tree without papering over evidence:
-
-- Move stalled `in_review` leaves with no participant to `todo` with a precise next action and named owner ([PAP-2335](/PAP/issues/PAP-2335)).
-- Detach cancelled/dead blockers from chains they were holding hostage; do not silently mark issues `done` to clear backlog.
-- Leave a comment on the original named issue summarizing what changed and why; never hide the recovery chain history.
-
-### 8. Final close-out
-
-When the phase chain is complete, post a board-level summary comment on the parent issue: what changed, what the new contract is, what the rollout step is (e.g. "restart the control-plane to pick up the new response shape"), and the live state of the originally-named tree. Then close the parent.
-
-## Pitfalls
-
-- **Coding before approval.** The user has said "make a plan first" on every recent diagnostic issue. Producing code in the forensic phase wastes the round-trip.
-- **Restating one invariant at the cost of another.** Bound continuation too tightly and productive work stalls; loosen recovery and infinite loops return. Always check all three.
-- **Skipping the recent-work survey.** Proposing a contract that contradicts what shipped 24 hours ago is the easiest way to get the plan rejected.
-- **Letting "in_review" mean done.** A leaf assigned to another agent with no participant or active run is not progress; treat it as a stop.
-- **Bypassing company scoping.** Cross-company forensics needs a board-approved diagnostic path, not a database read.
-- **Recursive recovery.** Stranded-work recovery that recovers its own recovery issues is the canonical infinite loop ([PAP-2486](/PAP/issues/PAP-2486)). Detect it and refuse to deepen.
-- **Hiding the chain.** Don't silently delete or hide the symptomatic recovery issues — the operator needs the audit trail.
-
-## Verification checklist (before posting the plan)
-
-- [ ] The exact stop point in the named tree is identified with run ids / comment ids.
-- [ ] Recent shipped work in the same area was surveyed and is referenced.
-- [ ] Every non-progressing issue is classified human-needed / agent-actionable / already-covered.
-- [ ] The proposed rule is stated as a contract, not a patch.
-- [ ] All three invariants are explicitly preserved.
-- [ ] No code change has landed in this heartbeat.
-- [ ] A `request_confirmation` against the latest plan revision is open.
-- [ ] Phase 0 of the plan addresses the live named tree without destroying evidence.
-- [ ] Implementation phases name specialty-appropriate assignees and `blockedByIssueIds` dependencies.
+Frame any repair as a general behavioral contract, check it against a productive
+counterexample, and follow the new skill through the authorized finish line.
+Historical task links and private incident access are not required.

.agents/skills/fix-failed-paperclip-run/SKILL.md

@@ -0,0 +1,204 @@
+---
+name: fix-failed-paperclip-run
+description: Diagnose and fix failed, stuck, slow, or incorrectly blocked Paperclip tasks and runs. Use with a task/run URL or failure report to trace the root cause, prove a regression red then green, check second-order effects, verify the real workflow, and deliver a PR with passing checks.
+---
+
+# Fix Failed Paperclip Run
+
+Own the failure from evidence through a tested repair and a green PR. A diagnosis,
+a proposed plan, a successful provider turn, and a passing unit test are intermediate
+results. The finish line is a verified user workflow and a PR whose current commit
+passes the required checks and review.
+
+Honor an explicit diagnosis-only request or a calling workflow's approval boundary.
+Otherwise proceed through implementation without a separate plan-approval pause.
+Carry forward existing authorization for merge, rollout, and recovery; do not ask
+again. When the assignment ends at a PR, distinguish local verification from
+deployment and recovery of the original task. This skill does not itself authorize
+merging or deploying.
+
+## 1. Establish the incident and the finish line
+
+Start from the supplied task/run URL or failure report. Resolve these facts from
+available tools and records before asking the user for missing information:
+
+- Instance, company, task, affected run(s), time window, and intended useful result.
+- Current state versus state at failure; parent/child tasks and pending interactions
+  only where they control progress. Multiple failed runs can have different causes.
+- The code actually served, runner/provider pack versions, operating system and
+  architecture, adapter, model/effort, execution workspace, trust mode, and credential
+  source. Trace effective values back through task overrides, inherited settings,
+  defaults, and saved session state. Redact credential values.
+- Which endpoint is authorized: verified fix and green PR, or also rollout and
+  recovery. Preserve the requested work and success criteria throughout.
+
+Use authorized diagnostic access. If company scoping blocks a read, use an
+already-authorized diagnostic path or label the missing evidence; do not bypass
+the boundary.
+
+Read repository instructions and the relevant current execution contract. In this
+repository, use `doc/execution-semantics.md` for ownership, continuation, waiting,
+finalization, and recovery. Inspect recent changes and open/merged repairs in the
+affected area. A correct fix that has not reached the target is a deployment gap;
+do not create a second patch without evidence that the first is insufficient.
+Choose code repair, configuration repair, or recovery from that evidence. If an
+existing fix is sufficient, verify it and continue to the authorized endpoint
+using its PR; do not manufacture code changes for an already-fixed defect.
+
+Keep a compact evidence record in the working notes or driving issue: environment,
+revision, causal timeline, reproduction, safety assumptions, and acceptance result.
+Preserve useful logs and saved work before retries or cleanup. Do not put secrets
+or private incident links in public code, fixtures, skills, or PR descriptions.
+
+## 2. Find the first causal failure
+
+Reconstruct the run from dispatch through preparation, provider execution, tool
+calls, result persistence, finalization, and task disposition. For chat/email work,
+follow outbound delivery and the user-visible result too. Align run events, logs,
+controller/provider records, task activity, and traces by time and identity.
+
+Separate these questions:
+
+1. What first prevented useful work or its delivery?
+2. Why did the responsible component make that decision or emit that value?
+3. Why did recovery or continuation fail to complete the already-authorized work?
+4. Which later errors are consequences, misleading summaries, or independent bugs?
+
+Trace the causal value or transition to its producer and shared contract. Verify
+the hypothesis against a counterexample or a reproduction. Check logs before and
+after the apparent error: a recovered authorization warning may not be fatal, and
+“final response,” HTTP 200, or “run succeeded” may not mean the task is complete.
+Missing error detail is an evidence gap, not proof of a provider outage. When
+evidence cannot distinguish hypotheses, name the uncertainty and add the smallest
+bounded, redacted observation or fault injection that will resolve it.
+
+For a stalled tree, classify each relevant non-progressing node as genuinely
+waiting on a named human/external dependency, agent-actionable but unrouted, or
+already covered by existing work. Preserve all three invariants:
+
+- Productive work has a real next action and owner.
+- Genuine approval, credential, budget, and dependency blockers remain effective.
+- Recovery and continuation are bounded and cannot recursively recover themselves.
+
+Consult [failure patterns](references/failure-patterns.md) for matching mechanisms
+and test oracles. Treat them as hypotheses to check, not a catalog of explanations.
+
+## 3. Prove the bug before changing production code
+
+Write the smallest behavioral regression at the boundary that owns the defect.
+Run it against the unfixed code and capture the expected failure. A setup error,
+missing dependency, or assertion about a mock's own return value is not the red.
+
+- Use the real policy/resolver/serializer and realistic persisted input. Mock only
+  boundaries outside the behavior under test. If a query duplicates rows, exercise
+  that query with the relational shape that produces duplicates.
+- For recovery or races, control the interruption point and assert the resulting
+  ownership, command receipts, side effects, and continuation. Restarting between
+  completed turns does not reproduce a restart during an in-flight side effect.
+- For packaging, process startup, platform, or cross-language defects, reproduce
+  using the actual affected artifact or protocol. A host binary or mocked provider
+  cannot prove that a sandbox image contains an executable compatible binary.
+- For latency, measure the user's full interval and the suspected stage under the
+  relevant cold/warm conditions. Use a stable controlled regression where possible;
+  keep noisy wall-clock comparisons as measured acceptance evidence.
+
+Keep this failing regression as the starting point for the repair. If the defect
+requires a real integration environment, capture a
+repeatable failing integration probe before the fix and repeat it afterward;
+add a durable regression at the closest reliable boundary. If no meaningful red
+can be observed, report that gap and keep working on reproduction. Do not claim
+red-green proof from a test that was first run after the fix.
+
+## 4. Repair the cause and prove the safety assumptions
+
+Fix the responsible producer or shared transition at the narrowest correct scope.
+Check sibling callers/adapters that share that contract; generalize only as far as
+the evidence requires. Avoid historical string allowlists, swallowed errors,
+unbounded retries, or relaxing an integrity check to accommodate malformed input.
+Preserve the requested model, trust, isolation, and performance behavior unless a
+change or explicit fallback policy is part of the assignment.
+
+Before implementation, state the behavioral rule and the strongest plausible
+second-order failure. Trace callers, downstream consumers, persisted state, and
+runtime variants affected by the changed contract. Turn the important safety
+assumptions into observable regression cases, not just assurances in the PR:
+
+| Changed boundary | Safety evidence to consider |
+| --- | --- |
+| Ownership, cancellation, retry, or continuation | No second executor or repeated external side effect; legitimate queued work still runs; cancellation remains effective. An unchanged assignee field alone is insufficient. |
+| Recovery or finalization | Saved files, results, and sessions survive; known-complete work is not rerun; interrupted work continues once; indeterminate outcomes reconcile before retry. |
+| Waiting or task disposition | Real approval/budget/credential gates still stop work; actionable work has a durable wake; recovery is bounded; no recursive recovery tree. |
+| Input, tool, or credential plumbing | Company and trust boundaries hold; intentional configuration reaches execution without ambient secret forwarding; distinct tools still dispatch correctly; sensitive logging remains redacted. |
+| Packaging, compatibility, or session identity | Relevant local/sandbox, platform, fresh/resumed, and old/new artifact combinations work; obsolete saved values are handled by a contract rather than a list of incident exceptions. |
+| Performance or lifecycle | Warm reuse remains warm where intended; queueing and finalization do not regress; configured deadlines retain their meaning. |
+
+Select cases that exercise the actual blast radius. Do not create an unrelated
+platform matrix. Test the sibling success path and the negative case that the old
+guard protected. Keep reproduction and safety tests through later revisions.
+
+With the red established and the safety assumptions identified, make the repair
+and run the same regression green. Run the relevant safety cases and existing
+tests. Refactor only while they stay green. Investigate unexpected failures
+instead of weakening an assertion or deleting the regression to finish.
+
+Routine infrastructure facts belong to the harness when it can determine them
+under existing authority. Reconcile process ownership, saved results, or eligible
+continuations in code instead of asking a user to certify those facts in a form.
+Keep UI choices for genuine user intent, authorization, or unavailable information.
+
+## 5. Verify the real workflow at the fixed revision
+
+Apply the repository's `verify-paperclip` skill and affected map recipes when
+available. Its essential requirement here is a real run through the affected user
+entry point, using the fixed source or package and the relevant runtime configuration.
+Run the narrow regression first so each hypothesis gets fast feedback; use a local
+server with the real remote provider when that reproduces the failure. Do not wait
+for a full deployment for every experiment when a faithful shorter loop exists.
+
+Exercise the original failure mechanism with a representative task: actual adapter,
+provider, tools, trust boundary, persisted state, artifact, and interruption point
+where each matters. A mock, Storybook, provider-free benchmark, or generic “hello”
+smoke is supporting evidence only when the incident depends on a different path.
+
+Verify the useful output, persistence after reload/reconnect, finalization, and
+correct task state with no stranded lock or pending operation. For resume and
+conversation defects, run a second turn and check retained context/files. For
+delivery defects, check the destination and user-visible outcome, not just receipt
+of an HTTP request. For slowness, compare the end-to-end interval with the baseline.
+
+Record the tested revision and runtime versions. Re-run affected acceptance after
+changes that invalidate that evidence. Report any missing credentials, access, or
+environment precisely; an unavailable live check remains unverified. Continue
+independent repair and PR work instead of substituting an unsupported “works” claim.
+
+## 6. Deliver the PR and close the incident to the authorized endpoint
+
+Use `prepare-paperclip-pr` and `prcheckloop` when available. Follow repository
+instructions for checks, branch isolation, PR template, and review. Open the PR,
+address review and CI failures, and inspect required checks and review on the
+current head SHA. Pending, skipped, and passing are distinct. Retry a failed check
+only with a reason; retain its evidence and fix repeatable failures. Resolve
+companion repository/artifact changes needed by the fix and state their ordering.
+
+The PR must explain the causal defect, behavioral change, failing-before/passing-
+after evidence, safety regressions, actual acceptance environment, and limitations.
+Do not declare it ready while required checks or review remain unresolved. If a
+required service or permission prevents completion, name the exact remaining gate.
+
+When rollout/recovery is already authorized, continue: confirm the served revision
+and required artifact versions, re-read the original task, and recover it through
+the normal product path. Reconcile existing work before starting another executor.
+Observe the original useful result or a genuine, named remaining blocker. Do not
+cancel, replace, or shrink the user's original task merely to make a test pass.
+
+Finish with a concise report:
+
+- Root cause and what changed, with unresolved hypotheses labeled.
+- Red/green regression and second-order behavior checked.
+- Real workflow exercised, revision/environment, and observed result.
+- PR URL, current check/review status, and companion changes if any.
+- Whether the original task is recovered, still awaiting rollout, or blocked on a
+  specific remaining dependency. Separate code readiness from incident recovery.
+
+Do not stop at a plan when a fix is authorized. Do not call the incident resolved
+because its error message improved or a successful run badge appeared.

.agents/skills/fix-failed-paperclip-run/references/failure-patterns.md

@@ -0,0 +1,138 @@
+# Failure patterns and useful regression oracles
+
+These are self-contained examples distilled from task investigations. They are
+mechanisms to test, not assumptions about the current incident. Read only the
+sections that match the evidence. Resolve current contracts and implementation
+from the repository; no historical task or private conversation is required.
+
+## An interrupted turn becomes a permanent task failure
+
+**Mechanism:** A server restart restores session history, but the old active turn
+is now failed or indeterminate. Recovery treats that provider status as a final
+task outcome, or validates saved input against today's defaults. Useful work is
+left behind a manual gate even though continuation was authorized.
+
+**Prove:** Interrupt at the observed point, including an in-flight tool call when
+relevant. Exercise both a surviving process and a lost process. Reconcile command
+receipts and preserve the session's input identity. Assert one owner, no duplicate
+side effect, retained files/context, bounded continuation, and correct disposition.
+Repeat an interruption to prove the recovery budget and genuine stops still work.
+
+**Trap:** A reconnect between completed turns proves less. A list of old prompt
+strings does not define compatibility. Disabling warm reuse may hide the failure
+while imposing a new startup cost on every turn.
+
+## A successful result gets stranded during finalization
+
+**Mechanism:** The provider completed useful work, but result persistence, artifact
+copyback, checkpoint reconciliation, outbound delivery, or disposition failed.
+The UI then shows either success without the deliverable or failure that invites
+rerunning an already-complete external action.
+
+**Prove:** Retain the original result and side-effect receipt, fail the relevant
+finalization stage, then recover. Assert the result is delivered/persisted once,
+no provider work is repeated unnecessarily, the lock is released correctly, and
+reload shows the final output. A known result can be recovered even if the original
+task has since moved on; re-read current state before applying it.
+
+**Trap:** A successful provider turn, request acknowledgement, or status badge is
+not proof that a user received the report, document, chat reply, or voice answer.
+
+## A routing repair permits a second executor
+
+**Mechanism:** A mention or chat follow-up targets someone other than the assignee.
+A shortcut lets that agent execute without changing the assignee field. A unit
+test proves the field stayed constant while both agents can now perform the work.
+
+**Prove:** Attempt the conflicting wake while the real owner is active. Assert
+execution and external side effects cannot start twice, and the legitimate owner's
+queued work still proceeds. Test cancellation and follow-up routing through the
+real shared policy, including identity, company, and trust boundaries as affected.
+
+**Trap:** Database ownership alone does not prove execution ownership. Broadening
+credentials or bypassing checkout is not a routing fix.
+
+## Runtime inputs are malformed before the model starts
+
+**Mechanism:** A relational join duplicates tools; names exceed a provider limit
+after namespacing; dispatch binds a tool to the wrong connection; an explicit
+environment variable is lost at an allowlist; execution and logging hash different
+versions of a redacted payload. Later errors obscure the first invalid input.
+
+**Prove:** Exercise the real query, final name builder, environment filters, or
+cross-language serializer with the failing input shape. Assert both the rejection
+case and the intended valid case. Preserve distinct connection bindings, full
+authorized execution input, redacted logs, and integrity checks. Where config is
+cached, test added, changed, and removed values in fresh and resumed sessions.
+
+**Trap:** Deduplicating every same-named tool, disabling digest validation, or
+forwarding all ambient variables can turn a startup fix into a correctness or
+credential leak. Retrying the same invalid tool definition is not recovery.
+
+## The deployed artifact differs from the tested artifact
+
+**Mechanism:** A dependency exists in source but not the image actually served; a
+test checks an obsolete image target; a native package skipped installation; a host
+binary works while the sandbox platform cannot run it; a fix is merged but absent
+from the tenant's served revision. Separately built artifacts may differ despite
+sharing a source revision.
+
+**Prove:** Inspect the target revision and artifact identity, actual image target,
+installed executable, OS/architecture, and launch protocol. Start the real packaged
+provider in that environment. Exercise the relevant fallback/install path and a
+second turn if session reuse is involved. Keep model/account compatibility separate
+from process launch failures.
+
+**Trap:** Adding a dependency to a manifest or obtaining a 200 from a provider
+probe does not prove the launched runner can execute it. A new patch is unnecessary
+when the existing repair only needs to reach the target.
+
+## Recovery confuses useful waiting with a blocker
+
+**Mechanism:** Work awaiting CI, an eligible monitor, or a durable continuation is
+marked blocked or abandoned. The opposite error removes a real approval, budget,
+credential, or uncertainty gate to force progress. Recovery issues can recursively
+create more recovery issues.
+
+**Prove:** Cover actionable work with a durable next wake, a genuine human wait,
+an exhausted budget, already-covered work, and repeated recovery. Assert the right
+next actor and bounded attempts. Let the harness determine facts it can observe;
+reserve human input for decisions or facts it cannot establish.
+
+**Trap:** Changing a status label without scheduling useful work is not liveness.
+An operator form asking whether an internal controller is dead is not a substitute
+for ownership reconciliation.
+
+## “It works” hides a slow or incomplete user journey
+
+**Mechanism:** The model is quick, but preparation, shared repository locks,
+database queries, remote startup, or finalization dominates latency. A local mock
+benchmark removes the stage responsible for most of the delay. Separately, an
+integration test may never reach real authentication or the destination output.
+
+**Prove:** Measure submit-to-usable-output and the critical stage with the affected
+configuration. Compare cold and warm cases where relevant. Use the real remote
+provider from a local harness when it shortens iteration without removing the
+failure. Check actual output and persisted state, including the next turn. If
+optimizing a query, verify equivalent results and authorization as well as speed.
+
+**Trap:** Extra trace spans only improve observability until a measured repair
+improves the path. A fresh greeting or an OAuth setup screen does not establish that
+the original task completed. Do not quietly change the model or effort to get a
+faster number.
+
+## Error evidence is incomplete or misleading
+
+**Mechanism:** The UI reports a generic native failure, an earlier recovered
+warning looks fatal, or an exception disappears between a tenant and an error
+collector. An enabled reporting flag and successful requests do not prove a
+particular event arrived.
+
+**Prove:** Preserve the original structured cause across the boundary with bounded
+redaction. Distinguish event creation, transport acceptance, and recorded receipt.
+Verify the concrete error is shown to the user when actionable. Test the failure
+path itself, not only the happy-path logger configuration.
+
+**Trap:** Better future evidence cannot establish an uncaptured historical cause.
+Report what remains unknown; do not turn an instrumentation PR into a claim that
+the original task failure was fixed.