Files
PaperClipAI/server/src/services/native-runtime/native-agent-blocker.integration.test.ts
T
DottaandClaude Opus 5.5 1c556d5340 fix(native): never rest a task blocked on the agent's own blocker (#22140)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run ends with one semantic report. A `paperclip_block`
report names a blocker, its owner and the unblock action
> - The status arbiter binds a task-wide blocker. When the owner is not
a person, it binds the blocker to the same agent and wakes that agent
> - The dispatcher refuses those self-wakes
(`chat_control_recovery_proof_unresolved`), because only continuation
and retry effects are proof for a native wake. The task then rests
blocked with no owner, no wake and no Retry
> - #21959 continues a system blocker in a fresh session, but only for
the reason codes `environment_unavailable` and `tool_unavailable`. A
restart-chaos rerun showed the same dead end with the reason code
`tool_output_failure`
> - This pull request makes sure that a blocker that the agent reports
for itself always has a next owner: the automatic environment lane, one
bounded self-continuation, or a person with a Retry that works
> - The benefit is that no task rests blocked with nobody to act, and no
path wakes the agent in a loop

## Linked Issues or Issue Description

Refs #21959

**What happened?**

A native OpenCode run (DeepSeek) said that every tool call returned the
text "cryptic". It reported `paperclip_block` with owner `{kind:
system}`, scope `task_wide`, reason code `tool_output_failure`, and an
unblock action about the tool-output failure. The run succeeded with
disposition `blocked`. The arbiter bound the blocker to the same agent
(`task_wide_blocker_bound`). The `notify-owner` wake coalesced, and the
`blocker-owner` wake was skipped as
`chat_control_recovery_proof_unresolved`. The task rested blocked
(`needs_attention`, 0 blockers), with no Retry and no wake. An earlier
rerun showed the same shape with reason code `environment_unavailable`,
which #21959 fixed for those two codes only.

**Expected behavior**

A blocker that the agent reports for its own tools or environment is a
harness problem. Paperclip continues it automatically, with a bound. A
blocker that the agent says it owns gets one fresh session. A blocker
that needs a person goes to a person, with a clear prompt and a Retry
that starts the task again. Nothing is left with nobody.

**Steps to reproduce**

1. Run a native task. Make the agent call `paperclip_block` with owner
kind `system`, scope `task_wide` and a reason code other than
`environment_unavailable` or `tool_unavailable` (for example
`tool_output_failure`).
2. See the decision `task_wide_blocker_bound` with the agent as the
unblock owner.
3. See both owner wakes refused, and the task blocked with no Retry.

**Paperclip version or commit**

`master` at 9e1a85fb46.

## What Changed

- `native-environment-blocker.ts`: a system-owned blocker now uses the
bounded fresh-session environment lane for any reason code, except the
person-only codes `permission_denied`, `credential_missing`,
`policy_restricted` and `budget_exhausted`. An agent-owned blocker with
`environment_unavailable` or `tool_unavailable` also uses it. The lane
is unchanged: at most 2 retries on the shared failure counter, then a
board blocker.
- New `native-agent-blocker.ts` and a new arbiter branch for a task-wide
blocker that is not person-owned, from the current owner's own succeeded
run (not a chat, not a review):
- Owner kind `agent`: one fresh-session continuation
(`native_agent_blocker_continuation`). It uses the shared failure
counter with a maximum of 1, so it never runs after another automatic
retry in the same chain. A paused task does not get it.
- All other cases, and a repeated blocker: the blocker goes to the board
with the agent's unblock action (`native_agent_blocker_needs_person`).
There is no self-wake.
- `native-provider-capacity-retry.ts`, `status-decision-committer.ts`,
`native-run-finalizer.ts`: materialize the continuation through the
existing fresh-session retry lane (`forceFreshSession`, `retryOfRunId`,
30 s delay, refuses a stopped run). Both new reason codes need the
provider-failure owner recheck under the issue lock. The finalizer reads
the pause hold for every blocked result.
- `heartbeat.ts`, `run-dispatch/adapters/postgres.ts`: the continuation
gets the same dispatch ownership checks as the environment retry.
- `task-interruption-presentation.ts`, shared
`TaskInterruptionPresentation.userAction`: a task that is blocked by a
`native_agent_blocker_needs_person` decision on its latest run shows
`userAction: "unblock"` with `canRetry: true`. `ExecutionBlockerNotice`
shows "The agent stopped and needs your help to continue. Read its last
comment, do what it asks, then Retry." with a Retry button. The inbox
label is "The agent needs your help". The assigned-work invariant reads
it as `assignee_declared_block`.
- `blocked-question-answer.ts`: an answer to the agent's open question
still wakes it after the new hand-off.
- Runner `completion-result.ts`: the "Paperclip continues this blocker
automatically" message now matches the server's environment lane.
- I did not change `readChatControlNativeParent`. The fix removes the
self-bound blocker wakes instead of accepting them as proof, so a
self-wake loop cannot form.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-agent-blocker.integration.test.ts`
(new, embedded Postgres, real executor, finalizer and committer). Red
before the fix (3 of 3 fail with `task_wide_blocker_bound`), green
after:
- The t205 shape (system owner, `tool_output_failure`) gets one
fresh-session retry, no owner wakes, and the retry run that reports done
moves the task to `done`.
- An agent-owned blocker continues once. The same blocker in the
continuation goes to the board, with no further retry and no owner
wakes.
- A system blocker with `credential_missing` goes to the board with no
automatic retry.
- An external-owned blocker goes to the board. The task shows
`userAction: "unblock"` and `canRetry: true`. `requestTaskRetry` accepts
the Retry and `resumeTaskRetry` starts exactly one new run of the task.
- `npx vitest run src/services/native-runtime/status-arbiter.test.ts`:
new case for the once-only continuation, the hand-off, the pause hold,
the chat exclusion and the non-owner exclusion.
- `cd ui && npx vitest run
src/components/ExecutionBlockerNotice.test.tsx`: new case for the help
prompt and its Retry.
- `cd packages/paperclip-runner && npx vitest run
src/contracts/completion-result.test.ts`: new cases for the conflict
message.
- Related suites pass (11 files, 523 tests): arbiter corpus,
task-interruption presentation and query, assigned-work invariant,
blocked-question answer, liveness escalation, run-dispatch adapter,
failed-turn continuation, tool-receipt limit and provider capacity.
- `tsc --noEmit` passes for `server`, `packages/shared`,
`packages/paperclip-runner` and `ui` (only the existing generated
Storybook catalog errors). `scripts/check-token-gates.mjs` is clean.
- `pnpm verify` exits 2: the source SDK pin refresh is a global CI
input, so required CI supplies the broad coverage.

## Risks

- More automatic runs: an agent-owned blocker can add 1 run per chain,
and a system blocker with a new reason code can add up to 2 (the
existing lane). Both share the failure-retry counter, so a chain never
gets more than the existing maximum. The company-wide automatic-run
brake admits these runs like any other recovery run. Judge the change by
company-wide automatic run starts per hour and their success rate.
- A blocker that only a person can resolve, but that the agent labels as
its own, now takes one extra run (about 30 s later) before it reaches a
person. This is intended.
- Board-owned blockers now show an attention card and a Retry, where
before they named the agent and showed nothing. A Retry starts a new run
of the task; the person decides when.
- Previous release: no persisted shape changes. The new reason codes are
new values in existing text columns. Decisions saved by the previous
release keep their saved effects.
- Chats keep their current behavior, because the person in the chat
answers directly.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (code search, editing, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 02:43:28 -06:00

170 lines
13 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { and, eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, agentWakeupRequests, companies, companyMemberships, completionContracts, createDb, heartbeatRuns, issues,
nativeRunFinalizations, statusDecisions } from "@paperclipai/db";
import type { ControlPlanePort, NativeExecutionInputV1, PrpStructuredRunResult, PrpTerminalState } from "@paperclipai/paperclip-runner";
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { CONTROL_PLANE_CONFORMANCE_RESULT } from "../../vendor/paperclip-runner/testing.js";
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()), executeNativeSession: provider.execute,
}));
import { executePaperclipNativeSession } from "./native-session-executor.js";
import { buildNativeCompletionContract } from "./completion-contracts.js";
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
import { finalizeNativeRun } from "./native-run-finalizer.js";
import { NATIVE_ENVIRONMENT_BLOCKER_CODE, NATIVE_ENVIRONMENT_BLOCKER_HINT, NATIVE_ENVIRONMENT_BLOCKER_MESSAGE } from "./native-environment-blocker.js";
import { NATIVE_AGENT_BLOCKER_CONTINUATION_CODE, NATIVE_AGENT_BLOCKER_HANDOFF_CODE, NATIVE_AGENT_BLOCKER_HINT } from "./native-agent-blocker.js";
import { taskInterruptionPresentation } from "../task-interruption-presentation.js";
import { requestTaskRetry } from "../task-retry.js";
import { heartbeatService } from "../heartbeat.js";
type Blocker = NonNullable<PrpStructuredRunResult["blocker"]>;
const toolOutputBlocker: Blocker = {
owner: { kind: "system", name: "execution environment" }, scope: "task_wide", reasonCode: "tool_output_failure",
unblockAction: "Resolve the persistent tool-output failure ('cryptic' returned for every tool call) and/or provide the current contents of the ledger file.",
};
const selfBlocker: Blocker = {
owner: { kind: "agent", name: "ledger worker" }, scope: "task_wide", reasonCode: "work_incomplete",
unblockAction: "Re-read the ledger file and save the ledger document.",
};
describe("native run that reports a task-wide blocker it owns", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: ReturnType<typeof createDb>;
const companyId = randomUUID(), agentId = randomUUID(), boardUserId = "ledger-board-user";
beforeAll(async () => {
temporary = await startEmbeddedPostgresTestDatabase("native-agent-blocker-");
db = createDb(temporary.connectionString);
await db.insert(companies).values({ id: companyId, name: "Agent blocker regression", issuePrefix: "AGB" });
await db.insert(agents).values({ id: agentId, companyId, name: "Ledger worker", status: "active", adapterType: "paperclip_runner",
adapterConfig: { provider: "opencode", model: "test-model" } });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: boardUserId, status: "active", membershipRole: "member" });
});
afterAll(async () => { await temporary?.cleanup(); });
async function createTask() {
const issueId = randomUUID();
await db.insert(issues).values({ id: issueId, companyId, title: "Ledger task", status: "in_progress", workMode: "standard",
assigneeAgentId: agentId, createdByUserId: boardUserId, responsibleUserId: boardUserId });
return issueId;
}
async function runTurn(issueId: string, report: { blocker: Blocker } | { done: true }, runId: string = randomUUID()) {
const [existing] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
if (!existing) await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: boardUserId, contextSnapshot: { issueId } });
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
const [run] = await db.update(heartbeatRuns).set({ status: "running", runtimeMode: "native", nativeIssueId: issueId, startedAt: new Date() })
.where(eq(heartbeatRuns.id, runId)).returning();
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
const contractInput = buildNativeCompletionContract(issue, { revision: contract.revision });
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" });
const execution: NativeExecutionInputV1 = {
schema: "paperclip.native-execution-input.v1", binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
provider: { kind: "codex", model: "test-model" },
task: { identifier: issueId, title: issue.title, description: null, prompt: "Continue the task using its history.", workMode: "standard" },
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", contract: contractInput },
interactionResponses: [], credentialBindings: [],
};
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
const disposition = "blocker" in report ? "blocked" as const : "done" as const;
const terminal: PrpTerminalState = { schema: "paperclip.prp.terminal.v1", turnTerminalState: "completed", runTerminalState: "succeeded", reportedWorkDisposition: disposition };
const result: PrpStructuredRunResult = "blocker" in report
? { ...CONTROL_PLANE_CONFORMANCE_RESULT, reportedWorkDisposition: "blocked",
summary: "All rounds ran, but every tool call returned 'cryptic', so the ledger could not be read.",
completionClaim: { contractRevision: contractInput.revision, objectiveSatisfied: false,
criteria: contractInput.criteria.map(criterion => ({ criterionId: criterion.id, status: "not_satisfied" as const, evidenceRefs: [] })),
remainingWork: [{ description: "Save the ledger document", blocksCompletion: true }] },
evidence: [], verification: [], attentionRequests: [], blocker: report.blocker }
: { ...CONTROL_PLANE_CONFORMANCE_RESULT, reportedWorkDisposition: "done", summary: "Saved the ledger and posted the comment.",
completionClaim: { contractRevision: contractInput.revision, objectiveSatisfied: true,
criteria: contractInput.criteria.map(criterion => ({ criterionId: criterion.id, status: "satisfied" as const, evidenceRefs: [] })),
remainingWork: [] },
evidence: [], verification: [], attentionRequests: [] };
provider.execute.mockReset().mockImplementation(async (options: { controlPlane: ControlPlanePort }) => {
await options.controlPlane.openRun({ identity: { companyId, issueId, runId, agentId, sessionId: native.normalizedSessionId }, backendKind: "mock", sourceInstanceId: native.runnerInstanceId });
await options.controlPlane.completeRun({ result, terminal, turnId: native.turnId });
return { result, terminal, turnId: native.turnId, normalizedSessionId: native.normalizedSessionId, providerSessionId: "fixture-session", driverKind: "codex_app_server", driverVersion: "fixture", nativeEventCount: 0, highestContiguousSourceSeq: 0, usage: null };
});
const unexpected = async () => { throw new Error("Unexpected provider call"); };
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: { descriptor: unexpected, openSession: unexpected } });
for (let replay = 0; replay < 2; replay++) await finalizeNativeRun({ db, runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
const retries = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.retryOfRunId, runId)));
const [settled] = await db.select().from(issues).where(eq(issues.id, issueId));
return { runId, decision: decision!, retries, issue: settled! };
}
const ownerWakes = async (issueId: string) => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId)))
.filter(wake => wake.payload?.issueId === issueId && wake.requestedByActorId === "native-status-committer");
it("continues a tool-output blocker in a fresh session, and the continuation finishes the task", async () => {
const issueId = await createTask();
const first = await runTurn(issueId, { blocker: toolOutputBlocker });
expect(first.decision.reasonCode).toBe(NATIVE_ENVIRONMENT_BLOCKER_CODE);
expect(first.issue).toMatchObject({ status: "in_progress", unblockDescriptor: null });
expect(first.retries).toHaveLength(1);
expect(first.retries[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: NATIVE_ENVIRONMENT_BLOCKER_CODE,
error: NATIVE_ENVIRONMENT_BLOCKER_MESSAGE,
contextSnapshot: { forceFreshSession: true, livenessContinuationInstruction: NATIVE_ENVIRONMENT_BLOCKER_HINT } });
expect(NATIVE_ENVIRONMENT_BLOCKER_MESSAGE).not.toContain("unavailable");
expect(NATIVE_ENVIRONMENT_BLOCKER_HINT).toContain("Check whether that blocker still applies");
expect(await ownerWakes(issueId)).toEqual([]);
const second = await runTurn(issueId, { done: true }, first.retries[0]!.id);
expect(second.issue.status).toBe("done");
}, 120_000);
it("continues a blocker the agent owns once, then hands a repeated one to a person instead of retrying again", async () => {
const issueId = await createTask();
const first = await runTurn(issueId, { blocker: selfBlocker });
expect(first.decision.reasonCode).toBe(NATIVE_AGENT_BLOCKER_CONTINUATION_CODE);
expect(first.issue).toMatchObject({ status: "in_progress", unblockDescriptor: null });
expect(first.retries).toHaveLength(1);
expect(first.retries[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: NATIVE_AGENT_BLOCKER_CONTINUATION_CODE,
scheduledRetryAttempt: 1, contextSnapshot: { forceFreshSession: true, livenessContinuationInstruction: NATIVE_AGENT_BLOCKER_HINT } });
expect(await taskInterruptionPresentation(db, first.issue)).toMatchObject({ canRetry: false });
const second = await runTurn(issueId, { blocker: selfBlocker }, first.retries[0]!.id);
expect(second.decision.reasonCode).toBe(NATIVE_AGENT_BLOCKER_HANDOFF_CODE);
expect(second.retries).toHaveLength(0);
expect(second.issue).toMatchObject({ status: "blocked", unblockDescriptor: { owner: "board", action: selfBlocker.unblockAction } });
expect(await ownerWakes(issueId)).toEqual([]);
}, 120_000);
it("hands a system blocker that needs a credential to a person without automatic retries", async () => {
const issueId = await createTask();
const blocked = await runTurn(issueId, { blocker: { ...toolOutputBlocker, reasonCode: "credential_missing",
unblockAction: "Connect the deploy account." } });
expect(blocked.decision.reasonCode).toBe(NATIVE_AGENT_BLOCKER_HANDOFF_CODE);
expect(blocked.retries).toHaveLength(0);
expect(blocked.issue).toMatchObject({ status: "blocked", unblockDescriptor: { owner: "board", action: "Connect the deploy account." } });
}, 120_000);
it("hands a blocker that only a person can resolve to the board with a Retry that starts the task again", async () => {
const issueId = await createTask();
const blocked = await runTurn(issueId, { blocker: { ...selfBlocker, owner: { kind: "external", name: "deploy key holder" },
reasonCode: "credential_missing", unblockAction: "Add the deploy key to the project." } });
expect(blocked.decision.reasonCode).toBe(NATIVE_AGENT_BLOCKER_HANDOFF_CODE);
expect(blocked.retries).toHaveLength(0);
expect(blocked.issue).toMatchObject({ status: "blocked", unblockDescriptor: { owner: "board", action: "Add the deploy key to the project." } });
expect(await ownerWakes(issueId)).toEqual([]);
expect(await taskInterruptionPresentation(db, blocked.issue)).toMatchObject({ responsibility: "user", state: "waiting_for_user",
userAction: "unblock", canRetry: true, observedRunId: blocked.runId });
const receipt = await requestTaskRetry({ db, companyId, issueId, actorId: boardUserId, actorSource: "session",
idempotencyKey: randomUUID(), observedRunId: blocked.runId });
expect(receipt).toMatchObject({ accepted: true, stale: false });
const heartbeat = heartbeatService(db);
await heartbeat.resumeTaskRetry(companyId, receipt.requestId!);
const retried = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agentId)))
.filter(run => run.contextSnapshot?.issueId === issueId && run.contextSnapshot?.taskRetryRequestId === receipt.requestId);
expect(retried).toHaveLength(1);
await heartbeat.drainActiveRunExecutions();
}, 120_000);
});