mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 02:43:25 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run ends with one semantic report. A `paperclip_block`
report names a blocker, its owner and the unblock action
> - The status arbiter binds a task-wide blocker. When the owner is not
a person, it binds the blocker to the same agent and wakes that agent
> - The dispatcher refuses those self-wakes
(`chat_control_recovery_proof_unresolved`), because only continuation
and retry effects are proof for a native wake. The task then rests
blocked with no owner, no wake and no Retry
> - #21959 continues a system blocker in a fresh session, but only for
the reason codes `environment_unavailable` and `tool_unavailable`. A
restart-chaos rerun showed the same dead end with the reason code
`tool_output_failure`
> - This pull request makes sure that a blocker that the agent reports
for itself always has a next owner: the automatic environment lane, one
bounded self-continuation, or a person with a Retry that works
> - The benefit is that no task rests blocked with nobody to act, and no
path wakes the agent in a loop
## Linked Issues or Issue Description
Refs #21959
**What happened?**
A native OpenCode run (DeepSeek) said that every tool call returned the
text "cryptic". It reported `paperclip_block` with owner `{kind:
system}`, scope `task_wide`, reason code `tool_output_failure`, and an
unblock action about the tool-output failure. The run succeeded with
disposition `blocked`. The arbiter bound the blocker to the same agent
(`task_wide_blocker_bound`). The `notify-owner` wake coalesced, and the
`blocker-owner` wake was skipped as
`chat_control_recovery_proof_unresolved`. The task rested blocked
(`needs_attention`, 0 blockers), with no Retry and no wake. An earlier
rerun showed the same shape with reason code `environment_unavailable`,
which #21959 fixed for those two codes only.
**Expected behavior**
A blocker that the agent reports for its own tools or environment is a
harness problem. Paperclip continues it automatically, with a bound. A
blocker that the agent says it owns gets one fresh session. A blocker
that needs a person goes to a person, with a clear prompt and a Retry
that starts the task again. Nothing is left with nobody.
**Steps to reproduce**
1. Run a native task. Make the agent call `paperclip_block` with owner
kind `system`, scope `task_wide` and a reason code other than
`environment_unavailable` or `tool_unavailable` (for example
`tool_output_failure`).
2. See the decision `task_wide_blocker_bound` with the agent as the
unblock owner.
3. See both owner wakes refused, and the task blocked with no Retry.
**Paperclip version or commit**
`master` at 9e1a85fb46.
## What Changed
- `native-environment-blocker.ts`: a system-owned blocker now uses the
bounded fresh-session environment lane for any reason code, except the
person-only codes `permission_denied`, `credential_missing`,
`policy_restricted` and `budget_exhausted`. An agent-owned blocker with
`environment_unavailable` or `tool_unavailable` also uses it. The lane
is unchanged: at most 2 retries on the shared failure counter, then a
board blocker.
- New `native-agent-blocker.ts` and a new arbiter branch for a task-wide
blocker that is not person-owned, from the current owner's own succeeded
run (not a chat, not a review):
- Owner kind `agent`: one fresh-session continuation
(`native_agent_blocker_continuation`). It uses the shared failure
counter with a maximum of 1, so it never runs after another automatic
retry in the same chain. A paused task does not get it.
- All other cases, and a repeated blocker: the blocker goes to the board
with the agent's unblock action (`native_agent_blocker_needs_person`).
There is no self-wake.
- `native-provider-capacity-retry.ts`, `status-decision-committer.ts`,
`native-run-finalizer.ts`: materialize the continuation through the
existing fresh-session retry lane (`forceFreshSession`, `retryOfRunId`,
30 s delay, refuses a stopped run). Both new reason codes need the
provider-failure owner recheck under the issue lock. The finalizer reads
the pause hold for every blocked result.
- `heartbeat.ts`, `run-dispatch/adapters/postgres.ts`: the continuation
gets the same dispatch ownership checks as the environment retry.
- `task-interruption-presentation.ts`, shared
`TaskInterruptionPresentation.userAction`: a task that is blocked by a
`native_agent_blocker_needs_person` decision on its latest run shows
`userAction: "unblock"` with `canRetry: true`. `ExecutionBlockerNotice`
shows "The agent stopped and needs your help to continue. Read its last
comment, do what it asks, then Retry." with a Retry button. The inbox
label is "The agent needs your help". The assigned-work invariant reads
it as `assignee_declared_block`.
- `blocked-question-answer.ts`: an answer to the agent's open question
still wakes it after the new hand-off.
- Runner `completion-result.ts`: the "Paperclip continues this blocker
automatically" message now matches the server's environment lane.
- I did not change `readChatControlNativeParent`. The fix removes the
self-bound blocker wakes instead of accepting them as proof, so a
self-wake loop cannot form.
## Verification
- `cd server && npx vitest run
src/services/native-runtime/native-agent-blocker.integration.test.ts`
(new, embedded Postgres, real executor, finalizer and committer). Red
before the fix (3 of 3 fail with `task_wide_blocker_bound`), green
after:
- The t205 shape (system owner, `tool_output_failure`) gets one
fresh-session retry, no owner wakes, and the retry run that reports done
moves the task to `done`.
- An agent-owned blocker continues once. The same blocker in the
continuation goes to the board, with no further retry and no owner
wakes.
- A system blocker with `credential_missing` goes to the board with no
automatic retry.
- An external-owned blocker goes to the board. The task shows
`userAction: "unblock"` and `canRetry: true`. `requestTaskRetry` accepts
the Retry and `resumeTaskRetry` starts exactly one new run of the task.
- `npx vitest run src/services/native-runtime/status-arbiter.test.ts`:
new case for the once-only continuation, the hand-off, the pause hold,
the chat exclusion and the non-owner exclusion.
- `cd ui && npx vitest run
src/components/ExecutionBlockerNotice.test.tsx`: new case for the help
prompt and its Retry.
- `cd packages/paperclip-runner && npx vitest run
src/contracts/completion-result.test.ts`: new cases for the conflict
message.
- Related suites pass (11 files, 523 tests): arbiter corpus,
task-interruption presentation and query, assigned-work invariant,
blocked-question answer, liveness escalation, run-dispatch adapter,
failed-turn continuation, tool-receipt limit and provider capacity.
- `tsc --noEmit` passes for `server`, `packages/shared`,
`packages/paperclip-runner` and `ui` (only the existing generated
Storybook catalog errors). `scripts/check-token-gates.mjs` is clean.
- `pnpm verify` exits 2: the source SDK pin refresh is a global CI
input, so required CI supplies the broad coverage.
## Risks
- More automatic runs: an agent-owned blocker can add 1 run per chain,
and a system blocker with a new reason code can add up to 2 (the
existing lane). Both share the failure-retry counter, so a chain never
gets more than the existing maximum. The company-wide automatic-run
brake admits these runs like any other recovery run. Judge the change by
company-wide automatic run starts per hour and their success rate.
- A blocker that only a person can resolve, but that the agent labels as
its own, now takes one extra run (about 30 s later) before it reaches a
person. This is intended.
- Board-owned blockers now show an attention card and a Retry, where
before they named the agent and showed nothing. A Retry starts a new run
of the task; the person decides when.
- Previous release: no persisted shape changes. The new reason codes are
new values in existing text columns. Decisions saved by the previous
release keep their saved effects.
- Chats keep their current behavior, because the person in the chat
answers directly.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (code search, editing, shell, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
170 lines
13 KiB
TypeScript
170 lines
13 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, eq } from "drizzle-orm";
|
|
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { agents, agentWakeupRequests, companies, companyMemberships, completionContracts, createDb, heartbeatRuns, issues,
|
|
nativeRunFinalizations, statusDecisions } from "@paperclipai/db";
|
|
import type { ControlPlanePort, NativeExecutionInputV1, PrpStructuredRunResult, PrpTerminalState } from "@paperclipai/paperclip-runner";
|
|
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
|
import { CONTROL_PLANE_CONFORMANCE_RESULT } from "../../vendor/paperclip-runner/testing.js";
|
|
|
|
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
|
|
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
|
|
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()), executeNativeSession: provider.execute,
|
|
}));
|
|
import { executePaperclipNativeSession } from "./native-session-executor.js";
|
|
import { buildNativeCompletionContract } from "./completion-contracts.js";
|
|
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
|
|
import { finalizeNativeRun } from "./native-run-finalizer.js";
|
|
import { NATIVE_ENVIRONMENT_BLOCKER_CODE, NATIVE_ENVIRONMENT_BLOCKER_HINT, NATIVE_ENVIRONMENT_BLOCKER_MESSAGE } from "./native-environment-blocker.js";
|
|
import { NATIVE_AGENT_BLOCKER_CONTINUATION_CODE, NATIVE_AGENT_BLOCKER_HANDOFF_CODE, NATIVE_AGENT_BLOCKER_HINT } from "./native-agent-blocker.js";
|
|
import { taskInterruptionPresentation } from "../task-interruption-presentation.js";
|
|
import { requestTaskRetry } from "../task-retry.js";
|
|
import { heartbeatService } from "../heartbeat.js";
|
|
|
|
type Blocker = NonNullable<PrpStructuredRunResult["blocker"]>;
|
|
|
|
const toolOutputBlocker: Blocker = {
|
|
owner: { kind: "system", name: "execution environment" }, scope: "task_wide", reasonCode: "tool_output_failure",
|
|
unblockAction: "Resolve the persistent tool-output failure ('cryptic' returned for every tool call) and/or provide the current contents of the ledger file.",
|
|
};
|
|
const selfBlocker: Blocker = {
|
|
owner: { kind: "agent", name: "ledger worker" }, scope: "task_wide", reasonCode: "work_incomplete",
|
|
unblockAction: "Re-read the ledger file and save the ledger document.",
|
|
};
|
|
|
|
describe("native run that reports a task-wide blocker it owns", () => {
|
|
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: ReturnType<typeof createDb>;
|
|
const companyId = randomUUID(), agentId = randomUUID(), boardUserId = "ledger-board-user";
|
|
beforeAll(async () => {
|
|
temporary = await startEmbeddedPostgresTestDatabase("native-agent-blocker-");
|
|
db = createDb(temporary.connectionString);
|
|
await db.insert(companies).values({ id: companyId, name: "Agent blocker regression", issuePrefix: "AGB" });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Ledger worker", status: "active", adapterType: "paperclip_runner",
|
|
adapterConfig: { provider: "opencode", model: "test-model" } });
|
|
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: boardUserId, status: "active", membershipRole: "member" });
|
|
});
|
|
afterAll(async () => { await temporary?.cleanup(); });
|
|
|
|
async function createTask() {
|
|
const issueId = randomUUID();
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Ledger task", status: "in_progress", workMode: "standard",
|
|
assigneeAgentId: agentId, createdByUserId: boardUserId, responsibleUserId: boardUserId });
|
|
return issueId;
|
|
}
|
|
|
|
async function runTurn(issueId: string, report: { blocker: Blocker } | { done: true }, runId: string = randomUUID()) {
|
|
const [existing] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
|
if (!existing) await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: boardUserId, contextSnapshot: { issueId } });
|
|
const [issue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
|
const [run] = await db.update(heartbeatRuns).set({ status: "running", runtimeMode: "native", nativeIssueId: issueId, startedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, runId)).returning();
|
|
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
|
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
|
|
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
|
|
const contractInput = buildNativeCompletionContract(issue, { revision: contract.revision });
|
|
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" });
|
|
const execution: NativeExecutionInputV1 = {
|
|
schema: "paperclip.native-execution-input.v1", binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
|
|
provider: { kind: "codex", model: "test-model" },
|
|
task: { identifier: issueId, title: issue.title, description: null, prompt: "Continue the task using its history.", workMode: "standard" },
|
|
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
|
|
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
|
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1", contract: contractInput },
|
|
interactionResponses: [], credentialBindings: [],
|
|
};
|
|
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
|
|
const disposition = "blocker" in report ? "blocked" as const : "done" as const;
|
|
const terminal: PrpTerminalState = { schema: "paperclip.prp.terminal.v1", turnTerminalState: "completed", runTerminalState: "succeeded", reportedWorkDisposition: disposition };
|
|
const result: PrpStructuredRunResult = "blocker" in report
|
|
? { ...CONTROL_PLANE_CONFORMANCE_RESULT, reportedWorkDisposition: "blocked",
|
|
summary: "All rounds ran, but every tool call returned 'cryptic', so the ledger could not be read.",
|
|
completionClaim: { contractRevision: contractInput.revision, objectiveSatisfied: false,
|
|
criteria: contractInput.criteria.map(criterion => ({ criterionId: criterion.id, status: "not_satisfied" as const, evidenceRefs: [] })),
|
|
remainingWork: [{ description: "Save the ledger document", blocksCompletion: true }] },
|
|
evidence: [], verification: [], attentionRequests: [], blocker: report.blocker }
|
|
: { ...CONTROL_PLANE_CONFORMANCE_RESULT, reportedWorkDisposition: "done", summary: "Saved the ledger and posted the comment.",
|
|
completionClaim: { contractRevision: contractInput.revision, objectiveSatisfied: true,
|
|
criteria: contractInput.criteria.map(criterion => ({ criterionId: criterion.id, status: "satisfied" as const, evidenceRefs: [] })),
|
|
remainingWork: [] },
|
|
evidence: [], verification: [], attentionRequests: [] };
|
|
provider.execute.mockReset().mockImplementation(async (options: { controlPlane: ControlPlanePort }) => {
|
|
await options.controlPlane.openRun({ identity: { companyId, issueId, runId, agentId, sessionId: native.normalizedSessionId }, backendKind: "mock", sourceInstanceId: native.runnerInstanceId });
|
|
await options.controlPlane.completeRun({ result, terminal, turnId: native.turnId });
|
|
return { result, terminal, turnId: native.turnId, normalizedSessionId: native.normalizedSessionId, providerSessionId: "fixture-session", driverKind: "codex_app_server", driverVersion: "fixture", nativeEventCount: 0, highestContiguousSourceSeq: 0, usage: null };
|
|
});
|
|
const unexpected = async () => { throw new Error("Unexpected provider call"); };
|
|
await executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId, backend: { descriptor: unexpected, openSession: unexpected } });
|
|
for (let replay = 0; replay < 2; replay++) await finalizeNativeRun({ db, runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
|
const [decision] = await db.select().from(statusDecisions).where(eq(statusDecisions.runId, runId));
|
|
const retries = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.retryOfRunId, runId)));
|
|
const [settled] = await db.select().from(issues).where(eq(issues.id, issueId));
|
|
return { runId, decision: decision!, retries, issue: settled! };
|
|
}
|
|
|
|
const ownerWakes = async (issueId: string) => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId)))
|
|
.filter(wake => wake.payload?.issueId === issueId && wake.requestedByActorId === "native-status-committer");
|
|
|
|
it("continues a tool-output blocker in a fresh session, and the continuation finishes the task", async () => {
|
|
const issueId = await createTask();
|
|
const first = await runTurn(issueId, { blocker: toolOutputBlocker });
|
|
expect(first.decision.reasonCode).toBe(NATIVE_ENVIRONMENT_BLOCKER_CODE);
|
|
expect(first.issue).toMatchObject({ status: "in_progress", unblockDescriptor: null });
|
|
expect(first.retries).toHaveLength(1);
|
|
expect(first.retries[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: NATIVE_ENVIRONMENT_BLOCKER_CODE,
|
|
error: NATIVE_ENVIRONMENT_BLOCKER_MESSAGE,
|
|
contextSnapshot: { forceFreshSession: true, livenessContinuationInstruction: NATIVE_ENVIRONMENT_BLOCKER_HINT } });
|
|
expect(NATIVE_ENVIRONMENT_BLOCKER_MESSAGE).not.toContain("unavailable");
|
|
expect(NATIVE_ENVIRONMENT_BLOCKER_HINT).toContain("Check whether that blocker still applies");
|
|
expect(await ownerWakes(issueId)).toEqual([]);
|
|
const second = await runTurn(issueId, { done: true }, first.retries[0]!.id);
|
|
expect(second.issue.status).toBe("done");
|
|
}, 120_000);
|
|
|
|
it("continues a blocker the agent owns once, then hands a repeated one to a person instead of retrying again", async () => {
|
|
const issueId = await createTask();
|
|
const first = await runTurn(issueId, { blocker: selfBlocker });
|
|
expect(first.decision.reasonCode).toBe(NATIVE_AGENT_BLOCKER_CONTINUATION_CODE);
|
|
expect(first.issue).toMatchObject({ status: "in_progress", unblockDescriptor: null });
|
|
expect(first.retries).toHaveLength(1);
|
|
expect(first.retries[0]).toMatchObject({ status: "scheduled_retry", scheduledRetryReason: NATIVE_AGENT_BLOCKER_CONTINUATION_CODE,
|
|
scheduledRetryAttempt: 1, contextSnapshot: { forceFreshSession: true, livenessContinuationInstruction: NATIVE_AGENT_BLOCKER_HINT } });
|
|
expect(await taskInterruptionPresentation(db, first.issue)).toMatchObject({ canRetry: false });
|
|
const second = await runTurn(issueId, { blocker: selfBlocker }, first.retries[0]!.id);
|
|
expect(second.decision.reasonCode).toBe(NATIVE_AGENT_BLOCKER_HANDOFF_CODE);
|
|
expect(second.retries).toHaveLength(0);
|
|
expect(second.issue).toMatchObject({ status: "blocked", unblockDescriptor: { owner: "board", action: selfBlocker.unblockAction } });
|
|
expect(await ownerWakes(issueId)).toEqual([]);
|
|
}, 120_000);
|
|
|
|
it("hands a system blocker that needs a credential to a person without automatic retries", async () => {
|
|
const issueId = await createTask();
|
|
const blocked = await runTurn(issueId, { blocker: { ...toolOutputBlocker, reasonCode: "credential_missing",
|
|
unblockAction: "Connect the deploy account." } });
|
|
expect(blocked.decision.reasonCode).toBe(NATIVE_AGENT_BLOCKER_HANDOFF_CODE);
|
|
expect(blocked.retries).toHaveLength(0);
|
|
expect(blocked.issue).toMatchObject({ status: "blocked", unblockDescriptor: { owner: "board", action: "Connect the deploy account." } });
|
|
}, 120_000);
|
|
|
|
it("hands a blocker that only a person can resolve to the board with a Retry that starts the task again", async () => {
|
|
const issueId = await createTask();
|
|
const blocked = await runTurn(issueId, { blocker: { ...selfBlocker, owner: { kind: "external", name: "deploy key holder" },
|
|
reasonCode: "credential_missing", unblockAction: "Add the deploy key to the project." } });
|
|
expect(blocked.decision.reasonCode).toBe(NATIVE_AGENT_BLOCKER_HANDOFF_CODE);
|
|
expect(blocked.retries).toHaveLength(0);
|
|
expect(blocked.issue).toMatchObject({ status: "blocked", unblockDescriptor: { owner: "board", action: "Add the deploy key to the project." } });
|
|
expect(await ownerWakes(issueId)).toEqual([]);
|
|
expect(await taskInterruptionPresentation(db, blocked.issue)).toMatchObject({ responsibility: "user", state: "waiting_for_user",
|
|
userAction: "unblock", canRetry: true, observedRunId: blocked.runId });
|
|
const receipt = await requestTaskRetry({ db, companyId, issueId, actorId: boardUserId, actorSource: "session",
|
|
idempotencyKey: randomUUID(), observedRunId: blocked.runId });
|
|
expect(receipt).toMatchObject({ accepted: true, stale: false });
|
|
const heartbeat = heartbeatService(db);
|
|
await heartbeat.resumeTaskRetry(companyId, receipt.requestId!);
|
|
const retried = (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, agentId)))
|
|
.filter(run => run.contextSnapshot?.issueId === issueId && run.contextSnapshot?.taskRetryRequestId === receipt.requestId);
|
|
expect(retried).toHaveLength(1);
|
|
await heartbeat.drainActiveRunExecutions();
|
|
}, 120_000);
|
|
});
|