Files
PaperClipAI/server/src/modules
DottaandClaude Opus 5.5 7685b7a4fc fix(native): continue a run that reports a system environment blocker (#21959)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run ends with one semantic report: `paperclip_finish` or
`paperclip_block`
> - An agent can report a system environment blocker, for example when
it thinks its tool results are corrupted
> - Today that blocker binds to the same agent, its wake is dropped, and
the task stays blocked with no Retry
> - In a restart-chaos rerun, an OpenCode agent reported such a blocker,
then recovered in the same turn and did all the work, but its later
`paperclip_finish` was refused with an opaque conflict error
> - This pull request makes Paperclip continue a system environment
blocker automatically in a fresh session, and tells the agent clearly
that a turn's accepted blocker is final
> - The benefit is that a temporary environment problem no longer needs
a person to unstick the task

## Linked Issues or Issue Description

**What happened?**

A native OpenCode run reported `paperclip_block` with owner `{kind:
system, name: execution-environment}`, `reasonCode:
environment_unavailable` and the unblock action "Resume/retry the run".
In the same turn the agent recovered: it saved the task document and
posted the requested comment. Its later `paperclip_finish` failed with
"A different semantic result was already committed". The run succeeded
with disposition `blocked`. The status decision bound a task-wide
blocker to the same agent. Both owner wakes were skipped
(`chat_control_recovery_proof_unresolved`). The task stayed blocked with
no Retry.

**Expected behavior**

A system environment or tool blocker is a harness problem. Paperclip
must continue the task automatically, with a bound, in a fresh session.
A provider that tries to replace an accepted blocker must get clear
guidance, not an opaque conflict.

**Steps to reproduce**

1. Run a native task with an agent that calls `paperclip_block` with
owner kind `system` and reason code `environment_unavailable`.
2. In the same turn, call `paperclip_finish` with a done report.
3. See that the finish is refused with "A different semantic result was
already committed" and that the task stays blocked with no Retry.

**Paperclip version or commit**

`master` at b2b2c1e60b.

## What Changed

- `server/src/services/native-runtime/native-environment-blocker.ts`:
new lane constants and `isNativeEnvironmentBlocker`. A blocker qualifies
when its owner kind is `system` and its reason code is
`environment_unavailable` or `tool_unavailable`.
- `status-arbiter.ts`: for the current owner's environment blocker (not
a chat, not paused), schedule a fresh-session retry
(`native_environment_blocker`). After 2 retries on the shared failure
counter, bind a board-owned blocker
(`native_environment_blocker_exhausted`) with a clear next step. Other
blockers keep their current behavior.
- `native-run-finalizer.ts`: compute the environment-blocker fact from
the accepted result, only for a succeeded run that the task's owner
still holds and that nobody stopped. Both new reason codes require the
provider-failure owner recheck under the issue lock.
- `status-decision-committer.ts`, `native-provider-capacity-retry.ts`:
materialize the retry through the existing fresh-session retry lane
(`forceFreshSession`, `retryOfRunId`, 30 s then 60 s delay, refuses a
stopped run). The retry run carries a liveness instruction that tells
the agent to check saved files, documents and comments before it repeats
a step.
- Runner `completion-result.ts` and the OpenCode, Codex and ACPX
drivers: when a turn already committed a blocker, a later different
report gets `PRP_BLOCKER_ALREADY_REPORTED_MESSAGE`. It says the first
accepted report of a turn is final, tells the agent to list the
completed steps in its final response, and says that Paperclip continues
a system environment blocker automatically. Other conflicts keep the old
text.
- I kept the one-result-per-turn rule. runnerd
(`admit_terminal_tool_authority`, `paperclip/runResult`), the native
runtime and checkpoint replay all enforce it, so a superseding second
result would need a protocol change across the Rust runner, three
drivers and replay. The automatic fresh-session retry closes the gap
without that change.
- OpenCode permission denial after reattach: not a product bug. The
`external_directory` rules come only from the run's working directory
and its instruction root, which stay the same across a hand-off. The
event log shows the same workspace and session paths after both resumes.
Only one call was denied, during the model's burst of garbled tool
calls, which matches the intended workspace confinement.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-environment-blocker.integration.test.ts`
(new). It runs the real executor, finalizer and committer on embedded
Postgres. Red before the arbiter change (2 of 3 fail), green after.
- An environment blocker schedules one fresh-session retry. The task
stays `in_progress` and shows `retry_scheduled` with no Retry button.
- With the retries spent, the task is blocked with a board-owned unblock
action.
  - A user-owned `credential_missing` blocker still binds as before.
- `cd server && npx vitest run
src/services/native-runtime/status-arbiter.test.ts`: new case for the
bound, pause hold, chat and non-environment exclusions.
- `cd packages/paperclip-runner && npx vitest run
src/drivers/opencode/opencode-server-driver.test.ts -t "accepted blocker
is final"` (new). The fake OpenCode server blocks, then tries to finish
in the same turn. Red before the driver change, green after.
- Focused suites pass: OpenCode, Codex semantic-result and ACPX driver
tests plus runner `src/cli` (26 files, 554 tests); arbiter corpus,
tool-receipt-limit and provider-capacity integration tests (5 files, 198
tests).
- `tsc --noEmit` passes for `packages/paperclip-runner` and `server`.
- `pnpm verify` asks for broad CI coverage because the change touches a
shared runner fixture. Required CI supplies that coverage.

## Risks

- More automatic runs: each environment blocker can add at most 2 runs
in a chain. The retries share the failure counter with the other failure
lanes. The company-wide automatic-run brake admits them like any other
recovery run. Worst case during an outage is 2 extra starts for each
blocked task, and the brake holds them if they keep failing.
- A real, persistent environment problem now takes up to about 90
seconds of retries before it reaches the board. This is intended.
- Previous release: no persisted shape changes. The work assessment JSON
is unchanged, because the fact is computed from the saved result at
finalization. A run finalized by the previous release keeps its saved
decision.
- Agents see a new message when they try to replace an accepted blocker.
The tool descriptions are unchanged, so the prompt size is the same.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (code search, editing, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 22:40:37 -06:00
..

Feature modules

A feature module under server/src/modules/<name>/ uses three layers. Each layer has one rule: which layer below it, it may import from.

adapters  →  application  →  domain
  • domain/ holds pure business rules. A domain file takes plain data in and returns plain data out. A domain file must not import drizzle-orm, @paperclipai/db, a service under server/src/services/, a route under server/src/routes/, or a Node.js I/O module (node:child_process, node:fs, node:net). A domain function must not read the system clock; the caller passes now as an explicit Date value.
  • application/ holds use cases and the ports they need. A use case takes its ports as constructor arguments and calls domain functions for policy decisions. An application file must not import drizzle-orm, a SQL client, a concrete adapter, or the server's HTTP error helpers. The outer service or route translates application errors into transport responses.
  • adapters/ holds the concrete implementations of the ports: Postgres queries, transactions, and process control. An adapter file may import drizzle-orm, @paperclipai/db, and Node.js I/O modules.

A module exposes its commands through index.ts, which composes the adapters and the use cases behind a factory function. Code outside the module imports that entry point, never a file inside domain/, application/, or adapters/ directly.

pnpm check:module-boundaries enforces these rules for production source files. It also checks access to the company deletion entry point below.

Company deletion

The existing company deletion service owns the database transaction and deletion order. A module can expose a separate company-deletion.ts entry point. Only services/company-deletion.ts can import this entry point. The entry point exports an implementation of CompanyDeletionParticipant from lib/company-deletion.ts. Do not export it through the normal module index.

The service must complete external cleanup before it starts the transaction. It must lock the company before it deletes dependent records or calls a module. Creation commands must use the same company lock. The service removes dependent records before it calls the module. The module checks its deletion conditions and deletes the records it owns. Use the supplied transaction. Do not start another transaction or call external systems. Throw an error if deletion cannot proceed. An error must roll back all database changes in the deletion transaction.

Call each module explicitly in dependency order. No module registry is required.

The agent lifecycle module also has a restricted configuration.ts entry point. It saves configuration and invalidates verification in the caller's transaction. The lifecycle guide lists its permitted callers. Ordinary configuration updates use the root lifecycle command.

The agent lifecycle module owns agent creation, pause, resume, and termination. Use its commands for these changes.

The execution admission module owns project task-execution policy and the durable admission records that charge project capacity. Today it exposes only internal policy commands and the deletion guards that project, agent, and company deletion call. Settlement releases active charges but keeps each admission's complete project set for later retained-session resolution. Execution claims still belong to services/heartbeat/queue.ts and native review dispatch. See execution admission accounting.

Execution admission accounting

The execution-admission module (server/src/modules/execution-admission/) owns three tables. Nothing writes admissions or caps in production yet, and no claim path enforces a project maximum.

  • project_task_execution_policies stores a project's maximum concurrent tasks apart from projects, so project readers never return an unenforced cap. No row means never configured. A row with a null maximum means an explicit "no limit", which a later legacy migration must not overwrite. Maximums are positive. The row is deleted with its project.
  • execution_admissions stores one admission attempt of a run. A run can be released before dispatch and admitted again, so (run_id, attempt) is unique and a partial unique index allows one unsettled attempt per run. The row keeps the context revision, the nullable primary project as provenance (no foreign key) and the settlement identity, which is set together with settled_at. admitted_project_ids is the complete admitted project set, sorted, kept after settlement and immutable (a trigger rejects any change). It has no project foreign key, so settled history never blocks project deletion and keeps the deleted project's ID. {} is a known empty set. Null means unknown and is allowed only on settled rows: migration 0428 filled it from the charges of unsettled rows, but settled 0426 rows had already lost their set. A retained provider session resumes with the previous set ∪ the current references; an unknown previous set requires a fresh context, never an empty or primary-only guess.
  • execution_admission_project_charges stores one row per distinct admitted project, including unlimited projects. An admission without projects has no rows. Composite (company_id, …) foreign keys keep admissions and projects in one company. Rows are the only active capacity and exist only while the admission is unsettled: settlement deletes the whole set in one transaction, and (company_id, project_id) is indexed for active counts, so lookups scale with active work, not history.

Archiving a project changes no charge. Deleting a project with a charge returns 409 project_execution_admission_pending; the foreign key is the backstop. Deleting an agent refuses unsettled admissions on its runs and removes settled ones, with their sets, together with the runs. Company deletion removes the company's admissions before its runs and projects.

The proposed global lock order for the admission transaction is: company row (FOR NO KEY UPDATE), native finalization recovery owner, the whole project set in ascending ID order (FOR NO KEY UPDATE, one statement), issues, wake requests, the run, the external-admission advisory lock, the admission row and its charges, then budget reservations. Policy updates take the same project lock. Settlement takes no project lock. Only the project-set step is implemented so far; claim paths keep their current locks until they are integrated.