Files
PaperClipAI/doc/architecture/native-status-arbitration.md
T
DottaandClaude Opus 5.5 f92f91f0fb fix(native): finish a parent claim when its child results arrived before the run started (#21189)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task gets a wake when one of its child tasks completes. The
wake carries the child results.
> - When the parent already has a run, the child result wake waits
(`deferred_issue_execution`) behind that run.
> - At finalization, a parent completion claim stays open
(`native_child_completion_pending`) while such a wake is still queued,
claimed or deferred. This guard protects child results that arrive while
the parent works.
> - The guard did not check when the child result arrived. A child
result saved before the parent run started was already visible to that
run, but it still blocked the claim.
> - So the task stayed `in_progress`, the deferred wake started a second
run, and the agent repeated its completion summary.
> - This pull request counts a child result as undelivered only when it
was saved after the finishing run started.
> - The benefit is that a parent that already read its child results
finishes in one turn, and child results that arrive during the run still
get a fresh turn.

## Linked Issues or Issue Description

Refs #15372 (added the pending child-completion guard). Refs #20935
(same family: a self-wake after a settled run).

**What happened?**

Restart-chaos wave `w-20261112T181948-s7101`, task t001 (OpenCode,
native runner, staging-mix parent with two subtasks). Both subtasks were
done before parent run `8006a205` started at 18:45:29. The run read
them, posted its final comment and claimed `done` at 18:46:15.860. A
child-completion wake for the parent had been deferred since about 18:42
behind that run. The finalizer found that wake and committed
`native_child_completion_pending`, so the task stayed `in_progress`. 63
ms later the release promoted the deferred wake into run `9282de34`
(`invocationSource: automation`, `wakeCommentId: null`). Its
`paperclip.native-continuation.v1` turn carried the subtask results that
the first run had already used, plus the agent's own two comments
(resume delta since the base run). That run posted a second "complete"
summary and only then moved the task to `done`.

Root cause: `hasPendingNativeChildCompletion()`
(`server/src/services/native-runtime/native-child-completion-delivery.ts`)
treated every queued, claimed or deferred child-completion wake for the
issue as unread, except a wake bound to the finishing run itself. It did
not compare the wake's save time with the run's start.

**Expected behavior**

A child result that was saved before the finishing run started is not
new input for that run. The completion claim completes the task. The
deferred wake is cancelled at release as a wake on a terminal task. A
child result saved during the run still keeps the task open and gets a
fresh parent turn.

**Steps to reproduce**

1. Create a parent task with a done child. Save a deferred
`issue_children_completed` wake for the parent.
2. Start a native parent run after that wake was saved. Let it claim
`done`.
3. Finalize the run. On `master` the decision is
`native_child_completion_pending` and the task stays `in_progress`; the
deferred wake then starts a second run.

## What Changed

- `native-child-completion-delivery.ts`:
`hasPendingNativeChildCompletion()` counts a child-completion wake only
when its `createdAt` is at or after the finishing run's `startedAt`. A
run without a recorded start keeps the old behavior: every such wake
stays pending. The finalizer, the status-commit recheck and status
reassessment all use this function.
- `heartbeat-process-recovery.test.ts`: a regression test finalizes a
real native done claim with a deferred child-completion wake saved
before (task becomes `done`) or after (task stays `in_progress` as
`native_child_completion_pending`) the run started. It then releases the
issue execution through the wake queue. In the `before` case the
deferred wake is cancelled and there is one run total. In the `after`
case the wake is promoted into a fresh run that carries the child
results.
- `doc/architecture/native-status-arbitration.md`: describes when a
child result counts as undelivered.

## Verification

- RED on `master` (`ef6d347f79`): `heartbeat-process-recovery.test.ts`
"treats a child result saved before the parent run started…" fails with
decision `reasonCode: "native_child_completion_pending"`, `toStatus:
"in_progress"`. This is the same decision as the wave. The "after" case
passes on `master`.
- GREEN: both cases pass with the fix.
- Neighbouring suites pass: `native-status-arbiter-corpus` (includes the
queued / claimed / deferred / consumed child-delivery cases from
#15372), `heartbeat-dependency-scheduling` (revised child completion
delivered after a running parent), `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, and `src/services/native-runtime`.
- `pnpm --filter @paperclipai/server typecheck` passes.
- The regression test runs finalization and release end to end. The
cancel-at-release step itself is existing behavior, also covered by
`modules/wake-queue/application/use-cases.test.ts` ("cancels stale
assignee continuations before claiming promotion on %s tasks", `done`
case).

## Risks

- A child result saved before the run started no longer blocks the
claim. The child status was already `done` when the run started, so the
run could read it. If the clocks of two server instances differ, a child
result saved during the run could look older than the run start; the
window is the clock skew.
- A re-dispatched native status wake gets a new row with a later
`createdAt`. Such a row stays pending, which is the old behavior.
- No migration. Saved wakes from older builds work unchanged, because
the check reads only existing columns.
- Scope: this change does not alter the comment-wake promotion path or
the resume-delta builder.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:19:32 -06:00

429 lines
20 KiB
Markdown

# Native status arbitration
Native runner results do not directly mutate an issue's status. The model may
report that work is done, blocked, ready for review, or yielded, but Paperclip's
server remains the authority that decides and commits the resulting workflow
state.
The native status pipeline is:
```text
structured runner result
|
v
schema and terminal validation
|
v
evidence classification
|
v
pure status arbitration
|
v
transactional decision commit
|
+--> issue status/version
+--> durable side effects
+--> audit and recovery records
```
This separation prevents model prose from acting as a privileged status
command, protects newer issue state from stale runs, and makes every decision
replayable and auditable.
## Inputs to finalization
The runner returns a `paperclip.run_result.v1` result and a matching terminal
record. Important result fields include:
- `reportedWorkDisposition`: `done`, `blocked`, `needs_review`, or `yielded`;
- `completionClaim`, including the contract revision, criterion claims, and
remaining work;
- `verification` claims;
- evidence references;
- an optional blocker; and
- an optional continuation.
The server also owns facts the runner cannot choose:
- the persisted completion contract;
- the run's actual terminal state;
- whether workspace finalization succeeded;
- the issue's current status and status version;
- pending approvals, interactions, and execution-policy stages; and
- the completion-authority policy recorded on the contract.
## Evidence classification
`classifyNativeEvidence()` compares model claims with durable Paperclip
records. It recognizes these evidence families:
- run events with an authoritative control-plane evidence verdict;
- issue work products;
- approvals;
- issue-thread interactions; and
- attachments.
Each evidence reference becomes one of:
| Outcome | Meaning |
| --- | --- |
| `accepted` | A matching durable record exists and authoritatively supports the claim. |
| `missing` | The required claim or referenced record does not exist or is still pending. |
| `rejected` | The record or claim explicitly contradicts completion. |
| `unverifiable` | A record exists, or a string was supplied, but it is not authoritative evidence. |
For example, a model-authored reference such as `task-response` is not trusted
merely because it looks descriptive. Likewise, the model's own
`run.result.proposed` event is a claim, not independent proof.
The classifier produces a `NativeEvidenceAssessment` containing:
- contract-revision validity;
- criterion claim and evidence outcomes;
- verification claim and evidence outcomes;
- accepted, missing, rejected, and unverifiable references;
- blocking remaining work;
- normalized blocker or continuation data; and
- pending attention requests.
The classifier does not update the issue.
## Completion authority
The persisted completion contract controls how a `done` claim may be accepted.
### Durable-evidence completion
The strongest completion path requires all of the following:
- the runner reports `done`;
- the objective is satisfied;
- every contract criterion has accepted durable evidence;
- every verification has accepted durable evidence; and
- no remaining work blocks completion.
This produces `completion_contract_satisfied`.
### Low-risk claim-policy completion
Ordinary issue completion changes Paperclip workflow state, but it does not by
itself authorize deployments, spending, secret access, approvals, or arbitrary
API calls. Default native completion contracts therefore use low-risk
`agent_claim_policy` authority.
Under that policy, a result can complete the issue when:
- it reports `done`;
- its contract revision matches;
- every contract criterion is claimed `satisfied`;
- every verification is claimed `passed`; and
- no remaining work blocks completion.
This produces `completion_claim_policy_accepted`. Independently governed tools
and effects still enforce their own authorization and approval rules.
## Decision order
`arbitrateNativeStatus()` is a pure function. It evaluates higher-authority
conditions before model disposition:
| Condition | Status decision | Important effects/reason |
| --- | --- | --- |
| Issue is already `done` or `cancelled` | Preserve | `terminal_status_preserved` |
| Workspace finalization failed | Preserve | Record a retryable finalization error |
| Run performs a named native completion review | Preserve | The recorded review decision controls the child; unresolved review records a reviewer recovery action |
| Run was cancelled | Preserve | Release run resources |
| Run failed | Preserve | Schedule recovery |
| Approval, interaction, or execution stage is pending | `in_review` | Materialize/bind the governance gate and notify its owner |
| Completion satisfies its authority policy | `done` | Release checkout |
| Runner reports a concrete attention request with a reviewer and decision | `in_review` | Bind the requested reviewer |
| Runner reports `needs_review` without a decision, or an incomplete completion claim | Keep work with the agent | No automatic human approval; at most one corrective continuation, then a visible recovery action |
| Runner reports a task-wide blocker | `blocked` | Persist blocker owner and unblock action |
| Runner reports a current-track blocker | `in_progress` | Enqueue another productive track |
| Runner reports `yielded` / `response_wake` with no admitted response source | `in_progress` | `response_wake_waiting_for_input`: no wake; the task shows "Waiting for a reply" until a comment, answer or status change wakes it |
| Runner reports `yielded` with another valid continuation | `in_progress` | Enqueue the declared continuation |
| Ordinary task reports `response_wake` and blocking remaining work, without a real wait | `in_progress` | Use the bounded incomplete-work continuation, then a visible recovery error if repair was already used |
| Completion evidence is incomplete and continuation is forbidden | Preserve | Record a finalization error and named next action |
| Completion evidence is otherwise incomplete | `in_progress` | Enqueue a bounded, idempotent continuation |
The output is a `NativeStatusDecision` containing:
- the arbiter policy version;
- `statusAction` and `toStatus`;
- a stable reason code;
- an optional unblock descriptor; and
- declarative side effects.
The pure arbiter performs no database or network writes.
A response to a Board comment does not by itself justify parking unfinished
work. When a `yielded` / `response_wake` result contains
`remainingWork[].blocksCompletion: true`, `paperclip_finish` rejects the report
unless a real wait owns the next action. The agent can correct it in the same
turn. Finalization also checks accepted reports and uses the existing
`native-completion-incomplete` one-follow-up budget. Model-selected retry keys
cannot reset that budget. Pending interactions, approvals, dependencies, pause
holds, and conversation lifecycles retain their authority.
The current Board source is rechecked in the status transaction before a repair
is queued. Edited, deleted, or superseded comments do not authorize replay of an
old response. Recovery no longer treats an old committed passive wait as healthy
when its report admits blocking work and its Board source is still current.
Existing recovery gates continue to protect pauses, budgets, and ownership.
A `response_wake` asks for the next response; it is never a request to run
again now. Only a directly admitted Board comment, an authorized external chat,
an accepted plan or a governance gate gives it a specific wait record. Every
other `response_wake` parks the task as `response_wake_waiting_for_input`.
Recovery treats that applied decision as a durable wait while it is the issue's
latest decision and no user comment arrived after the run started. A comment,
an interaction answer or a status change by someone else wakes the agent through
its normal path.
Declared continuations that do enqueue a wake (`issue_status_changed` with a
`continuationKind`) are throttle candidates in the issue rewake throttle. After
two consecutive successful runs with no issue-visible progress, the wake waits
for the escalating cooldown. The native status dispatcher keeps the intent
queued and dispatches it again at the cooldown deadline; it is never dropped.
The native status committer's own `issue.updated` record does not count as
progress or new input. The "Waiting for a reply" execution label shows only
while the parked decision is still the issue's latest decision and the issue
is `in_progress`.
A completion claim stays `in_progress` as `native_child_completion_pending`
while an undelivered child-completion wake for the same issue and agent is
queued, claimed or deferred. Only a child result saved after the finishing run
started counts as undelivered. A result saved before that run started was
already visible to it, so the claim completes the task and the deferred wake
is cancelled at release instead of starting another turn. A run with no
recorded start keeps every such wake pending.
## Transactional decision commit
`commitNativeStatusDecision()` applies the decision against authoritative issue
state. It uses the issue's prior status, status version, and prior decision ID
as compare-and-swap inputs. If another actor changed the issue first, the commit
raises `NativeStatusRaceError`; finalization reloads current state, reassesses,
and retries a bounded number of times.
Within the transaction, the committer:
1. validates that the assessment and issue bindings still match;
2. records the status decision and reason;
3. updates the issue status and increments its version when required;
4. materializes declared effects;
5. writes an effect ledger with deterministic idempotency keys;
6. updates the native-finalization coordinator; and
7. persists audit activity.
After commit, activity publications are emitted. Reconciliation may redeliver a
pending effect, but it resumes the recorded ledger decision; it does not ask the
model or arbiter to invent a new decision.
## Durable side effects
Depending on the decision, materialized effects may include:
- an idempotent agent wake request;
- an issue-thread interaction;
- a reviewer binding or owner notification;
- a persisted blocker and unblock action;
- a scheduled retry or recovery action;
- a delegated child issue;
- checkout/resource release; or
- finalization/reconciliation records.
Effect rows bind company, issue, decision, target, ordinal, delivery state, and
idempotency key. This is what makes a status transition with follow-up work
recoverable after a process crash.
## Governance precedence
A model cannot bypass a pending governance gate by reporting `done`. Before
completion is considered, finalization checks:
- an active execution-policy stage;
- a pending issue-thread interaction; and
- a pending or revision-requested approval linked to the issue.
If one exists, the issue goes to `in_review` and the durable gate remains the
path forward.
## Failure and recovery behavior
Status finalization is coordinated by `native_run_finalizations`. Important
phases include observation, workspace finalization, assessment, arbitration,
commit, and retryable or terminal failure.
Examples:
- an unsafe workspace archive triggers one automatic export of confined entries,
omitting unsafe links; if it remains unsafe, discard the export and settle the saved
result with an informational run-log entry and no task warning or repair action;
- other workspace-finalization failures preserve the claim and record a retryable
error rather than falsely completing the issue;
- a failed provider run preserves partial evidence and schedules recovery;
- a status-version race causes bounded reassessment against current issue
state; and
- a materialization failure records the failed phase and next retry time rather
than silently dropping the side effect.
## Policy upgrades
The policy version on an assessment is audit metadata. New runs use the current
rules. A version change alone does not reassess an old run, change task status,
or ask a person to review completion. New evidence and explicit status changes
still use the existing reconciliation paths.
Reconciliation also withdraws pending review cards created solely by the old
policy-version check. It restores the previous status only if that exact decision
and status version are still current and no other review gate is pending. A later
user or agent decision takes precedence. The old assessments and decisions remain
in the audit history; cleanup does not accept or reject the agent's work.
## Diagnosing an unexpected status
Start with the terminal heartbeat run and inspect:
1. `resultJson.nativeResult.reportedWorkDisposition`;
2. completion-claim contract revision and criterion statuses;
3. verification statuses and evidence references;
4. `resultJson.assessmentId` and `decisionId`;
5. `resultJson.authoritativeDecision`;
6. `resultJson.issueStatusBefore` and `issueStatusAfter`;
7. `finalizationPhase` and `workspaceFinalizeStatus`; and
8. pending approvals, interactions, execution stages, blockers, or
continuations on the issue.
Common patterns:
- `done` claim + `in_progress` decision: evidence/claim policy did not accept
completion, or blocking work remained;
- `done` claim + `in_review`: a governance gate took precedence;
- successful run + unchanged status + retryable finalization: workspace or
status-effect commit failed;
- repeated `issue_continuation_needed` runs: the issue remains open without a
converging completion or durable wait path; and
- terminal issue unchanged by a stale run: terminal-state preservation or
status-version race protection worked as designed.
## Primary implementation locations
- `server/src/services/native-runtime/evidence-classifier.ts` — validates
claims against durable records.
- `server/src/services/native-runtime/status-arbiter.ts` — pure policy and
decision table.
- `server/src/services/native-runtime/status-decision-committer.ts` — CAS
commit, effect materialization, ledger, and audit.
- `server/src/services/native-runtime/native-run-finalizer.ts` — orchestrates
assessment, governance checks, arbitration, bounded race retry, and result
projection.
- `server/src/services/native-runtime/completion-contracts.ts` — creates and
versions default native completion contracts.
- `server/src/services/native-runtime/native-finalization-reconciler.ts` —
resumes interrupted finalization without re-inventing committed decisions.
- `server/src/services/recovery/service.ts` — repairs open issues that finish
without a live or durable wait path.
See also
[`durable-continuation-scheduler.md`](./durable-continuation-scheduler.md) for
the scheduler and recovery behavior that follows an `in_progress` decision.
## Explicit completion reviews
Ordinary task completion uses the agent's structured `done` claim under the
contract's low-risk claim policy. Unknown evidence references remain diagnostic
information; they do not create human approval requirements. Cancellation,
newer task state, unresolved dependencies, and explicit governance still win.
Paperclip no longer creates a generic "Native completion review" because a
report is incomplete, verification failed, or the agent says `needs_review`.
A new review interaction requires an explicit attention request naming the
reviewer's responsibility and the decision. The card displays that request.
Waiting for CI remains agent work, not a human completion approval.
The native runner returns current approval/dependency constraints to the agent
when it calls `paperclip_finish`. An empty `needs_review` report without an
existing gate is rejected with instructions to correct it. The final reply must
explain any required user action and link to the relevant task or approval.
The tool acknowledges receipt, not a premature status commit: final status is
committed only after the provider turn and workspace finalization settle.
On upgrade, bounded cleanup withdraws only unanswered, system-created fallback
cards proven by their decision/effect ledger, original prompt/target, empty
attention request list, and low-risk claim policy. Explicit or answered reviews
and stronger completion policies are preserved. Withdrawal has audit history
and retires chat actions. Reconciliation reassesses only the current successful
run's result, with the same task status/version and completion contract and no
newer execution owner. It applies normal governance and dependency checks and
appends a decision; it never marks every affected task done blindly. A persisted
withdrawal marker makes restart between cleanup and reassessment retryable.
### Agent review handoff
A review request and its next action must be saved together. For a native
completion review addressed to an agent, the status transaction saves both the
review card and one durable wake for that agent. The wake identifies the card
and the status decision. Retrying the transaction does not create another wake.
The reviewer runs on the child task, even when the reviewer's own parent task
is blocked on that child. The worker remains the child task's assignee. This
review role gives access to the child context, history, and documents, plus
`resolve_review`. It does not give access to ordinary task mutation tools.
The usual company, invokability, budget, and workspace gates still apply.
Human-only requests and governed actions do not acquire this review role.
This restriction applies to Paperclip tools. It is not a read-only filesystem
boundary. The reviewer retains the configured agent and environment permissions,
including the ability to run tests and create temporary files. An operator who
needs filesystem isolation must configure it in the execution environment.
The review role does not raise the provider permission mode or bypass a sandbox.
Admission claims the reviewer run, its wake, and the child execution lock in
one transaction. If another run holds the lock, the reviewer stays queued.
The provider does not start without this claim.
`resolve_review` accepts or rejects the one card assigned to the run. Rejection
requires specific requested changes. The server checks the reviewer, report,
and current task version again before it accepts a decision. It also checks that
the acting run is running, holds the execution lock, and names this exact card
and decision. Another run for the same agent cannot resolve the card. An old or
reassigned review does not grant access to newer work.
The final required acceptance completes the child through the existing review
resolution path and makes its blocked dependents eligible to run. Rejection
returns the requested changes to the worker. A reviewer must record that decision
before calling `paperclip_finish`. Finishing the review run cannot turn rejected
work into Done. If the reviewer cannot act, `paperclip_block` preserves the task
and records a recovery action for the reviewer without an automatic retry loop.
Both Codex and ACPX providers wait for server completion feedback before their
terminal tool call succeeds. A rejected completion report stays correctable in
the same provider turn. The runner does not latch it as an accepted result.
## Completed runs with saved monitors
A successful accepted result can report a monitor wait after the run has
already released execution and checkout. Finalization can preserve that wait
when the same agent still owns a valid ordinary task monitor and the completed
run has a server-recorded `set_task_monitor` receipt for the task. The current
monitor can have a later deadline; finalization does not restore an older timer.
The server rechecks the accepted source, current monitor, empty execution and
checkout slots, and governance under the coordinator and task locks. The
`scheduled_monitor_preserved` decision keeps task status and has no task effects.
It records the completed result without dispatching another provider turn.
The existing monitor scheduler receives a transaction-aware notification when
this finalization blocker clears. Replaying the committed decision sends no
second notification.
Live response selection and committed response repair retain the accepted
monitor explanation under the existing reply and publication authorization rules.
Missing receipts, changed ownership, cancellation intents, cleared or quota
monitors, and unfinished workspace finalization retain their recovery paths.
Due and exhausted monitors still belong to the scheduler. This does not change
their deadlines or attempts, clear unrelated recovery holds, or relax the
Cloud sleep and wake contract.