fix(native): park a response_wake with no admitted response instead of self-waking (#20935)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run ends with a structured result. The status arbiter
turns that result into an issue status and next actions.
> - A result can report `yielded` with `continuation.kind:
"response_wake"`. This means "wake me when somebody responds".
> - Only a direct Board comment gave this kind a passive wait. Every
other `response_wake` became an immediate `issue_status_changed` wake to
the same agent.
> - The agent woke, saw nothing new, reported `response_wake` again, and
woke again. Nothing bounded this loop. It spent provider budget and its
back-to-back runs caused other failures on warm sandboxes.
> - This pull request parks such a result as a visible wait, and puts
declared continuation wakes under the rewake throttle as a backstop.
> - The benefit is that an agent that waits for input stops and costs
nothing until a real response arrives.

## Linked Issues or Issue Description

**What happened?**

Restart-chaos waves on master `683fd1e9a5` (no faults injected) showed
one task run 30 to 66 times, about 12 s apart. Example: wave
`w-20261111T190352-s5101`, task t02 (W2C), 65 runs in 13 minutes. Each
run reported `yielded` with `continuation.kind: "response_wake"` and the
summary "No changes made. This status-change wake does not trigger phase
2". Each run was woken by `issue_status_changed`, and each finalization
committed `live_continuation_registered` with an `enqueue_continuation`
effect of kind `response_wake`.

Root cause: `arbitrateNativeStatus()`
(`server/src/services/native-runtime/status-arbiter.ts`) only treats
`response_wake` as passive when `boardResponseWaitAuthorized` (from
`readNativeBoardResponseWaitSource`), external chat, or plan wait is
true. Otherwise it falls to the generic yielded branch, and
`status-decision-committer.ts` (`enqueue_continuation`) enqueues an
`issue_status_changed` wake to the same agent. `issue_status_changed` is
not in `THROTTLED_ISSUE_REWAKE_REASONS`, and the native committer writes
an `issue.updated` activity row with the run's id for every decision, so
the rewake throttle saw every native run as progress.

**Expected behavior**

A `response_wake` with no admitted response source parks the task. A
comment, an interaction answer or a status change by somebody else wakes
the agent. No path can loop without a bound.

**Steps to reproduce**

Run `tests/restart-chaos/main.ts --env local --adapters codex
--workloads W2,W2Q,W2C --concurrency 10 --tasks 10 --seed 5101
--lifecycle warm --triggers time`. W2/W2C tasks loop after phase 1.

## What Changed

- `status-arbiter.ts`: a `yielded` / `response_wake` result with no
admitted response source and no blocking remaining work now commits
`response_wake_waiting_for_input`, keeps the task `in_progress`, and has
no effects. Policy version is `phase6-v14`. Board, external chat, plan,
governance, dependency, pause and unfinished-work branches are unchanged
and still come first.
- `native-runtime/native-response-wake-wait.ts` (new) and
`recovery/service.ts`: recovery treats that applied decision as a
durable wait while it is the issue's latest decision, the issue is
`in_progress` with the same agent assignee, and no user comment arrived
after the run started. This applies to the stranded-work sweep and to
the paused-agent passive-wait check.
- `execution-projection.ts`: a committed run with this reason shows the
phase `waiting_for_answer`, label "Waiting for a reply", and a next
action that tells the user to comment or change the status. This shows
only while that decision is still the issue's latest decision and the
issue is `in_progress`; an older run shows "Completed".
- Backstop in `issue-rewake-throttle.ts` and `heartbeat/queue.ts`: an
`issue_status_changed` wake that carries a declared `continuationKind`
(the `enqueue_continuation` effect) is now a throttle candidate. The
native status committer's own `issue.updated` / status-decision activity
no longer counts as run progress or as new input. After two consecutive
successful runs with no issue-visible progress, such a wake waits for
the existing escalating cooldown (2 min, doubling, max 30 min). The
native status dispatcher (`dispatchNativeStatusWakeBatch`) does not drop
a throttled intent: it keeps the intent `queued`, reports the cooldown
deadline to the deferred-receipt worker, and dispatches it again when
the cooldown ends. Status wakes without a declared continuation (blocker
owner, notify owner, retry) are unchanged.
- Restart chaos: new enforced universal check U5 (`universal.ts`). A
task with more than 10 runs fails as `U5:run_budget_exceeded`
(`RESTART_CHAOS_U5_MAX_RUNS`). The README records the calibration.
- `doc/architecture/native-status-arbitration.md`: the decision table
and the wait rules.

## Verification

- RED on master (same reason as the waves):
`heartbeat-process-recovery.test.ts` "parks a native response_wake that
no response admitted…" fails with decision `reasonCode:
"live_continuation_registered"` and effect `{ kind:
"enqueue_continuation", continuationKind: "response_wake" }`.
`heartbeat-issue-rewake-throttle.test.ts` "throttles a
same_agent/response_wake/retry continuation…" fails 3/3 because the wake
is admitted.
- GREEN: the integration test finalizes the run, dispatches native
status wakes and runs the stranded-work sweep three times. It asserts
one run total, one wake request, no recovery action, the task
`in_progress`, and the projection "Waiting for a reply".
- `heartbeat-process-recovery.test.ts` "delays a no-progress native
continuation until the rewake cooldown…" finalizes a real `same_agent`
continuation after a no-progress run, dispatches twice (one throttle
receipt, intent stays `queued`, deadline equals `nextAllowedAt`), then
dispatches after the cooldown and gets the run. It fails when the
dispatcher drops the throttled intent.
- Focused suites pass: `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, `status-arbiter`,
`native-status-arbiter-corpus`, `runtime-mode`, `execution-projection`,
`chat-run-milestone-work`, `heartbeat-dependency-scheduling`,
`heartbeat-run-summary` (904 tests), and all restart-chaos unit tests
(584 tests). `pnpm --filter @paperclipai/server typecheck` passes.
- Universal audit over the 17 saved waves of 2026-11-11: U5 flags 12
tasks, all looping tasks (12 to 66 runs), and 0 tasks on all-pass waves.
- Chaos rerun of local warm seed 5101 (W2, W2Q, W2C): result is added in
a comment on this PR.

## Risks

- Behavioral shift: a model that reports `response_wake` to mean
"continue now" no longer gets an immediate run. The task waits visibly
for a reply. The runner prompt already describes `response_wake` as a
real response wait, and `same_agent` stays available for server-owned
continuations.
- The throttle now applies to native runs. A chain of declared
continuations with no issue-visible output (no comment, document, work
product or other activity) waits for the cooldown after the second such
run. `same_agent` continuations are server-owned (repository
preparation), so this affects only repeated no-output chains.
- Old saved state: decisions committed under `phase6-v13` keep their
recorded effects. A run that finalizes after deploy uses the new policy.
No migration.
- Budget hard-stop behavior is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-11 15:40:44 -06:00
1 parent 80b2369f26
commit edd00526e7
16 files changed
+400 -29

No files matched your search

+11 -3
View File
@@ -1148,9 +1148,9 @@ that lock the run row still fail under the same contention on master. The conten
(`contention_produced`) fired only on tasks whose runs failed before any usage was reported; a run
that finished produced 208 log events and 20 receipts, well above the 40 and 6 thresholds.
## Universal checks (U1–U4)
## Universal checks (U1–U5)
Every task of every wave is also graded on four scenario-independent rules (`universal.ts`), whatever
Every task of every wave is also graded on five scenario-independent rules (`universal.ts`), whatever
its workload. They catch a stuck or misleading task that no scenario anticipated.
| Rule | Fails when | Bound (env override) |
@@ -1159,8 +1159,9 @@ its workload. They catch a stuck or misleading task that no scenario anticipated
| U2 retry loop | more than N consecutive failed runs with the same status and code while the issue status stays the same | 3 (`RESTART_CHAOS_U2_SAME_FAILURE_RUNS`) |
| U3 diagnosable | a failed run has no code or a generic one (`adapter_failed`, `unknown`, `error`, ...) and no recorded `failureCause` (a bounded code such as `40001`, `exit_code_137` or `provider_http_429`; a bare `Error` or `unknown_error` does not count), its stored error is empty or names only a bare cause (`last cause: Error`), or a recovery action's cause or next action is empty or bare | none |
| U4 stale notice | a Retry or failed notice stays on a task after its latest run succeeded, longer than the bound or at rest | 60 s (`RESTART_CHAOS_U4_STALE_SECONDS`) |
| U5 run budget | one task created more runs than the bound | 10 (`RESTART_CHAOS_U5_MAX_RUNS`) |
U1, U2 and U4 fail the task (signature `U<n>:...`). U3 is a warning: it is reported in each grade
U1, U2, U4 and U5 fail the task (signature `U<n>:...`). U3 is a warning: it is reported in each grade
(`grade.universal`), the ledger (`universal`, `universalWarnings`) and the wave's `SUMMARY.md`, and
never fails a task. U3 reads run error text and recovery-action causes from the wave database at
wave end, because the board's run list hides most error text.
@@ -1179,6 +1180,13 @@ node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/universal-audit.ts <r
| U3 | 30 | 3 | 0 |
| U4 | 0 | 0 | 0 |
U5 came later, from the 2026-11-11 `response_wake` self-wake loop: a yielded run with
`continuation.kind: "response_wake"` and no admitted response source enqueued an immediate
`issue_status_changed` wake to the same agent, so W2/W2C tasks ran 30 to 66 times ~12 s apart.
Over the saved waves, passing tasks outside that loop created at most 5 runs, and some looping
tasks still passed every other rule. A task with more than 10 runs now fails as
`U5:run_budget_exceeded`.
On passing tasks the longest ownerless stretch was 19 s, the longest queue wait 136 s (p99 40 s),
the longest accepted-Retry wait 278 s, and no failure repeated more than twice; with the 3-minute
ownerless clock on top, a queued run or a Retry wait is flagged after 8 minutes. U1 flags a C1 task that sat 181 s in `todo` with a resolved interruption (it also failed
+1 -1
View File
@@ -36,7 +36,7 @@ export interface LedgerLine {
masks?: string[];
/** Crash-point suite: the named point this wave armed. */
crashPoint?: string;
/** Universal checks U1–U4 (universal.ts): verdict per rule, and the details of warnings that did not fail the task. */
/** Universal checks U1–U5 (universal.ts): verdict per rule, and the details of warnings that did not fail the task. */
universal?: Record<string, string>;
universalWarnings?: string[];
/** DF1: faulted saves, the disk-full rest with a Retry offered, and whether that Retry worked. */
+1 -1
View File
@@ -478,7 +478,7 @@ export interface TaskGrade {
crossingFaults: string[];
rules: Record<RuleId, RuleResult>;
T: OracleResult;
/** Universal checks U1–U4 (universal.ts); absent on grades written before them. */
/** Universal checks U1–U5 (universal.ts); absent on grades written before them. */
universal?: import("./universal.js").UniversalGrade;
pass: boolean;
signature: string | null;
+2 -2
View File
@@ -1,5 +1,5 @@
/**
* Run the universal checks U1–U4 over saved wave evidence, to measure false
* Run the universal checks U1–U5 over saved wave evidence, to measure false
* positives before a rule is enforced.
*
* node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/universal-audit.ts [--json] <results dir> [...]
@@ -29,7 +29,7 @@ function percentiles(values: number[]) {
return { n: sorted.length, p50: at(0.5), p95: at(0.95), p99: at(0.99), max: sorted[sorted.length - 1] ?? 0 };
}
const RULES: UniversalId[] = ["U1", "U2", "U3", "U4"];
const RULES: UniversalId[] = ["U1", "U2", "U3", "U4", "U5"];
export function auditWaves(roots: readonly string[], bounds: UniversalBounds = DEFAULT_UNIVERSAL_BOUNDS): AuditReport {
const report: AuditReport = { waves: 0, tasks: 0, knownGoodTasks: 0, knownGoodWaves: 0,
+16 -1
View File
@@ -1,7 +1,7 @@
import { describe, expect, it } from "vitest";
import { gradeTask } from "./grader.js";
import type { OracleArtifacts, RunSample, TaskRecord, TaskSample } from "./types.js";
import { bareCauseMessage, DEFAULT_UNIVERSAL_BOUNDS as BOUNDS, diagnosableFailureCause, gradeU1, gradeU2, gradeU3, gradeU4, gradeUniversal, universalBounds } from "./universal.js";
import { bareCauseMessage, DEFAULT_UNIVERSAL_BOUNDS as BOUNDS, diagnosableFailureCause, gradeU1, gradeU2, gradeU3, gradeU4, gradeU5, gradeUniversal, universalBounds } from "./universal.js";
import { workloadDefinitions } from "./workloads.js";
const T0 = Date.parse("2026-11-09T12:00:00Z");
@@ -177,6 +177,21 @@ describe("U4 stale notices", () => {
});
});
describe("U5 run budget", () => {
it("fails a task that keeps starting runs (the response_wake loop shape)", () => {
const runs = Array.from({ length: 65 }, (_, index) => run(`r${index}`, "succeeded", index * 12 * SEC, index * 12 * SEC + 10 * SEC));
const samples = [sample(800 * SEC, "in_progress", runs)];
expect(gradeU5(record(samples, { restAt: null, finalStatus: null, timedOut: true }), BOUNDS))
.toMatchObject({ verdict: "fail", detail: "run_budget_exceeded 65 runs > 10", metrics: { runs: 65 } });
});
it("passes a task within the budget and reads the bound from the environment", () => {
const runs = Array.from({ length: 5 }, (_, index) => run(`r${index}`, "succeeded", index * 20 * SEC, index * 20 * SEC + 10 * SEC));
expect(gradeU5(record([sample(120 * SEC, "done", runs)]), BOUNDS).verdict).toBe("pass");
expect(universalBounds({ RESTART_CHAOS_U5_MAX_RUNS: "4" }).maxRuns).toBe(4);
expect(gradeU5(record([sample(120 * SEC, "done", runs)]), { ...BOUNDS, maxRuns: 4 }).verdict).toBe("fail");
});
});
describe("universal checks in the task grade", () => {
const artifacts: OracleArtifacts = { issueStatus: "done", documents: { brief: "CHAOS-abcd1234\n- one\n- two\n- three\n" },
agentComments: [], files: {}, commands: {}, workspacePath: null };
+14 -3
View File
@@ -1,5 +1,5 @@
/**
* Universal checks U1–U4: scenario-independent rules graded on every task of
* Universal checks U1–U5: scenario-independent rules graded on every task of
* every wave, whatever its workload. Pure functions over observer samples (no
* I/O), so they also run over saved evidence (`universal-audit.ts`).
*
@@ -14,7 +14,7 @@
*/
import type { RunSample, TaskRecord, TaskSample } from "./types.js";
export type UniversalId = "U1" | "U2" | "U3" | "U4";
export type UniversalId = "U1" | "U2" | "U3" | "U4" | "U5";
export interface UniversalResult { verdict: "pass" | "fail" | "n/a"; detail: string; enforced: boolean;
/** Measurements behind the verdict (milliseconds or counts), for bound tuning. */
metrics?: Record<string, number> }
@@ -31,6 +31,7 @@ export interface UniversalBounds {
sameFailureRuns: number;
/** U4: longest a failure or Retry notice may remain after a later run succeeded. */
staleNoticeMs: number;
maxRuns: number;
}
/**
@@ -45,10 +46,11 @@ export const DEFAULT_UNIVERSAL_BOUNDS: UniversalBounds = {
silentRunMs: 3 * 60_000,
sameFailureRuns: 3,
staleNoticeMs: 60_000,
maxRuns: 10,
};
/** Rules that fail a task. The rest are warnings until the audit shows no false positive on known-good waves. */
export const ENFORCED_UNIVERSAL: ReadonlySet<UniversalId> = new Set<UniversalId>(["U1", "U2", "U4"]);
export const ENFORCED_UNIVERSAL: ReadonlySet<UniversalId> = new Set<UniversalId>(["U1", "U2", "U4", "U5"]);
export function universalBounds(env: NodeJS.ProcessEnv = process.env): UniversalBounds {
const seconds = (name: string, fallback: number) => {
@@ -64,6 +66,7 @@ export function universalBounds(env: NodeJS.ProcessEnv = process.env): Universal
silentRunMs: DEFAULT_UNIVERSAL_BOUNDS.silentRunMs,
sameFailureRuns: Math.round(seconds("RESTART_CHAOS_U2_SAME_FAILURE_RUNS", DEFAULT_UNIVERSAL_BOUNDS.sameFailureRuns)),
staleNoticeMs: seconds("RESTART_CHAOS_U4_STALE_SECONDS", DEFAULT_UNIVERSAL_BOUNDS.staleNoticeMs / 1000) * 1000,
maxRuns: Math.round(seconds("RESTART_CHAOS_U5_MAX_RUNS", DEFAULT_UNIVERSAL_BOUNDS.maxRuns)),
};
}
@@ -293,6 +296,13 @@ export function gradeU4(record: TaskRecord, bounds: UniversalBounds): Omit<Unive
return { verdict: "pass", metrics: { staleNoticeMs: worst }, detail: worst ? `notice cleared ${seconds(worst)} after success` : "no notice after success" };
}
export function gradeU5(record: TaskRecord, bounds: UniversalBounds): Omit<UniversalResult, "enforced"> {
const runs = finalRuns(record).size;
return runs > bounds.maxRuns
? { verdict: "fail", metrics: { runs }, detail: `run_budget_exceeded ${runs} runs > ${bounds.maxRuns}` }
: { verdict: "pass", metrics: { runs }, detail: `${runs} runs` };
}
export function gradeUniversal(record: TaskRecord, options: { bounds?: UniversalBounds; db?: UniversalDbEvidence;
enforced?: ReadonlySet<UniversalId> } = {}): UniversalGrade {
const bounds = options.bounds ?? universalBounds();
@@ -302,6 +312,7 @@ export function gradeUniversal(record: TaskRecord, options: { bounds?: Universal
U2: gradeU2(record, bounds),
U3: gradeU3(record, options.db),
U4: gradeU4(record, bounds),
U5: gradeU5(record, bounds),
};
return Object.fromEntries(Object.entries(results).map(([id, result]) => [id, { ...result, enforced: enforced.has(id as UniversalId) }])) as UniversalGrade;
}