Files
PaperClipAI/server/src/services/issue-rewake-throttle.ts
T
DottaandClaude Opus 5.5 edd00526e7 fix(native): park a response_wake with no admitted response instead of self-waking (#20935)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run ends with a structured result. The status arbiter
turns that result into an issue status and next actions.
> - A result can report `yielded` with `continuation.kind:
"response_wake"`. This means "wake me when somebody responds".
> - Only a direct Board comment gave this kind a passive wait. Every
other `response_wake` became an immediate `issue_status_changed` wake to
the same agent.
> - The agent woke, saw nothing new, reported `response_wake` again, and
woke again. Nothing bounded this loop. It spent provider budget and its
back-to-back runs caused other failures on warm sandboxes.
> - This pull request parks such a result as a visible wait, and puts
declared continuation wakes under the rewake throttle as a backstop.
> - The benefit is that an agent that waits for input stops and costs
nothing until a real response arrives.

## Linked Issues or Issue Description

**What happened?**

Restart-chaos waves on master `683fd1e9a5` (no faults injected) showed
one task run 30 to 66 times, about 12 s apart. Example: wave
`w-20261111T190352-s5101`, task t02 (W2C), 65 runs in 13 minutes. Each
run reported `yielded` with `continuation.kind: "response_wake"` and the
summary "No changes made. This status-change wake does not trigger phase
2". Each run was woken by `issue_status_changed`, and each finalization
committed `live_continuation_registered` with an `enqueue_continuation`
effect of kind `response_wake`.

Root cause: `arbitrateNativeStatus()`
(`server/src/services/native-runtime/status-arbiter.ts`) only treats
`response_wake` as passive when `boardResponseWaitAuthorized` (from
`readNativeBoardResponseWaitSource`), external chat, or plan wait is
true. Otherwise it falls to the generic yielded branch, and
`status-decision-committer.ts` (`enqueue_continuation`) enqueues an
`issue_status_changed` wake to the same agent. `issue_status_changed` is
not in `THROTTLED_ISSUE_REWAKE_REASONS`, and the native committer writes
an `issue.updated` activity row with the run's id for every decision, so
the rewake throttle saw every native run as progress.

**Expected behavior**

A `response_wake` with no admitted response source parks the task. A
comment, an interaction answer or a status change by somebody else wakes
the agent. No path can loop without a bound.

**Steps to reproduce**

Run `tests/restart-chaos/main.ts --env local --adapters codex
--workloads W2,W2Q,W2C --concurrency 10 --tasks 10 --seed 5101
--lifecycle warm --triggers time`. W2/W2C tasks loop after phase 1.

## What Changed

- `status-arbiter.ts`: a `yielded` / `response_wake` result with no
admitted response source and no blocking remaining work now commits
`response_wake_waiting_for_input`, keeps the task `in_progress`, and has
no effects. Policy version is `phase6-v14`. Board, external chat, plan,
governance, dependency, pause and unfinished-work branches are unchanged
and still come first.
- `native-runtime/native-response-wake-wait.ts` (new) and
`recovery/service.ts`: recovery treats that applied decision as a
durable wait while it is the issue's latest decision, the issue is
`in_progress` with the same agent assignee, and no user comment arrived
after the run started. This applies to the stranded-work sweep and to
the paused-agent passive-wait check.
- `execution-projection.ts`: a committed run with this reason shows the
phase `waiting_for_answer`, label "Waiting for a reply", and a next
action that tells the user to comment or change the status. This shows
only while that decision is still the issue's latest decision and the
issue is `in_progress`; an older run shows "Completed".
- Backstop in `issue-rewake-throttle.ts` and `heartbeat/queue.ts`: an
`issue_status_changed` wake that carries a declared `continuationKind`
(the `enqueue_continuation` effect) is now a throttle candidate. The
native status committer's own `issue.updated` / status-decision activity
no longer counts as run progress or as new input. After two consecutive
successful runs with no issue-visible progress, such a wake waits for
the existing escalating cooldown (2 min, doubling, max 30 min). The
native status dispatcher (`dispatchNativeStatusWakeBatch`) does not drop
a throttled intent: it keeps the intent `queued`, reports the cooldown
deadline to the deferred-receipt worker, and dispatches it again when
the cooldown ends. Status wakes without a declared continuation (blocker
owner, notify owner, retry) are unchanged.
- Restart chaos: new enforced universal check U5 (`universal.ts`). A
task with more than 10 runs fails as `U5:run_budget_exceeded`
(`RESTART_CHAOS_U5_MAX_RUNS`). The README records the calibration.
- `doc/architecture/native-status-arbitration.md`: the decision table
and the wait rules.

## Verification

- RED on master (same reason as the waves):
`heartbeat-process-recovery.test.ts` "parks a native response_wake that
no response admitted…" fails with decision `reasonCode:
"live_continuation_registered"` and effect `{ kind:
"enqueue_continuation", continuationKind: "response_wake" }`.
`heartbeat-issue-rewake-throttle.test.ts` "throttles a
same_agent/response_wake/retry continuation…" fails 3/3 because the wake
is admitted.
- GREEN: the integration test finalizes the run, dispatches native
status wakes and runs the stranded-work sweep three times. It asserts
one run total, one wake request, no recovery action, the task
`in_progress`, and the projection "Waiting for a reply".
- `heartbeat-process-recovery.test.ts` "delays a no-progress native
continuation until the rewake cooldown…" finalizes a real `same_agent`
continuation after a no-progress run, dispatches twice (one throttle
receipt, intent stays `queued`, deadline equals `nextAllowedAt`), then
dispatches after the cooldown and gets the run. It fails when the
dispatcher drops the throttled intent.
- Focused suites pass: `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, `status-arbiter`,
`native-status-arbiter-corpus`, `runtime-mode`, `execution-projection`,
`chat-run-milestone-work`, `heartbeat-dependency-scheduling`,
`heartbeat-run-summary` (904 tests), and all restart-chaos unit tests
(584 tests). `pnpm --filter @paperclipai/server typecheck` passes.
- Universal audit over the 17 saved waves of 2026-11-11: U5 flags 12
tasks, all looping tasks (12 to 66 runs), and 0 tasks on all-pass waves.
- Chaos rerun of local warm seed 5101 (W2, W2Q, W2C): result is added in
a comment on this PR.

## Risks

- Behavioral shift: a model that reports `response_wake` to mean
"continue now" no longer gets an immediate run. The task waits visibly
for a reply. The runner prompt already describes `response_wake` as a
real response wait, and `same_agent` stays available for server-owned
continuations.
- The throttle now applies to native runs. A chain of declared
continuations with no issue-visible output (no comment, document, work
product or other activity) waits for the cooldown after the second such
run. `same_agent` continuations are server-owned (repository
preparation), so this affects only repeated no-output chains.
- Old saved state: decisions committed under `phase6-v13` keep their
recorded effects. A run that finalizes after deploy uses the new policy.
No migration.
- Budget hard-stop behavior is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:40:44 -06:00

187 lines
7.4 KiB
TypeScript

/**
* PAP-13775: throttle no-information issue re-wakes.
*
* After a process death (or any stall), external drivers — assignment pollers,
* stranded-issue reconcilers, on-demand invokes — can re-wake the same agent
* for the same issue every few seconds for as long as the issue stays
* `in_progress`. When each of those runs ends without changing any
* issue-visible state, every wake pays a full adapter session for zero new
* information (the Phase 4 interruption-recovery smoke paid 25 sessions and
* 2.4x cost for one recovery this way).
*
* This module decides when such a wake should be skipped: once an issue has
* accumulated a streak of consecutive succeeded-but-no-issue-progress runs by
* the same agent, further event-free wakes are held back for an escalating
* cooldown anchored to the last run's finish time. Fresh issue activity, an
* explicit resume, forceFreshSession, and event-carrying wake reasons bypass
* the throttle. Human comment wakes also bypass it. Agent-authored comment
* wakes deliberately stay in the normal throttle class so a cross-issue write
* cannot smuggle human wake privileges.
*
* Server-side recovery retries (process-loss retries, missing-comment
* follow-ups) insert their runs directly and never pass through this gate, so
* crash recovery stays immediate; only repeated no-op re-invocations slow
* down.
*/
/** Consecutive no-progress runs required before the cooldown engages. */
export const ISSUE_REWAKE_NO_PROGRESS_THRESHOLD = 2;
/** Cooldown after the threshold streak; doubles per additional no-progress run. */
export const ISSUE_REWAKE_BASE_COOLDOWN_MS = 120_000;
/** Upper bound for the escalating cooldown. */
export const ISSUE_REWAKE_MAX_COOLDOWN_MS = 30 * 60_000;
/** Only runs newer than this feed the streak; older history is ignored. */
export const ISSUE_REWAKE_LOOKBACK_MS = 6 * 60 * 60_000;
/** How many recent terminal runs to sample when computing the streak. */
export const ISSUE_REWAKE_RUN_SAMPLE_LIMIT = 8;
/**
* Wake reasons that assert issue state rather than deliver a new event.
* These (plus reason-less on-demand invokes) are the only wakes the throttle
* applies to; every event-shaped reason (comments, mentions, blockers
* resolved, interactions, approvals, monitors, reviews, …) passes through.
*/
export const THROTTLED_ISSUE_REWAKE_REASONS: ReadonlySet<string> = new Set([
"issue_assigned",
"issue_continuation_needed",
"issue_assignment_recovery",
"issue_graph_liveness_backstop",
]);
/**
* Activity actions that count as issue-visible progress when attributed to a
* run. Deliberately narrower than run-liveness "concrete action evidence":
* tool calls inside the workspace do not move the issue, so they do not reset
* the streak — a run must leave a comment, mutation, document, work product,
* interaction, or scheduled continuation behind.
*/
export const ISSUE_PROGRESS_ACTIVITY_ACTIONS: string[] = [
"issue.updated",
"issue.comment_added",
"issue.created",
"issue.child_created",
"issue.assigned",
"issue.released",
"issue.blockers_updated",
"issue.document_upserted",
"issue.document_updated",
"issue.document_deleted",
"issue.document_restored",
"issue.document_annotation_comment_added",
"issue.document_annotation_thread_created",
"issue.document_annotation_thread_resolved",
"issue.work_product_created",
"issue.work_product_updated",
"issue.work_product_deleted",
"issue.attachment_added",
"issue.attachment_removed",
"issue.thread_interaction_created",
"issue.monitor_scheduled",
"issue.approval_linked",
];
/**
* Activity on the issue that counts as new external input since the last run
* finished — anything a waiting agent should be woken for, including board
* responses to interactions.
*/
export const ISSUE_NEW_INPUT_ACTIVITY_ACTIONS: string[] = [
...ISSUE_PROGRESS_ACTIVITY_ACTIONS,
"issue.thread_interaction_accepted",
"issue.thread_interaction_answered",
"issue.thread_interaction_item_verdicts_submitted",
"issue.blockers_resolved_wake_emitted",
];
export interface IssueRewakeCandidateInput {
reason: string | null;
wakeCommentId: string | null;
requestedByActorType?: "user" | "agent" | "system" | null;
forceFreshSession: boolean;
hasExplicitResume: boolean;
declaredContinuationKind?: string | null;
}
export const NATIVE_STATUS_COMMITTER_ACTOR_ID = "native-status-committer";
/**
* Whether a wake is even a candidate for throttling. Wakes that carry new
* information or an explicit operator escalation always pass.
*/
export function isThrottleCandidateIssueRewake(input: IssueRewakeCandidateInput): boolean {
if (input.forceFreshSession) return false;
// Explicit resume is an operator privilege, not an actor-class escape hatch.
// Agent-authored resume comments remain subject to the normal rewake throttle.
if (input.hasExplicitResume && input.requestedByActorType !== "agent") return false;
if (input.wakeCommentId) return input.requestedByActorType === "agent";
if (input.reason === null) return true;
if (input.reason === "issue_status_changed" && input.declaredContinuationKind) return true;
return THROTTLED_ISSUE_REWAKE_REASONS.has(input.reason);
}
export interface RecentIssueRunSample {
id: string;
status: string;
finishedAt: Date | null;
}
export interface IssueRewakeThrottleInput {
now: Date;
/** Terminal runs for the same (agent, issue), newest finish first. */
recentTerminalRuns: RecentIssueRunSample[];
/** Runs among the sample that produced issue-visible progress. */
runIdsWithIssueProgress: ReadonlySet<string>;
/** New issue input landed after the newest run finished. */
hasNewIssueInputSinceLastRun: boolean;
}
export type IssueRewakeThrottleDecision =
| { blocked: false; noProgressStreak: number }
| {
blocked: true;
noProgressStreak: number;
cooldownMs: number;
lastRunFinishedAt: Date;
nextAllowedAt: Date;
};
export function computeIssueRewakeCooldownMs(noProgressStreak: number): number {
const doublings = Math.max(0, noProgressStreak - ISSUE_REWAKE_NO_PROGRESS_THRESHOLD);
// Guard the exponent so an absurd streak can't overflow into Infinity.
const factor = 2 ** Math.min(doublings, 16);
return Math.min(ISSUE_REWAKE_BASE_COOLDOWN_MS * factor, ISSUE_REWAKE_MAX_COOLDOWN_MS);
}
export function evaluateIssueRewakeThrottle(input: IssueRewakeThrottleInput): IssueRewakeThrottleDecision {
const runs = input.recentTerminalRuns;
if (runs.length === 0) return { blocked: false, noProgressStreak: 0 };
if (input.hasNewIssueInputSinceLastRun) return { blocked: false, noProgressStreak: 0 };
let noProgressStreak = 0;
for (const run of runs) {
// A failed/cancelled/interrupted run breaks the streak: its follow-up is
// recovery, not a redundant re-poll, and must not be delayed.
if (run.status !== "succeeded" || !run.finishedAt) break;
if (input.runIdsWithIssueProgress.has(run.id)) break;
noProgressStreak += 1;
}
if (noProgressStreak < ISSUE_REWAKE_NO_PROGRESS_THRESHOLD) {
return { blocked: false, noProgressStreak };
}
const lastRunFinishedAt = runs[0]?.finishedAt;
if (!lastRunFinishedAt) return { blocked: false, noProgressStreak };
const cooldownMs = computeIssueRewakeCooldownMs(noProgressStreak);
const nextAllowedAt = new Date(lastRunFinishedAt.getTime() + cooldownMs);
if (input.now.getTime() < nextAllowedAt.getTime()) {
return { blocked: true, noProgressStreak, cooldownMs, lastRunFinishedAt, nextAllowedAt };
}
return { blocked: false, noProgressStreak };
}