mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run ends with a structured result. The status arbiter
turns that result into an issue status and next actions.
> - A result can report `yielded` with `continuation.kind:
"response_wake"`. This means "wake me when somebody responds".
> - Only a direct Board comment gave this kind a passive wait. Every
other `response_wake` became an immediate `issue_status_changed` wake to
the same agent.
> - The agent woke, saw nothing new, reported `response_wake` again, and
woke again. Nothing bounded this loop. It spent provider budget and its
back-to-back runs caused other failures on warm sandboxes.
> - This pull request parks such a result as a visible wait, and puts
declared continuation wakes under the rewake throttle as a backstop.
> - The benefit is that an agent that waits for input stops and costs
nothing until a real response arrives.
## Linked Issues or Issue Description
**What happened?**
Restart-chaos waves on master `683fd1e9a5` (no faults injected) showed
one task run 30 to 66 times, about 12 s apart. Example: wave
`w-20261111T190352-s5101`, task t02 (W2C), 65 runs in 13 minutes. Each
run reported `yielded` with `continuation.kind: "response_wake"` and the
summary "No changes made. This status-change wake does not trigger phase
2". Each run was woken by `issue_status_changed`, and each finalization
committed `live_continuation_registered` with an `enqueue_continuation`
effect of kind `response_wake`.
Root cause: `arbitrateNativeStatus()`
(`server/src/services/native-runtime/status-arbiter.ts`) only treats
`response_wake` as passive when `boardResponseWaitAuthorized` (from
`readNativeBoardResponseWaitSource`), external chat, or plan wait is
true. Otherwise it falls to the generic yielded branch, and
`status-decision-committer.ts` (`enqueue_continuation`) enqueues an
`issue_status_changed` wake to the same agent. `issue_status_changed` is
not in `THROTTLED_ISSUE_REWAKE_REASONS`, and the native committer writes
an `issue.updated` activity row with the run's id for every decision, so
the rewake throttle saw every native run as progress.
**Expected behavior**
A `response_wake` with no admitted response source parks the task. A
comment, an interaction answer or a status change by somebody else wakes
the agent. No path can loop without a bound.
**Steps to reproduce**
Run `tests/restart-chaos/main.ts --env local --adapters codex
--workloads W2,W2Q,W2C --concurrency 10 --tasks 10 --seed 5101
--lifecycle warm --triggers time`. W2/W2C tasks loop after phase 1.
## What Changed
- `status-arbiter.ts`: a `yielded` / `response_wake` result with no
admitted response source and no blocking remaining work now commits
`response_wake_waiting_for_input`, keeps the task `in_progress`, and has
no effects. Policy version is `phase6-v14`. Board, external chat, plan,
governance, dependency, pause and unfinished-work branches are unchanged
and still come first.
- `native-runtime/native-response-wake-wait.ts` (new) and
`recovery/service.ts`: recovery treats that applied decision as a
durable wait while it is the issue's latest decision, the issue is
`in_progress` with the same agent assignee, and no user comment arrived
after the run started. This applies to the stranded-work sweep and to
the paused-agent passive-wait check.
- `execution-projection.ts`: a committed run with this reason shows the
phase `waiting_for_answer`, label "Waiting for a reply", and a next
action that tells the user to comment or change the status. This shows
only while that decision is still the issue's latest decision and the
issue is `in_progress`; an older run shows "Completed".
- Backstop in `issue-rewake-throttle.ts` and `heartbeat/queue.ts`: an
`issue_status_changed` wake that carries a declared `continuationKind`
(the `enqueue_continuation` effect) is now a throttle candidate. The
native status committer's own `issue.updated` / status-decision activity
no longer counts as run progress or as new input. After two consecutive
successful runs with no issue-visible progress, such a wake waits for
the existing escalating cooldown (2 min, doubling, max 30 min). The
native status dispatcher (`dispatchNativeStatusWakeBatch`) does not drop
a throttled intent: it keeps the intent `queued`, reports the cooldown
deadline to the deferred-receipt worker, and dispatches it again when
the cooldown ends. Status wakes without a declared continuation (blocker
owner, notify owner, retry) are unchanged.
- Restart chaos: new enforced universal check U5 (`universal.ts`). A
task with more than 10 runs fails as `U5:run_budget_exceeded`
(`RESTART_CHAOS_U5_MAX_RUNS`). The README records the calibration.
- `doc/architecture/native-status-arbitration.md`: the decision table
and the wait rules.
## Verification
- RED on master (same reason as the waves):
`heartbeat-process-recovery.test.ts` "parks a native response_wake that
no response admitted…" fails with decision `reasonCode:
"live_continuation_registered"` and effect `{ kind:
"enqueue_continuation", continuationKind: "response_wake" }`.
`heartbeat-issue-rewake-throttle.test.ts` "throttles a
same_agent/response_wake/retry continuation…" fails 3/3 because the wake
is admitted.
- GREEN: the integration test finalizes the run, dispatches native
status wakes and runs the stranded-work sweep three times. It asserts
one run total, one wake request, no recovery action, the task
`in_progress`, and the projection "Waiting for a reply".
- `heartbeat-process-recovery.test.ts` "delays a no-progress native
continuation until the rewake cooldown…" finalizes a real `same_agent`
continuation after a no-progress run, dispatches twice (one throttle
receipt, intent stays `queued`, deadline equals `nextAllowedAt`), then
dispatches after the cooldown and gets the run. It fails when the
dispatcher drops the throttled intent.
- Focused suites pass: `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, `status-arbiter`,
`native-status-arbiter-corpus`, `runtime-mode`, `execution-projection`,
`chat-run-milestone-work`, `heartbeat-dependency-scheduling`,
`heartbeat-run-summary` (904 tests), and all restart-chaos unit tests
(584 tests). `pnpm --filter @paperclipai/server typecheck` passes.
- Universal audit over the 17 saved waves of 2026-11-11: U5 flags 12
tasks, all looping tasks (12 to 66 runs), and 0 tasks on all-pass waves.
- Chaos rerun of local warm seed 5101 (W2, W2Q, W2C): result is added in
a comment on this PR.
## Risks
- Behavioral shift: a model that reports `response_wake` to mean
"continue now" no longer gets an immediate run. The task waits visibly
for a reply. The runner prompt already describes `response_wake` as a
real response wait, and `same_agent` stays available for server-owned
continuations.
- The throttle now applies to native runs. A chain of declared
continuations with no issue-visible output (no comment, document, work
product or other activity) waits for the cooldown after the second such
run. `same_agent` continuations are server-owned (repository
preparation), so this affects only repeated no-output chains.
- Old saved state: decisions committed under `phase6-v13` keep their
recorded effects. A run that finalizes after deploy uses the new policy.
No migration.
- Budget hard-stop behavior is unchanged.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
187 lines
7.4 KiB
TypeScript
187 lines
7.4 KiB
TypeScript
/**
|
|
* PAP-13775: throttle no-information issue re-wakes.
|
|
*
|
|
* After a process death (or any stall), external drivers — assignment pollers,
|
|
* stranded-issue reconcilers, on-demand invokes — can re-wake the same agent
|
|
* for the same issue every few seconds for as long as the issue stays
|
|
* `in_progress`. When each of those runs ends without changing any
|
|
* issue-visible state, every wake pays a full adapter session for zero new
|
|
* information (the Phase 4 interruption-recovery smoke paid 25 sessions and
|
|
* 2.4x cost for one recovery this way).
|
|
*
|
|
* This module decides when such a wake should be skipped: once an issue has
|
|
* accumulated a streak of consecutive succeeded-but-no-issue-progress runs by
|
|
* the same agent, further event-free wakes are held back for an escalating
|
|
* cooldown anchored to the last run's finish time. Fresh issue activity, an
|
|
* explicit resume, forceFreshSession, and event-carrying wake reasons bypass
|
|
* the throttle. Human comment wakes also bypass it. Agent-authored comment
|
|
* wakes deliberately stay in the normal throttle class so a cross-issue write
|
|
* cannot smuggle human wake privileges.
|
|
*
|
|
* Server-side recovery retries (process-loss retries, missing-comment
|
|
* follow-ups) insert their runs directly and never pass through this gate, so
|
|
* crash recovery stays immediate; only repeated no-op re-invocations slow
|
|
* down.
|
|
*/
|
|
|
|
/** Consecutive no-progress runs required before the cooldown engages. */
|
|
export const ISSUE_REWAKE_NO_PROGRESS_THRESHOLD = 2;
|
|
|
|
/** Cooldown after the threshold streak; doubles per additional no-progress run. */
|
|
export const ISSUE_REWAKE_BASE_COOLDOWN_MS = 120_000;
|
|
|
|
/** Upper bound for the escalating cooldown. */
|
|
export const ISSUE_REWAKE_MAX_COOLDOWN_MS = 30 * 60_000;
|
|
|
|
/** Only runs newer than this feed the streak; older history is ignored. */
|
|
export const ISSUE_REWAKE_LOOKBACK_MS = 6 * 60 * 60_000;
|
|
|
|
/** How many recent terminal runs to sample when computing the streak. */
|
|
export const ISSUE_REWAKE_RUN_SAMPLE_LIMIT = 8;
|
|
|
|
/**
|
|
* Wake reasons that assert issue state rather than deliver a new event.
|
|
* These (plus reason-less on-demand invokes) are the only wakes the throttle
|
|
* applies to; every event-shaped reason (comments, mentions, blockers
|
|
* resolved, interactions, approvals, monitors, reviews, …) passes through.
|
|
*/
|
|
export const THROTTLED_ISSUE_REWAKE_REASONS: ReadonlySet<string> = new Set([
|
|
"issue_assigned",
|
|
"issue_continuation_needed",
|
|
"issue_assignment_recovery",
|
|
"issue_graph_liveness_backstop",
|
|
]);
|
|
|
|
/**
|
|
* Activity actions that count as issue-visible progress when attributed to a
|
|
* run. Deliberately narrower than run-liveness "concrete action evidence":
|
|
* tool calls inside the workspace do not move the issue, so they do not reset
|
|
* the streak — a run must leave a comment, mutation, document, work product,
|
|
* interaction, or scheduled continuation behind.
|
|
*/
|
|
export const ISSUE_PROGRESS_ACTIVITY_ACTIONS: string[] = [
|
|
"issue.updated",
|
|
"issue.comment_added",
|
|
"issue.created",
|
|
"issue.child_created",
|
|
"issue.assigned",
|
|
"issue.released",
|
|
"issue.blockers_updated",
|
|
"issue.document_upserted",
|
|
"issue.document_updated",
|
|
"issue.document_deleted",
|
|
"issue.document_restored",
|
|
"issue.document_annotation_comment_added",
|
|
"issue.document_annotation_thread_created",
|
|
"issue.document_annotation_thread_resolved",
|
|
"issue.work_product_created",
|
|
"issue.work_product_updated",
|
|
"issue.work_product_deleted",
|
|
"issue.attachment_added",
|
|
"issue.attachment_removed",
|
|
"issue.thread_interaction_created",
|
|
"issue.monitor_scheduled",
|
|
"issue.approval_linked",
|
|
];
|
|
|
|
/**
|
|
* Activity on the issue that counts as new external input since the last run
|
|
* finished — anything a waiting agent should be woken for, including board
|
|
* responses to interactions.
|
|
*/
|
|
export const ISSUE_NEW_INPUT_ACTIVITY_ACTIONS: string[] = [
|
|
...ISSUE_PROGRESS_ACTIVITY_ACTIONS,
|
|
"issue.thread_interaction_accepted",
|
|
"issue.thread_interaction_answered",
|
|
"issue.thread_interaction_item_verdicts_submitted",
|
|
"issue.blockers_resolved_wake_emitted",
|
|
];
|
|
|
|
export interface IssueRewakeCandidateInput {
|
|
reason: string | null;
|
|
wakeCommentId: string | null;
|
|
requestedByActorType?: "user" | "agent" | "system" | null;
|
|
forceFreshSession: boolean;
|
|
hasExplicitResume: boolean;
|
|
declaredContinuationKind?: string | null;
|
|
}
|
|
|
|
export const NATIVE_STATUS_COMMITTER_ACTOR_ID = "native-status-committer";
|
|
|
|
/**
|
|
* Whether a wake is even a candidate for throttling. Wakes that carry new
|
|
* information or an explicit operator escalation always pass.
|
|
*/
|
|
export function isThrottleCandidateIssueRewake(input: IssueRewakeCandidateInput): boolean {
|
|
if (input.forceFreshSession) return false;
|
|
// Explicit resume is an operator privilege, not an actor-class escape hatch.
|
|
// Agent-authored resume comments remain subject to the normal rewake throttle.
|
|
if (input.hasExplicitResume && input.requestedByActorType !== "agent") return false;
|
|
if (input.wakeCommentId) return input.requestedByActorType === "agent";
|
|
if (input.reason === null) return true;
|
|
if (input.reason === "issue_status_changed" && input.declaredContinuationKind) return true;
|
|
return THROTTLED_ISSUE_REWAKE_REASONS.has(input.reason);
|
|
}
|
|
|
|
export interface RecentIssueRunSample {
|
|
id: string;
|
|
status: string;
|
|
finishedAt: Date | null;
|
|
}
|
|
|
|
export interface IssueRewakeThrottleInput {
|
|
now: Date;
|
|
/** Terminal runs for the same (agent, issue), newest finish first. */
|
|
recentTerminalRuns: RecentIssueRunSample[];
|
|
/** Runs among the sample that produced issue-visible progress. */
|
|
runIdsWithIssueProgress: ReadonlySet<string>;
|
|
/** New issue input landed after the newest run finished. */
|
|
hasNewIssueInputSinceLastRun: boolean;
|
|
}
|
|
|
|
export type IssueRewakeThrottleDecision =
|
|
| { blocked: false; noProgressStreak: number }
|
|
| {
|
|
blocked: true;
|
|
noProgressStreak: number;
|
|
cooldownMs: number;
|
|
lastRunFinishedAt: Date;
|
|
nextAllowedAt: Date;
|
|
};
|
|
|
|
export function computeIssueRewakeCooldownMs(noProgressStreak: number): number {
|
|
const doublings = Math.max(0, noProgressStreak - ISSUE_REWAKE_NO_PROGRESS_THRESHOLD);
|
|
// Guard the exponent so an absurd streak can't overflow into Infinity.
|
|
const factor = 2 ** Math.min(doublings, 16);
|
|
return Math.min(ISSUE_REWAKE_BASE_COOLDOWN_MS * factor, ISSUE_REWAKE_MAX_COOLDOWN_MS);
|
|
}
|
|
|
|
export function evaluateIssueRewakeThrottle(input: IssueRewakeThrottleInput): IssueRewakeThrottleDecision {
|
|
const runs = input.recentTerminalRuns;
|
|
if (runs.length === 0) return { blocked: false, noProgressStreak: 0 };
|
|
if (input.hasNewIssueInputSinceLastRun) return { blocked: false, noProgressStreak: 0 };
|
|
|
|
let noProgressStreak = 0;
|
|
for (const run of runs) {
|
|
// A failed/cancelled/interrupted run breaks the streak: its follow-up is
|
|
// recovery, not a redundant re-poll, and must not be delayed.
|
|
if (run.status !== "succeeded" || !run.finishedAt) break;
|
|
if (input.runIdsWithIssueProgress.has(run.id)) break;
|
|
noProgressStreak += 1;
|
|
}
|
|
|
|
if (noProgressStreak < ISSUE_REWAKE_NO_PROGRESS_THRESHOLD) {
|
|
return { blocked: false, noProgressStreak };
|
|
}
|
|
|
|
const lastRunFinishedAt = runs[0]?.finishedAt;
|
|
if (!lastRunFinishedAt) return { blocked: false, noProgressStreak };
|
|
|
|
const cooldownMs = computeIssueRewakeCooldownMs(noProgressStreak);
|
|
const nextAllowedAt = new Date(lastRunFinishedAt.getTime() + cooldownMs);
|
|
if (input.now.getTime() < nextAllowedAt.getTime()) {
|
|
return { blocked: true, noProgressStreak, cooldownMs, lastRunFinishedAt, nextAllowedAt };
|
|
}
|
|
return { blocked: false, noProgressStreak };
|
|
}
|