test(restart-chaos): add a result trigger and recovered-run comment-wake guards (#19689)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness kills the control plane at chosen moments and grades whether each task still finishes correctly > - One wave lost a user's follow-up comment after a kill that landed after the run accepted its result but before the run settled > - No existing trigger aims a fault at that moment, so the case was hard to reproduce on purpose > - The recovery path that settles such a run had no regression test that covers the saved comment > - This pull request adds a `result` trigger to the harness and a regression test for the recovery path > - The benefit is that the harness can aim faults at the result-accepted window, and a change that drops, repeats, or re-wakes a saved comment fails a test ## Linked Issues or Issue Description **What happened?** In a restart-chaos storm wave, a user posted a comment while a native run was active. A hard kill (K1) landed after the run accepted its result and before the run settled. The run settled `succeeded` and the task stayed `done`, but no follow-up run read the comment. **Steps to reproduce** The original failure did not reproduce. Three targeted waves ran 12 tasks with K1 at comment time or in the new result window. Every follow-up comment was consumed exactly once. The new regression test also passes on current `master`. The test and the trigger guard the path and do not claim a fix. **Expected behavior** A comment posted during a run is consumed exactly once after the run ends, including when restart recovery settles the run. ## What Changed - `tests/restart-chaos`: add `--triggers result` (C1 only). K1 fires 0–3 s after the run that received the follow-up proposes or accepts its result (`run.result.proposed` or `run.result.accepted`). The fault fires only while that run is still `running`: the observer checks the latest sample and rereads the runs right before dispatch. A run that settled during the delay, or a follow-up posted at rest, records a `result` miss. Add poll-driven unit tests for these cases and a README row. - `server/src/__tests__/native-run-projection-work.test.ts`: add embedded Postgres tests in which background recovery, not the executor, settles the run (projection, environment settlement, and deferred-receipt workers), with a deferred user comment and an active local lease left by the killed executor. The tests check: - The comment is promoted into exactly one follow-up run, and a second restart adds none. - A comment the run already received as a same-turn steer is not woken again. The other saved comment still is. - A queue whose only comment was steered stays cancelled, and no follow-up starts. - A Retry click does not consume the comment a second time. A click that observed the source is stale. A click that observed the follow-up joins it. ## Verification - `cd tests/restart-chaos && npx vitest run --config vitest.config.ts` (96 tests pass) - `cd server && npx vitest run src/__tests__/native-run-projection-work.test.ts` (54 tests pass) - Live, local, ACPX Claude, Codex and OpenCode: `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --workloads C1 --triggers comment --faults K1 --adapters codex,claude,opencode --tasks 4 --concurrency 2 --rc06-masked` passed 4 of 4. Two waves with `--triggers result` consumed each follow-up exactly once. One task in those waves found a separate defect: a kill 1.4 s after the result commit left the run `running` behind the dead controller's 20-minute finalization lease. A separate pull request fixes that defect. ## Risks - Low risk. These are test and harness changes only. No production code changes. - The `result` trigger depends on the observer's poll interval (about 3 s), so the exact kill point within the window varies. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 16:02:36 +01:00 · 2026-11-06 19:53:58 -06:00
1 parent 5e4d24b6c8
commit d26372dc8a
@@ -11,7 +11,7 @@ import { join } from "node:path";
 import { setTimeout as delay } from "node:timers/promises";
 import { eq, sql } from "drizzle-orm";
 import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
-import { agentWakeupRequests, agents, companies, completionContracts, createDb, heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, workspaceOperations, type Db } from "@paperclipai/db";
+import { agentWakeupRequests, agents, authUsers, companies, companyMemberships, completionContracts, createDb, environmentLeases, environments, heartbeatRuns, issueComments, issues, nativeRunFinalizations, nativeRunResults, workspaceOperations, type Db } from "@paperclipai/db";
 import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
 import { nativeRunProjectionWork } from "../services/native-runtime/native-run-projection-work.js";
 import { projectCommittedRun } from "../services/native-runtime/native-run-projection.js";
@@ -22,6 +22,9 @@ import * as statusCommitter from "../services/native-runtime/status-decision-com
 import { PaperclipControlPlanePort } from "../services/native-runtime/paperclip-control-plane-port.js";
 import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js";
 import { workspaceOperationService } from "../services/workspace-operations.js";
+import { environmentService } from "../services/environments.js";
+import { withQueuedCommentIdsInWakePayload } from "../services/issue-queued-comment-queue.js";
+import { requestTaskRetry } from "../services/task-retry.js";
 import { idleWorkSnapshot } from "../services/task-admission.js";
 import { CONTROL_PLANE_CONFORMANCE_RESULT, CONTROL_PLANE_CONFORMANCE_TERMINAL } from "../vendor/paperclip-runner/testing.js";
 import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
@@ -652,4 +655,147 @@ describe("committed native run projection lifecycle", () => {
      skipNativeResponsePresentation: true, skipNativeReviewResponse: true, onWorkspaceSettled: settled });
    expect(settled).toHaveBeenCalledTimes(1);
  });
+
+  describe("a comment deferred behind a run whose projection is left to restart recovery", () => {
+    const commenter = "projection-comment-user";
+    let busyRunId: string | undefined;
+    afterEach(async () => {
+      if (busyRunId) await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, busyRunId));
+      await db.update(agents).set({ runtimeConfig: {} }).where(eq(agents.id, agentId));
+      busyRunId = undefined;
+    });
+    async function deferredComment(binding: Awaited<ReturnType<typeof seed>>, bodies = ["Also include the FOLLOWUP line."]) {
+      await db.insert(authUsers).values({ id: commenter, name: "Commenter", email: `${commenter}@example.test`,
+        createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing();
+      await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: commenter,
+        status: "active", membershipRole: "operator" }).onConflictDoNothing();
+      // One provider slot is held by another task, so a promoted follow-up stays
+      // queued here instead of starting a provider.
+      await db.update(agents).set({ runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } }).where(eq(agents.id, agentId));
+      [{ id: busyRunId }] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running",
+        contextSnapshot: { issueId: randomUUID() } }).returning({ id: heartbeatRuns.id });
+      const comments = await db.insert(issueComments).values(bodies.map(body => ({ companyId, issueId: binding.issueId,
+        authorUserId: commenter, body }))).returning();
+      const ids = comments.map(comment => comment.id), latest = ids[ids.length - 1];
+      // The durable receipt the comment route writes while the run is active.
+      const [wake] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation",
+        triggerDetail: "system", reason: "issue_commented", status: "deferred_issue_execution",
+        requestedByActorType: "user", requestedByActorId: commenter,
+        payload: { issueId: binding.issueId, commentId: latest, _paperclipWakeContext: {
+          issueId: binding.issueId, taskId: binding.issueId, commentId: latest, wakeCommentId: latest,
+          wakeCommentIds: ids, wakeReason: "issue_commented", source: "issue.comment" } },
+      }).returning();
+      // The killed executor never reached its cleanup, so its local lease is still active.
+      const environmentId = (await db.select({ id: environments.id }).from(environments).where(eq(environments.driver, "local")))[0]?.id
+        ?? (await db.insert(environments).values({ name: "Local", driver: "local" }).returning())[0].id;
+      const lease = await environmentService(db).acquireLease({ companyId, environmentId, heartbeatRunId: binding.runId,
+        issueId: binding.issueId, leasePolicy: "ephemeral", provider: "local" });
+      return { comment: comments[0], comments, wake, lease };
+    }
+    // The durable writes of an acknowledged same-turn steer (the queued-comment
+    // steer route): the message leaves the saved queue and the live run records
+    // that its turn received it.
+    async function steer(binding: Awaited<ReturnType<typeof seed>>, wakeId: string, commentId: string) {
+      const wake = await readWake(wakeId);
+      const remaining = (wake.payload as { _paperclipWakeContext: { wakeCommentIds: string[] } })
+        ._paperclipWakeContext.wakeCommentIds.filter(id => id !== commentId);
+      await db.update(agentWakeupRequests).set({ payload: withQueuedCommentIdsInWakePayload(wake.payload, remaining) })
+        .where(eq(agentWakeupRequests.id, wakeId));
+      const run = await read(binding.runId);
+      await db.update(heartbeatRuns).set({ resultJson: { ...(run.resultJson ?? {}), queuedSteeringAcknowledgements: {
+        [commentId]: { status: "acknowledged", queueId: wakeId, turnId: "steered-turn", acknowledgedAt: new Date().toISOString() } } } })
+        .where(eq(heartbeatRuns.id, binding.runId));
+    }
+    async function startRecovered() {
+      coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => admitted,
+        onError: error => errors.push(error) });
+      const heartbeat = heartbeatService(db);
+      worker = heartbeat.registerNativeRunProjectionWork(coordinator, () => admitted);
+      const receipts = heartbeat.registerDeferredReceiptWork(coordinator, () => admitted);
+      const settlement = heartbeat.registerNativeEnvironmentSettlementWork(coordinator, () => admitted);
+      await Promise.all([worker.ready, receipts.ready, settlement.ready]);
+    }
+    const followUps = async (issueId: string, sourceRunId: string) => (await db.select().from(heartbeatRuns)
+      .where(sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${issueId}`)).filter(run => run.id !== sourceRunId);
+
+    it("promotes the comment exactly once after the recovered run is projected terminal", async () => {
+      const binding = await seed(true);
+      const { comment, wake, lease } = await deferredComment(binding);
+      await complete(binding, "control_plane"); await barrier(binding);
+      // Startup recovery runs while the killed run is still `running`: the
+      // comment correctly stays deferred behind it.
+      await startRecovered(); await flush();
+      expect(await readWake(wake.id)).toMatchObject({ status: "deferred_issue_execution", runId: null });
+      // The accepted result commits, but its executor died before projecting the
+      // run. The background projection worker settles the run instead.
+      await finalizeNativeRun({ db, runId: binding.runId, workspaceFinalizeStatus: "succeeded" });
+      await flush();
+      expect((await read(binding.runId)).status).toBe("succeeded");
+      expect((await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)))[0].status).not.toBe("active");
+      const promoted = await readWake(wake.id);
+      expect(promoted).toMatchObject({ status: "queued", runId: expect.any(String) });
+      const [followUp, ...extra] = await followUps(binding.issueId, binding.runId);
+      expect(extra).toEqual([]);
+      expect(followUp).toMatchObject({ id: promoted.runId, status: "queued" });
+      expect(followUp.contextSnapshot?.wakeCommentIds).toEqual([comment.id]);
+      // Another restart re-derives nothing new: the comment is not consumed twice.
+      await coordinator!.stop(); await startRecovered(); await flush();
+      await heartbeatService(db).resumeQueuedRuns();
+      expect((await followUps(binding.issueId, binding.runId)).map(run => run.id)).toEqual([followUp.id]);
+    });
+
+    it("does not wake again for a comment the recovered run already received as a steer", async () => {
+      const binding = await seed(true);
+      const { comments: [steered, saved], wake } = await deferredComment(binding,
+        ["Steer this into the live turn.", "Also include the FOLLOWUP line."]);
+      await steer(binding, wake.id, steered.id);
+      await complete(binding, "control_plane"); await barrier(binding);
+      await startRecovered(); await flush();
+      await finalizeNativeRun({ db, runId: binding.runId, workspaceFinalizeStatus: "succeeded" });
+      await flush();
+      const [followUp, ...extra] = await followUps(binding.issueId, binding.runId);
+      expect(extra).toEqual([]);
+      expect(followUp.contextSnapshot?.wakeCommentIds).toEqual([saved.id]);
+      expect((await read(binding.runId)).resultJson?.queuedSteeringAcknowledgements).toMatchObject({
+        [steered.id]: { status: "acknowledged" } });
+    });
+
+    it("cancels the saved queue and starts no follow-up when its only comment was steered", async () => {
+      const binding = await seed(true);
+      const { comment, wake } = await deferredComment(binding);
+      // The steer route cancels a queue whose final message was delivered.
+      await steer(binding, wake.id, comment.id);
+      await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() })
+        .where(eq(agentWakeupRequests.id, wake.id));
+      await complete(binding, "control_plane"); await barrier(binding);
+      await startRecovered(); await flush();
+      await finalizeNativeRun({ db, runId: binding.runId, workspaceFinalizeStatus: "succeeded" });
+      await flush(); await heartbeatService(db).resumeQueuedRuns();
+      expect((await read(binding.runId)).status).toBe("succeeded");
+      expect(await followUps(binding.issueId, binding.runId)).toEqual([]);
+      expect(await readWake(wake.id)).toMatchObject({ status: "cancelled", runId: null });
+    });
+
+    it("joins a Retry click to the promoted follow-up instead of consuming the comment twice", async () => {
+      const binding = await seed(true);
+      const { comment, wake } = await deferredComment(binding);
+      await complete(binding, "control_plane"); await barrier(binding);
+      await startRecovered(); await flush();
+      await finalizeNativeRun({ db, runId: binding.runId, workspaceFinalizeStatus: "succeeded" });
+      await flush();
+      const [followUp] = await followUps(binding.issueId, binding.runId);
+      expect(followUp).toMatchObject({ id: (await readWake(wake.id)).runId, status: "queued" });
+      // A click that observed the recovered source is stale; one that observed
+      // the follow-up joins it. Neither admits another run with the comment.
+      expect(await requestTaskRetry({ db, companyId, issueId: binding.issueId, actorId: commenter,
+        idempotencyKey: randomUUID(), observedRunId: binding.runId })).toMatchObject({ accepted: false, stale: true });
+      expect(await requestTaskRetry({ db, companyId, issueId: binding.issueId, actorId: commenter,
+        idempotencyKey: randomUUID(), observedRunId: followUp.id })).toMatchObject({ accepted: true });
+      await flush(); await heartbeatService(db).resumeQueuedRuns();
+      const runs = await followUps(binding.issueId, binding.runId);
+      expect(runs.map(run => run.id)).toEqual([followUp.id]);
+      expect(runs.filter(run => (run.contextSnapshot?.wakeCommentIds as string[] | undefined)?.includes(comment.id)))
+        .toHaveLength(1);
+    });
+  });
 });