mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:10:00 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A person can add a comment to a task while the agent's run on that
task is still working
> - #21175 made a `get_task_history` read of a queued comment's current
revision count as delivery
> - A read is not handling. In `agent-chat.multiple-handoffs`, Morgan's
running turn read the "Launch sooner" comment, set it aside, and saved
"Launch later". No continuation ran, so the instruction was lost.
> - This pull request counts a read as delivery only when the run's
accepted result cites the comment as handled. Otherwise one continuation
still delivers it, and that continuation is told the comment may already
be applied.
> - The benefit is that a person's comment is never silently lost, and
the #21175 protection against repeated completed work stays in place
## Linked Issue or Issue Description
Refs #21175
Refs #21203
Refs #21210
**What happened?**
Paid run 43321773136 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`), case
`agent-chat.multiple-handoffs`. Morgan's running turn called
`get_task_history` and received the CEO's queued "Launch sooner" comment
at its current revision (`task_history.read` at seq 75). The model
judged the comment as an unauthenticated agent instruction and saved
"Launch later" from the brief. Its accepted result did not mention the
comment. `deliveredQueuedCommentIds` counted the read as delivery, the
release drain dropped the held comment, and no continuation ran.
**Expected behavior**
A read counts as delivery only when the run records the comment as
handled. A comment that was read but not recorded as handled is
delivered once in a new turn. The new turn's prompt says that an earlier
turn already read the comment and may have applied it.
**Steps to reproduce**
1. Start a native run on a task.
2. Queue a comment behind the run.
3. Let the run read `get_task_history` and finish without applying the
comment.
4. Before this change, no continuation runs and the comment is lost.
## What Changed
- `server/src/services/queued-comment-delivery.ts`: the history-read
delivery rule now also requires the reading run's `run.result.accepted`
event to cite `comment:<id>` (top-level `evidence[].ref` or a
criterion's `evidenceRefs`, prefix match so an annotated ref counts).
The read revision, succeeded-run and finished-after-revision rules are
unchanged. All consumers (release drain, stale-continuation save,
ownership reconciler, first-turn queued input, remote-Stop resume) use
this one rule.
- New `readByEarlierTurnCommentIds`, `commentsArrivedDuringRun` and
`taskHistoryHandlingGuidance`.
- `get_task_history` (both tool authorities) returns
`arrivedDuringThisTurn` (the `comment:<id>` refs of comments other
authors posted or edited after the run started) and a `handling`
instruction to cite each applied comment in `paperclip_finish` /
`paperclip_block` evidence. The comment list itself is unchanged. A
failure to compute the list returns no guidance, so the comment stays
owed.
- `buildPaperclipWakePayload` marks a wake comment `readByEarlierTurn`
when another finished run of the agent read its current revision.
`renderPaperclipWakePrompt` adds one line after that comment: check the
current task state first, and do not repeat work that is already done.
- `doc/run-log-events.md`: the delivery rule and the new fields.
- Evals: `agent-chat.multiple-handoffs` (oracle
`runtime-multiple-handoffs.v6`) and `steering.edit-queued-message`
(oracle `edit-queued-message.structural.v6`) accept in-turn consumption
only with a current-revision read and a citation. A read without a
citation must get exactly one continuation whose saved wake marks
`readByEarlierTurn`, and the final saved revision must hold the change
("Launch sooner"; blue). `runtime-queue.ts` reads citations from the
run's `run.result.accepted` event. No scenario YAML changed.
## Verification
- RED on master `b1ef1f3022`,
`server/src/__tests__/heartbeat-held-comment-wake.test.ts`: 10 of 41
fail. The key one: "settles a held message as soon as its run closes the
task (read mid-turn: true, cited: false)" delivered 0, expected 1.
- GREEN: 41/41. The three required shapes:
- read, not cited: one continuation, and its wake marks
`readByEarlierTurn` (`...read mid-turn: true, cited: false` and the
stale-queued-run variant).
- read and cited: no continuation (`...cited: true`).
- never read: one continuation, as before.
- Edited after the read, or a failed reader: still owed.
- `server/src/services/queued-comment-history-read.test.ts` (remote-Stop
resume): succeeded+cited not owed; succeeded+uncited owed; failed+cited
owed. 4/4.
- `packages/adapter-utils/src/server-utils.test.ts`: the prompt line
renders only for `readByEarlierTurn`.
- Neighbouring suites: `muse-control`, `dot-runner`,
`paperclip-runner-tool-authority`, `paperclip-control-plane-port`,
`muse-native-follow-up`, `runner-prp-coordinator`,
`native-session-handoff`, `execution-continuation-history-read`,
`heartbeat-comment-wake-batching`, `ownership-reconciliation-report`,
`run-preparation`: all pass. Server and adapter-utils `tsc --noEmit`:
clean.
- Eval graders: `npx vitest run --config
tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog
tests/runner-e2e/chat-catalog-domains`: 283 files pass. New calibration:
the observed lost-instruction case (read, not cited, no continuation)
fails; read-not-cited with one warned continuation passes; an unwarned
continuation, a missing final change, and a cited read plus a
continuation fail.
- `pnpm test:e2e:runner:typecheck`: pass. `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10/10.
`node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`:
`changed: false`.
- Paid run: Runner Full-Stack E2E 43338695850 on throwaway branch
`chat-eval/qa-mr-paid` (this change plus the model-retry fix on master
19f8ed1ea2). Result to follow in a comment.
## Risks
- A run that applies a queued comment but does not cite it gets one more
turn on that task. That turn is told the comment may already be applied.
In the restart-chaos t005 shape (#21175) this can reopen a done task
when the agent does not cite the comment. This is the chosen tradeoff: a
lost comment is worse than one extra checked turn.
- Each `get_task_history` call adds one small indexed read of the run's
`started_at`.
- Upgrade-safe: runs saved by older builds have no citation, so a
read-only delivery from before this change stays owed and is delivered
once more. No migration.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
898 lines
67 KiB
TypeScript
898 lines
67 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { mkdtemp, rm } from "node:fs/promises";
|
|
import { tmpdir } from "node:os";
|
|
import path from "node:path";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
|
import { activityLog, agents, agentWakeupRequests, authUsers, budgetPolicies, companies, companyMemberships, costEvents, createDb, heartbeatRunEvents,
|
|
heartbeatRuns, issueComments, issues } from "@paperclipai/db";
|
|
import { accountRunCost } from "../services/run-cost-accounting.js";
|
|
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../services/delivery-work-notifications.js";
|
|
import { requestTaskRetry } from "../services/task-retry.js";
|
|
import { queueIssueAssignmentWakeup } from "../services/issue-assignment-wakeup.js";
|
|
import { registerServerAdapter, unregisterServerAdapter } from "../adapters/index.js";
|
|
import { heartbeatService } from "../services/heartbeat.js";
|
|
import { createOwnershipReportState, runOwnershipReconciliationPass } from "../services/recovery/ownership-invariant-report.js";
|
|
import { applyOwnershipReportBackstops } from "../services/native-runtime/native-finalization-reconciler.js";
|
|
import { promoteDeferredWorkIntents, TEMPORARY_REFUSAL_TRIGGERS, WORK_INTENT_TERMINAL_OUTCOMES } from "../services/work-intent-outbox.js";
|
|
import { deliveredQueuedCommentIds } from "../services/queued-comment-delivery.js";
|
|
import { PaperclipRunnerToolAuthority } from "../services/native-runtime/paperclip-runner-tool-authority.js";
|
|
import { recoveryPostgresSupport, startRecoveryPostgres } from "./helpers/recovery-postgres.js";
|
|
|
|
const describePostgres = recoveryPostgresSupport.supported ? describe : describe.skip;
|
|
if (!recoveryPostgresSupport.supported) console.warn(`Skipping held comment wake tests: ${recoveryPostgresSupport.reason}`);
|
|
|
|
/**
|
|
* RC-37: a follow-up comment saved behind a run is promoted when that run ends.
|
|
* Another run's unfinished accounting used to hold company admission and
|
|
* refuse it. Since Phase 3 accounting never gates admission (invariant 3), and
|
|
* each saved intent is delivered exactly once (invariant 4).
|
|
*/
|
|
describePostgres("saved intents while another run's accounting is pending", () => {
|
|
let temporary: Awaited<ReturnType<typeof startRecoveryPostgres>>;
|
|
let db: ReturnType<typeof createDb>;
|
|
let heartbeat: ReturnType<typeof heartbeatService>;
|
|
const adapterType = "codex_local";
|
|
const dispatched: Array<{ runId: string; context: Record<string, unknown> }> = [];
|
|
let duringTurn: ((runId: string) => Promise<void>) | null = null;
|
|
const oldHome = process.env.PAPERCLIP_HOME;
|
|
let home: string;
|
|
|
|
beforeAll(async () => {
|
|
temporary = await startRecoveryPostgres("heartbeat-held-comment-");
|
|
db = createDb(temporary.connectionString);
|
|
home = await mkdtemp(path.join(tmpdir(), "paperclip-held-comment-"));
|
|
process.env.PAPERCLIP_HOME = home;
|
|
heartbeat = heartbeatService(db);
|
|
registerServerAdapter({
|
|
type: adapterType, supportsLocalAgentJwt: false,
|
|
execute: async input => {
|
|
dispatched.push({ runId: input.runId, context: input.context as Record<string, unknown> });
|
|
await duringTurn?.(input.runId);
|
|
// The agent finishes the task, so no disposition repair follows.
|
|
await db.update(issues).set({ status: "done" }).where(eq(issues.id, String(input.context.issueId)));
|
|
return { exitCode: 0, signal: null, timedOut: false, resultJson: {} };
|
|
},
|
|
testEnvironment: async () => ({ adapterType, status: "pass", checks: [], testedAt: new Date().toISOString() }),
|
|
});
|
|
}, 240_000);
|
|
|
|
afterAll(async () => {
|
|
await heartbeat?.drainActiveRunExecutions();
|
|
unregisterServerAdapter(adapterType);
|
|
await db?.$client.end({ timeout: 0 });
|
|
await temporary?.cleanup();
|
|
if (oldHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = oldHome;
|
|
if (home) await rm(home, { recursive: true, force: true });
|
|
});
|
|
|
|
/** A task whose run just ended with a user comment saved behind it, in a
|
|
* company that holds new work while another run's accounting settles. */
|
|
async function seedHeldComment(options: { hardStopReached?: boolean } = {}) {
|
|
const companyId = randomUUID(), agentId = randomUUID(), otherAgentId = randomUUID(), issueId = randomUUID();
|
|
const userId = `comment-user-${companyId}`;
|
|
await db.insert(authUsers).values({ id: userId, name: "Commenter", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
|
|
await db.insert(companies).values({ id: companyId, name: "Held comment fixture", issuePrefix: `C${companyId.slice(0, 6)}` });
|
|
await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" });
|
|
await db.insert(agents).values([
|
|
{ id: agentId, companyId, name: "Assignee", adapterType, status: "idle", adapterConfig: {} },
|
|
{ id: otherAgentId, companyId, name: "Earlier worker", adapterType, status: "idle", adapterConfig: {} },
|
|
]);
|
|
await db.insert(budgetPolicies).values({ companyId, scopeType: "company", scopeId: companyId, windowKind: "calendar_month_utc",
|
|
amount: options.hardStopReached ? 1 : 2_000, hardStopEnabled: true, unpricedUsagePolicy: "allow" });
|
|
const [pending] = await db.insert(heartbeatRuns).values({ companyId, agentId: otherAgentId, invocationSource: "automation",
|
|
triggerDetail: "system", status: "succeeded", startedAt: new Date(Date.now() - 60_000), finishedAt: new Date(),
|
|
costAccountingPending: true, contextSnapshot: {},
|
|
usageJson: { costUsd: 0.01, usageSource: "per_run", billingType: "metered_api", provider: "openai", model: "gpt" } }).returning();
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Task with a follow-up comment", status: "in_progress",
|
|
assigneeAgentId: agentId, responsibleUserId: userId, issueNumber: 1 });
|
|
const [source] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "assignment",
|
|
triggerDetail: "system", status: "succeeded", startedAt: new Date(Date.now() - 30_000), finishedAt: new Date(),
|
|
contextSnapshot: { issueId, wakeReason: "issue_assigned" } }).returning();
|
|
const [comment] = await db.insert(issueComments).values({ companyId, issueId, authorUserId: userId,
|
|
body: "Also include the FOLLOWUP line." }).returning();
|
|
// The durable receipt the comment route writes while the source run is active.
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation",
|
|
triggerDetail: "system", reason: "issue_commented", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: userId,
|
|
payload: { issueId, commentId: comment.id, mutation: "comment", _paperclipWakeContext: {
|
|
issueId, taskId: issueId, commentId: comment.id, wakeCommentId: comment.id, wakeCommentIds: [comment.id],
|
|
wakeReason: "issue_commented", source: "issue.comment" } },
|
|
}).returning();
|
|
return { companyId, agentId, issueId, userId, pendingRunId: pending.id, sourceRunId: source.id, commentId: comment.id, wakeId: wake.id };
|
|
}
|
|
|
|
const issueRuns = (f: { companyId: string; issueId: string }) => db.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, f.companyId), sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${f.issueId}`));
|
|
const readWake = async (id: string) => (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, id)))[0];
|
|
const deliveries = (commentId: string) => dispatched.filter(turn =>
|
|
Array.isArray(turn.context.wakeCommentIds) && (turn.context.wakeCommentIds as string[]).includes(commentId));
|
|
|
|
/** Settle the other run's accounting through the production path, and
|
|
* return which passive owners it woke. Held work waits on these wakes. */
|
|
async function settleAccounting(f: { pendingRunId: string }) {
|
|
const woken = new Set<string>(), owner = db;
|
|
const stops = [DELIVERY_QUEUES.queuedRunDispatch, DELIVERY_QUEUES.deferredExecutionReceipts, DELIVERY_QUEUES.strandedAssignments]
|
|
.map(queue => subscribeDeliveryWork(owner, queue, () => woken.add(queue)));
|
|
try {
|
|
expect(await accountRunCost(db, f.pendingRunId)).toBe(true);
|
|
await new Promise(resolve => setTimeout(resolve, 200));
|
|
} finally { for (const stop of stops) stop(); }
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.pendingRunId)))[0].costAccountingPending).toBe(false);
|
|
return woken;
|
|
}
|
|
|
|
/** The agent's run closed the task, as a native status decision records it. */
|
|
const closeByAgent = (f: { companyId: string; issueId: string; agentId: string }, actorType = "agent") =>
|
|
db.insert(activityLog).values({ companyId: f.companyId, actorType, actorId: actorType === "agent" ? f.agentId : "person",
|
|
action: "issue.updated", entityType: "issue", entityId: f.issueId,
|
|
details: { source: "native_status_decision", fromStatus: "in_progress", toStatus: "done" } });
|
|
|
|
/** The comment route's `issue.comment_added` record, which carries the comment's own resume intent. */
|
|
const commentAdded = (f: { companyId: string; issueId: string }, agentId: string, commentId: string, resumeIntent: boolean) =>
|
|
db.insert(activityLog).values({ companyId: f.companyId, actorType: "agent", actorId: agentId, agentId,
|
|
action: "issue.comment_added", entityType: "issue", entityId: f.issueId,
|
|
details: { commentId, ...(resumeIntent ? { resumeIntent: true, followUpRequested: true } : {}) } });
|
|
|
|
async function settle(f: { companyId: string }) {
|
|
await heartbeat.resumeQueuedRuns();
|
|
await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.companyId });
|
|
await heartbeat.drainActiveRunExecutions();
|
|
}
|
|
|
|
// Phase 3 (invariant 3): another run's unfinished accounting never holds
|
|
// admission. Each intent that used to wait out the accounting hold is now
|
|
// delivered at once, exactly once, and settling accounting later never
|
|
// delivers it again. Before Phase 3 every one of these waited.
|
|
it("delivers a saved comment once while another run's accounting is still pending", async () => {
|
|
const f = await seedHeldComment();
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(1);
|
|
expect((await issueRuns(f)).filter(run => run.status === "cancelled")).toEqual([]);
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.pendingRunId)))[0].costAccountingPending).toBe(true);
|
|
await settleAccounting(f);
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
it("admits a comment posted during pending accounting at enqueue and delivers it once", async () => {
|
|
// s7532/t06: the enqueue gate refused the comment during the hold.
|
|
const f = await seedHeldComment();
|
|
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
|
authorUserId: f.userId, body: "Posted while accounting is pending." }).returning();
|
|
await heartbeat.wakeup(f.agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented",
|
|
payload: { issueId: f.issueId, commentId: comment.id, mutation: "comment" },
|
|
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: comment.id, wakeCommentId: comment.id,
|
|
source: "issue.comment", wakeReason: "issue_commented" },
|
|
requestedByActorType: "user", requestedByActorId: f.userId });
|
|
const wakes = (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId)))
|
|
.filter(wake => (wake.payload as { commentId?: string }).commentId === comment.id);
|
|
expect(wakes.map(wake => wake.status)).not.toContain("skipped");
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(comment.id)).toHaveLength(1);
|
|
await settleAccounting(f);
|
|
await settle(f);
|
|
expect(deliveries(comment.id)).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
it("starts a Retry clicked while another run's accounting is pending exactly once", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "adapter_failed" }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
const retry = await requestTaskRetry({ db, companyId: f.companyId, issueId: f.issueId, actorId: f.userId,
|
|
idempotencyKey: randomUUID(), observedRunId: f.sourceRunId });
|
|
expect(retry).toMatchObject({ accepted: true });
|
|
await settle(f); await settle(f);
|
|
const started = () => issueRuns(f).then(runs => runs.filter(run => run.id !== f.sourceRunId && run.startedAt));
|
|
expect(await started()).toHaveLength(1);
|
|
await settleAccounting(f);
|
|
await settle(f);
|
|
expect(await started()).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
it("starts a new task assigned while another run's accounting is pending", async () => {
|
|
const f = await seedHeldComment();
|
|
const [task] = await db.insert(issues).values({ companyId: f.companyId, title: "Created during pending accounting", status: "todo",
|
|
assigneeAgentId: f.agentId, responsibleUserId: f.userId, issueNumber: 2 }).returning();
|
|
await queueIssueAssignmentWakeup({ heartbeat, issue: { id: task.id, assigneeAgentId: f.agentId, status: "todo" },
|
|
reason: "issue_assigned", mutation: "create", contextSource: "issue.create",
|
|
requestedByActorType: "user", requestedByActorId: f.userId, rethrowOnError: true });
|
|
await heartbeat.drainActiveRunExecutions();
|
|
const started = (await issueRuns({ companyId: f.companyId, issueId: task.id })).filter(run => run.startedAt);
|
|
expect(started).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
it("continues a task whose run ended while another run's accounting is pending", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.companyId });
|
|
await heartbeat.drainActiveRunExecutions();
|
|
expect((await issueRuns(f)).filter(run => run.id !== f.sourceRunId && run.startedAt)).toHaveLength(1);
|
|
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).not.toBe("blocked");
|
|
}, 180_000);
|
|
|
|
// Section E (invariant 4): a pause is the temporary hold left after Phase 3.
|
|
// A message refused by it is saved with its refusal and trigger, promoted
|
|
// by the one hold-release function, and delivered exactly once.
|
|
it.each([
|
|
{ hold: "agent pause", refusal: "agent.not_invokable",
|
|
apply: (f: { agentId: string }) => db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agentId)),
|
|
lift: (f: { agentId: string }) => db.update(agents).set({ status: "idle" }).where(eq(agents.id, f.agentId)) },
|
|
{ hold: "company pause", refusal: "company.inactive",
|
|
apply: (f: { companyId: string }) => db.update(companies).set({ status: "paused", pauseReason: "manual", pausedAt: new Date() })
|
|
.where(eq(companies.id, f.companyId)),
|
|
lift: (f: { companyId: string }) => db.update(companies).set({ status: "active", pauseReason: null, pausedAt: null })
|
|
.where(eq(companies.id, f.companyId)) },
|
|
])("saves a message refused by a $hold and delivers it once when the pause lifts", async ({ refusal, apply, lift }) => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
await apply(f as never);
|
|
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
|
authorUserId: f.userId, body: "Posted during the pause." }).returning();
|
|
await expect(heartbeat.wakeup(f.agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented",
|
|
payload: { issueId: f.issueId, commentId: comment.id, mutation: "comment" },
|
|
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: comment.id, wakeCommentId: comment.id,
|
|
source: "issue.comment", wakeReason: "issue_commented" },
|
|
requestedByActorType: "user", requestedByActorId: f.userId })).rejects.toMatchObject({ status: 409 });
|
|
const saved = (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId)))
|
|
.filter(wake => (wake.payload as { commentId?: string }).commentId === comment.id);
|
|
// Never a terminal skip: one deferred intent naming its refusal and trigger.
|
|
expect(saved).toEqual([expect.objectContaining({ status: "deferred_issue_execution", runId: null,
|
|
payload: expect.objectContaining({ executionWait: expect.objectContaining({ reason: refusal, trigger: "pause_lifted" }) }) })]);
|
|
await settle(f);
|
|
expect(deliveries(comment.id)).toHaveLength(0);
|
|
// The pause's own stop sweep (agent lifecycle "pausing") keeps the message
|
|
// it deferred; a terminate or cleanup stop would end it.
|
|
expect(await heartbeat.stopInvocationsForAgents([f.agentId], "Agent lifecycle stop requested",
|
|
{ preservePauseDeferred: true })).toBe(true);
|
|
expect((await readWake(saved[0].id)).status).toBe("deferred_issue_execution");
|
|
|
|
await lift(f as never);
|
|
await db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "pause_lifted", issueId: f.issueId }));
|
|
// Promotion skips the 30 s refusal backoff; it does not wait it out.
|
|
expect((await readWake(saved[0].id)).updatedAt.getTime()).toBeLessThanOrEqual(Date.now() - 29_000);
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(comment.id)).toHaveLength(1);
|
|
// A repeated release event, or the reconciler backstop, never delivers it twice.
|
|
await db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "reconciler_backstop" }));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(comment.id)).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
// A company pause stops runs asynchronously. A message posted while the
|
|
// task's run is still stopping is saved behind that run, and delivered once
|
|
// after the run ends and the pause lifts.
|
|
it("saves a message posted while the paused company's run is still stopping, and delivers it once", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(companies).set({ status: "paused", pauseReason: "manual", pausedAt: new Date() }).where(eq(companies.id, f.companyId));
|
|
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
|
authorUserId: f.userId, body: "Posted while the run was stopping." }).returning();
|
|
await expect(heartbeat.wakeup(f.agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented",
|
|
payload: { issueId: f.issueId, commentId: comment.id, mutation: "comment" },
|
|
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: comment.id, wakeCommentId: comment.id,
|
|
source: "issue.comment", wakeReason: "issue_commented" },
|
|
requestedByActorType: "user", requestedByActorId: f.userId })).rejects.toMatchObject({ status: 409 });
|
|
const saved = (await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId)))
|
|
.filter(wake => (wake.payload as { commentId?: string }).commentId === comment.id);
|
|
expect(saved).toEqual([expect.objectContaining({ status: "deferred_issue_execution",
|
|
payload: expect.objectContaining({ executionWait: expect.objectContaining({ reason: "company.inactive" }) }) })]);
|
|
// The run stops without reading the message; then the pause lifts.
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(companies).set({ status: "active", pauseReason: null, pausedAt: null }).where(eq(companies.id, f.companyId));
|
|
await db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "pause_lifted", issueId: f.issueId }));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(comment.id)).toHaveLength(1);
|
|
await settle(f);
|
|
expect(deliveries(comment.id)).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
// A pause stops the task's run with a clean, acknowledged cancellation. The
|
|
// release drain never promotes queued work behind an acknowledged stop, so
|
|
// the lifted pause issues the task's next explicit wake, which adopts the
|
|
// saved message once. The pause's own stop sweep keeps the message.
|
|
it("delivers a message saved behind a run the pause stopped, once, after the pause lifts", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
await db.update(heartbeatRuns).set({ status: "running", finishedAt: null }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issues).set({ executionRunId: f.sourceRunId }).where(eq(issues.id, f.issueId));
|
|
await db.update(companies).set({ status: "paused", pauseReason: "manual", pausedAt: new Date() }).where(eq(companies.id, f.companyId));
|
|
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
|
authorUserId: f.userId, body: "Posted while the pause stopped the run." }).returning();
|
|
await expect(heartbeat.wakeup(f.agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented",
|
|
payload: { issueId: f.issueId, commentId: comment.id, mutation: "comment" },
|
|
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: comment.id, wakeCommentId: comment.id,
|
|
source: "issue.comment", wakeReason: "issue_commented" },
|
|
requestedByActorType: "user", requestedByActorId: f.userId })).rejects.toMatchObject({ status: 409 });
|
|
// The pause's stop is acknowledged: the run ends cancelled with a clean stop
|
|
// record, and the stop releases the task's execution lock.
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date(), errorCode: "cancelled",
|
|
resultJson: { executionCancellation: { state: "acknowledged" } } }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, f.issueId));
|
|
expect(await heartbeat.stopInvocationsForAgents([f.agentId], "Agent lifecycle stop requested",
|
|
{ preservePauseDeferred: true })).toBeTypeOf("boolean");
|
|
expect(deliveries(comment.id)).toHaveLength(0);
|
|
await db.update(companies).set({ status: "active", pauseReason: null, pausedAt: null }).where(eq(companies.id, f.companyId));
|
|
await db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "pause_lifted", issueId: f.issueId }));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(comment.id)).toHaveLength(1);
|
|
await settle(f);
|
|
expect(deliveries(comment.id)).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
// Consumption is per comment: a retry that already received the paused
|
|
// message as task history consumed it, so lifting the pause never sends it
|
|
// again (RC-53 shape through the pause-lift path).
|
|
it("does not replay a paused message that a completed retry already received", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
await db.update(companies).set({ status: "paused", pauseReason: "manual", pausedAt: new Date() }).where(eq(companies.id, f.companyId));
|
|
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
|
authorUserId: f.userId, body: "Posted during the pause.", createdAt: new Date(Date.now() - 5_000),
|
|
updatedAt: new Date(Date.now() - 5_000) }).returning();
|
|
await expect(heartbeat.wakeup(f.agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented",
|
|
payload: { issueId: f.issueId, commentId: comment.id, mutation: "comment" },
|
|
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: comment.id, wakeCommentId: comment.id,
|
|
source: "issue.comment", wakeReason: "issue_commented" },
|
|
requestedByActorType: "user", requestedByActorId: f.userId })).rejects.toMatchObject({ status: 409 });
|
|
// A second message, posted later in the pause, that no run has read.
|
|
const [later] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
|
authorUserId: f.userId, body: "Also posted during the pause." }).returning();
|
|
await expect(heartbeat.wakeup(f.agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented",
|
|
payload: { issueId: f.issueId, commentId: later.id, mutation: "comment" },
|
|
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: later.id, wakeCommentId: later.id,
|
|
source: "issue.comment", wakeReason: "issue_commented" },
|
|
requestedByActorType: "user", requestedByActorId: f.userId })).rejects.toMatchObject({ status: 409 });
|
|
// A retry of the source run read the whole thread, the first comment included.
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, invocationSource: "automation",
|
|
triggerDetail: "system", status: "succeeded", startedAt: new Date(), finishedAt: new Date(), retryOfRunId: f.sourceRunId,
|
|
contextSnapshot: { issueId: f.issueId, wakeReason: "transient_failure_retry",
|
|
executionContinuation: { messages: [{ id: comment.id }] } } });
|
|
await db.update(companies).set({ status: "active", pauseReason: null, pausedAt: null }).where(eq(companies.id, f.companyId));
|
|
await db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "pause_lifted", issueId: f.issueId }));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(comment.id)).toHaveLength(0);
|
|
const saved = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),
|
|
sql`${agentWakeupRequests.payload}->>'commentId' = ${comment.id}`));
|
|
expect(saved).toEqual([expect.objectContaining({ status: "cancelled",
|
|
error: "Queued messages were already delivered to a completed run" })]);
|
|
// The wake that adopts the task's saved messages delivers only the unread one.
|
|
expect(deliveries(later.id)).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
// External chat input resumes only through its live inbound authorization,
|
|
// never through the generic pause-lift wake.
|
|
it("never replays a chat-inbound receipt through the pause-lift path", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date() }).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
|
authorUserId: f.userId, body: "From an external chat." }).returning();
|
|
const [receipt] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId,
|
|
source: "automation", triggerDetail: "system", reason: "issue_commented", status: "deferred_issue_execution",
|
|
requestedByActorType: "user", requestedByActorId: f.userId, idempotencyKey: `chat-inbound:${randomUUID()}`,
|
|
payload: { issueId: f.issueId, commentId: comment.id, mutation: "comment",
|
|
executionWait: { reason: "agent.not_invokable", trigger: "pause_lifted", message: "Agent is paused." },
|
|
_paperclipWakeContext: { issueId: f.issueId, taskId: f.issueId, wakeCommentId: comment.id, wakeCommentIds: [comment.id],
|
|
wakeReason: "issue_commented", source: "chat:slack" } } }).returning();
|
|
await db.transaction(tx => promoteDeferredWorkIntents(tx, { companyId: f.companyId, trigger: "pause_lifted", issueId: f.issueId }));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(comment.id)).toHaveLength(0);
|
|
expect((await readWake(receipt.id)).runId).toBeNull();
|
|
}, 180_000);
|
|
|
|
// A task that never ran has no terminal run for the release drain to promote
|
|
// a saved message behind. Its message stays in the task, and stranded-work
|
|
// recovery starts the task once the pause lifts.
|
|
it("starts a never-run task commented during an agent pause once the pause lifts", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
const [task] = await db.insert(issues).values({ companyId: f.companyId, title: "Never ran", status: "todo",
|
|
assigneeAgentId: f.agentId, responsibleUserId: f.userId, issueNumber: 3 }).returning();
|
|
await db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agentId));
|
|
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: task.id,
|
|
authorUserId: f.userId, body: "Please start with the FOLLOWUP line." }).returning();
|
|
await expect(heartbeat.wakeup(f.agentId, { source: "automation", triggerDetail: "system", reason: "issue_commented",
|
|
payload: { issueId: task.id, commentId: comment.id, mutation: "comment" },
|
|
contextSnapshot: { issueId: task.id, taskId: task.id, commentId: comment.id, wakeCommentId: comment.id,
|
|
source: "issue.comment", wakeReason: "issue_commented" },
|
|
requestedByActorType: "user", requestedByActorId: f.userId })).rejects.toMatchObject({ status: 409 });
|
|
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId)))
|
|
.filter(wake => wake.status === "deferred_issue_execution" && (wake.payload as { issueId?: string }).issueId === task.id)).toEqual([]);
|
|
await db.update(agents).set({ status: "idle" }).where(eq(agents.id, f.agentId));
|
|
await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.companyId });
|
|
await heartbeat.drainActiveRunExecutions();
|
|
const started = (await issueRuns({ companyId: f.companyId, issueId: task.id })).filter(run => run.startedAt);
|
|
expect(started).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
// Chaos s9401/t69: a restart landed between the comment commit (which
|
|
// reopened the finished task to todo) and its wake write. Reconciler rule:
|
|
// an undelivered user comment newer than the task's last run gets a wake.
|
|
it("delivers once a user comment whose wake a restart lost after a succeeded run", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
// The run that finished the task was native, as in the wave, and the
|
|
// comment came after it; the comment's wake was never written.
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "native", startedAt: new Date(Date.now() - 120_000),
|
|
createdAt: new Date(Date.now() - 120_000) }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
await db.update(issueComments).set({ createdAt: new Date(Date.now() - 60_000), updatedAt: new Date(Date.now() - 60_000) })
|
|
.where(eq(issueComments.id, f.commentId));
|
|
await db.update(issues).set({ status: "todo" }).where(eq(issues.id, f.issueId));
|
|
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false });
|
|
expect(report.violations.filter(row => row.reason === "user_comment_without_wake").map(row => row.recordId))
|
|
.toEqual([f.commentId]);
|
|
await applyOwnershipReportBackstops(db, report);
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date(Date.now() - 60_000) })
|
|
.where(eq(agentWakeupRequests.companyId, f.companyId));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(1);
|
|
// Delivered: the rule no longer fires, and nothing delivers it again.
|
|
const again = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false });
|
|
expect(again.violations.filter(row => row.reason === "user_comment_without_wake")).toEqual([]);
|
|
await applyOwnershipReportBackstops(db, again);
|
|
await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
it("delivers once a comment whose wake a restart lost mid-run on a task the run then closed", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "native", startedAt: new Date(Date.now() - 180_000),
|
|
createdAt: new Date(Date.now() - 180_000), finishedAt: new Date(Date.now() - 30_000) }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
await db.update(issueComments).set({ createdAt: new Date(Date.now() - 120_000), updatedAt: new Date(Date.now() - 120_000) })
|
|
.where(eq(issueComments.id, f.commentId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
await closeByAgent(f);
|
|
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false });
|
|
expect(report.violations.filter(row => row.reason === "user_comment_without_wake").map(row => row.recordId))
|
|
.toEqual([f.commentId]);
|
|
await applyOwnershipReportBackstops(db, report);
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date(Date.now() - 60_000) })
|
|
.where(eq(agentWakeupRequests.companyId, f.companyId));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(1);
|
|
const again = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false });
|
|
expect(again.violations.filter(row => row.reason === "user_comment_without_wake")).toEqual([]);
|
|
await applyOwnershipReportBackstops(db, again);
|
|
await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
// Chaos s9401/t72: a message queued its own run while the first run was
|
|
// finishing. The first run completed the task, so the queued run was
|
|
// cancelled as a stale continuation and the unread message was dropped.
|
|
it("reopens a finished task for a message whose queued run turned stale, and delivers it once", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
// The message's own queued run; the first run then completed the task.
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
|
|
triggerDetail: "system", reason: "issue_commented", status: "queued", requestedByActorType: "user", requestedByActorId: f.userId,
|
|
payload: { issueId: f.issueId, commentId: f.commentId, mutation: "comment" } }).returning();
|
|
const [stale] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, invocationSource: "automation",
|
|
triggerDetail: "system", status: "queued", wakeupRequestId: wake.id,
|
|
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: f.commentId, wakeCommentId: f.commentId,
|
|
wakeCommentIds: [f.commentId], wakeReason: "issue_commented", source: "issue.comment" } }).returning();
|
|
await db.update(agentWakeupRequests).set({ runId: stale.id }).where(eq(agentWakeupRequests.id, wake.id));
|
|
// The first run, which started before the message, completed the task after it.
|
|
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() + 1_000) }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
await closeByAgent(f);
|
|
await settle(f); await settle(f);
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, stale.id)))[0]).toMatchObject({
|
|
status: "cancelled", errorCode: "continuation_task_ownership_changed" });
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date(Date.now() - 60_000) })
|
|
.where(and(eq(agentWakeupRequests.companyId, f.companyId), eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(1);
|
|
await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
// Staging run 8c71398f: a parent agent posted a comment with `resume: true`
|
|
// ("a changed requirement for your NEXT turn") 90 s after the child's turn
|
|
// ended but before its run settled. The comment queued its own run; the
|
|
// settling run then closed the task, the queued run was cancelled as a stale
|
|
// continuation, and the explicit resume request was never delivered.
|
|
it("reopens a finished task for another agent's explicit resume comment whose queued run turned stale", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
const [parent] = await db.select().from(agents).where(and(eq(agents.companyId, f.companyId), eq(agents.name, "Earlier worker")));
|
|
const [parentRun] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: parent!.id, invocationSource: "automation",
|
|
triggerDetail: "system", status: "succeeded", startedAt: new Date(Date.now() - 20_000), finishedAt: new Date(),
|
|
contextSnapshot: {} }).returning();
|
|
const [resume] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, authorAgentId: parent!.id,
|
|
createdByRunId: parentRun!.id, body: "Changed requirement for your next turn: revalidate the verdict." }).returning();
|
|
await commentAdded(f, parent!.id, resume!.id, true);
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
|
|
triggerDetail: "system", reason: "issue_commented", status: "queued", requestedByActorType: "agent", requestedByActorId: parent!.id,
|
|
payload: { issueId: f.issueId, commentId: resume!.id, mutation: "comment", resumeIntent: true, followUpRequested: true } }).returning();
|
|
const [stale] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, invocationSource: "automation",
|
|
triggerDetail: "system", status: "queued", wakeupRequestId: wake!.id,
|
|
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: resume!.id, wakeCommentId: resume!.id,
|
|
wakeCommentIds: [resume!.id], wakeReason: "issue_commented", source: "issue.comment",
|
|
resumeIntent: true, followUpRequested: true } }).returning();
|
|
await db.update(agentWakeupRequests).set({ runId: stale!.id }).where(eq(agentWakeupRequests.id, wake!.id));
|
|
// The child's run, which started before the comment, closed the task after it.
|
|
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() + 1_000) }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
await closeByAgent(f);
|
|
await settle(f); await settle(f);
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, stale!.id)))[0]).toMatchObject({
|
|
status: "cancelled", errorCode: "continuation_task_ownership_changed" });
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date(Date.now() - 60_000) })
|
|
.where(and(eq(agentWakeupRequests.companyId, f.companyId), eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(resume!.id)).toHaveLength(1);
|
|
await settle(f);
|
|
expect(deliveries(resume!.id)).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
// Queued runs merge comments of several requesters, and the run's wake is
|
|
// rewritten as comments join. The resume intent is each comment's own (its
|
|
// `issue.comment_added` record): another agent's plain note on the same
|
|
// queued run is not promoted, and the resuming agent's comment is not lost,
|
|
// whichever comment came first.
|
|
it.each(["resume first", "resume second"] as const)("keeps each merged comment's own resume intent when two agents share a stale queued run (%s)", async order => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
const [resumer] = await db.select().from(agents).where(and(eq(agents.companyId, f.companyId), eq(agents.name, "Earlier worker")));
|
|
const [notifier] = await db.insert(agents).values({ companyId: f.companyId, name: "Notifier", adapterType, status: "idle",
|
|
adapterConfig: {} }).returning();
|
|
const post = async (authorAgentId: string, body: string, resumeIntent: boolean) => {
|
|
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, authorAgentId, body }).returning();
|
|
await commentAdded(f, authorAgentId, comment!.id, resumeIntent);
|
|
return comment!;
|
|
};
|
|
const first = order === "resume first" ? await post(resumer!.id, "Changed requirement for your next turn.", true)
|
|
: await post(notifier!.id, "FYI: unrelated status note.", false);
|
|
const second = order === "resume first" ? await post(notifier!.id, "FYI: unrelated status note.", false)
|
|
: await post(resumer!.id, "Changed requirement for your next turn.", true);
|
|
const resume = order === "resume first" ? first : second, note = order === "resume first" ? second : first;
|
|
// The first comment's wake owns the queued run and now names the joining
|
|
// comment, as claiming a merged run rewrites it; the other wake coalesced.
|
|
const firstAuthor = order === "resume first" ? resumer! : notifier!, secondAuthor = order === "resume first" ? notifier! : resumer!;
|
|
const [primary] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
|
|
triggerDetail: "system", reason: "issue_commented", status: "queued", requestedByActorType: "agent", requestedByActorId: firstAuthor.id,
|
|
payload: { issueId: f.issueId, commentId: second.id, mutation: "comment",
|
|
...(order === "resume first" ? { resumeIntent: true, followUpRequested: true } : {}) } }).returning();
|
|
await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
|
|
triggerDetail: "system", reason: "issue_commented", status: "coalesced", requestedByActorType: "agent", requestedByActorId: secondAuthor.id,
|
|
payload: { issueId: f.issueId, commentId: second.id, mutation: "comment",
|
|
...(order === "resume second" ? { resumeIntent: true, followUpRequested: true } : {}) } });
|
|
const [stale] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, invocationSource: "automation",
|
|
triggerDetail: "system", status: "queued", wakeupRequestId: primary!.id,
|
|
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: second.id, wakeCommentId: second.id,
|
|
wakeCommentIds: [first.id, second.id], wakeReason: "issue_commented", source: "issue.comment",
|
|
resumeIntent: true, followUpRequested: true } }).returning();
|
|
await db.update(agentWakeupRequests).set({ runId: stale!.id }).where(eq(agentWakeupRequests.id, primary!.id));
|
|
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() + 1_000) }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
await closeByAgent(f);
|
|
await settle(f); await settle(f);
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date(Date.now() - 60_000) })
|
|
.where(and(eq(agentWakeupRequests.companyId, f.companyId), eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(resume.id)).toHaveLength(1);
|
|
expect(deliveries(note.id).filter(turn => !(turn.context.wakeCommentIds as string[]).includes(resume.id))).toHaveLength(0);
|
|
const [saved] = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),
|
|
eq(agentWakeupRequests.requestedByActorId, resumer!.id), sql`${agentWakeupRequests.payload}->>'commentId' = ${resume.id}`,
|
|
sql`${agentWakeupRequests.payload}->'_paperclipWakeContext' is not null`));
|
|
expect(saved?.payload).toMatchObject({ resumeIntent: true });
|
|
expect((saved?.payload as { _paperclipWakeContext?: { wakeCommentIds?: string[] } })._paperclipWakeContext?.wakeCommentIds)
|
|
.toEqual([resume.id]);
|
|
}, 180_000);
|
|
|
|
// Without `resume: true` an agent's comment stays communicative: the closed
|
|
// task is not reopened for it (issues route: only human comments reopen
|
|
// finished work implicitly).
|
|
it("does not reopen a finished task for another agent's comment without resume intent", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
const [parent] = await db.select().from(agents).where(and(eq(agents.companyId, f.companyId), eq(agents.name, "Earlier worker")));
|
|
const [note] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId, authorAgentId: parent!.id,
|
|
body: "FYI: the parent review finished." }).returning();
|
|
await commentAdded(f, parent!.id, note!.id, false);
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
|
|
triggerDetail: "system", reason: "issue_commented", status: "queued", requestedByActorType: "agent", requestedByActorId: parent!.id,
|
|
payload: { issueId: f.issueId, commentId: note!.id, mutation: "comment" } }).returning();
|
|
const [stale] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, invocationSource: "automation",
|
|
triggerDetail: "system", status: "queued", wakeupRequestId: wake!.id,
|
|
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: note!.id, wakeCommentId: note!.id,
|
|
wakeCommentIds: [note!.id], wakeReason: "issue_commented", source: "issue.comment" } }).returning();
|
|
await db.update(agentWakeupRequests).set({ runId: stale!.id }).where(eq(agentWakeupRequests.id, wake!.id));
|
|
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() + 1_000) }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
await closeByAgent(f);
|
|
await settle(f); await settle(f);
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date(Date.now() - 60_000) })
|
|
.where(and(eq(agentWakeupRequests.companyId, f.companyId), eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
await settle(f);
|
|
expect(deliveries(note!.id)).toHaveLength(0);
|
|
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]?.status).toBe("done");
|
|
}, 180_000);
|
|
|
|
// The durable half: the stale check ran before the closing run settled (or
|
|
// its save failed), so only the cancelled stale wake carries the message.
|
|
// Once the closing run settles, the reconciler saves it and the drain
|
|
// reopens the task and delivers it once.
|
|
it("recovers a message a stale continuation dropped once the closing run settles", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.update(agentWakeupRequests).set({ status: "cancelled", finishedAt: new Date(),
|
|
error: "continuation_task_ownership_changed" }).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
await db.update(heartbeatRuns).set({ finishedAt: new Date(Date.now() + 1_000) }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
// A person closed it: that decision stands.
|
|
await closeByAgent(f, "user");
|
|
const byPerson = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false });
|
|
expect(byPerson.violations.filter(row => row.reason === "user_comment_dropped_by_stale_continuation")).toEqual([]);
|
|
// The agent's run closed it later without reading the message.
|
|
await closeByAgent(f);
|
|
// Another message's own turn succeeded later; it did not receive this one.
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, invocationSource: "automation",
|
|
triggerDetail: "system", status: "succeeded", startedAt: new Date(Date.now() + 2_000), finishedAt: new Date(Date.now() + 3_000),
|
|
contextSnapshot: { issueId: f.issueId, wakeReason: "issue_commented", wakeCommentIds: [randomUUID()],
|
|
executionContinuation: { messages: [{ id: f.commentId }] } } });
|
|
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false });
|
|
expect(report.violations.filter(row => row.reason === "user_comment_dropped_by_stale_continuation")
|
|
.map(row => row.recordId)).toEqual([f.commentId]);
|
|
expect(await applyOwnershipReportBackstops(db, report)).toEqual({ retryNeeded: false });
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date(Date.now() - 60_000) })
|
|
.where(and(eq(agentWakeupRequests.companyId, f.companyId), eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(1);
|
|
const again = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false });
|
|
expect(again.violations.filter(row => row.recordId === f.commentId)).toEqual([]);
|
|
}, 180_000);
|
|
|
|
it("keeps a reconciler pass due when the backstop cannot save a message", async () => {
|
|
const report = { violations: [{ rule: "I4_exactly_once_intent", kind: "issue_comment", recordId: randomUUID(),
|
|
companyId: "not-a-company-id", runId: null, issueId: "not-an-issue-id", reason: "user_comment_without_wake" }], newViolations: [] };
|
|
expect(await applyOwnershipReportBackstops(db, report as never)).toEqual({ retryNeeded: true });
|
|
});
|
|
|
|
it("keeps the terminal outcomes closed and every temporary refusal mapped to a release trigger", () => {
|
|
expect([...WORK_INTENT_TERMINAL_OUTCOMES]).toEqual(["delivered", "consumed", "task_cancelled", "hard_stop"]);
|
|
for (const refusal of ["budget.blocked", "company.inactive", "agent.not_invokable", "issue_tree_hold_active",
|
|
"heartbeat.scheduling_suppressed", "wake_lost"]) expect(TEMPORARY_REFUSAL_TRIGGERS[refusal]).toBeTruthy();
|
|
});
|
|
|
|
it.each([
|
|
{ delivery: "task history", context: (commentId: string) => ({ executionContinuation: { messages: [{ id: commentId }] } }) },
|
|
{ delivery: "wake input", context: (commentId: string) => ({ wakeCommentIds: [commentId] }) },
|
|
])("consumes a saved comment that a later completed run already received as $delivery, without reopening it", async ({ context }) => {
|
|
// RC-53: the saved comment's source run was lost; its retry started after
|
|
// the comment, received it in its prompt, finished the task, and only then
|
|
// did the saved wake drain.
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "process_lost" }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, invocationSource: "automation",
|
|
triggerDetail: "system", status: "succeeded", startedAt: new Date(), finishedAt: new Date(), retryOfRunId: f.sourceRunId,
|
|
contextSnapshot: { issueId: f.issueId, wakeReason: "transient_failure_retry", ...context(f.commentId) } });
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(0);
|
|
expect(await readWake(f.wakeId)).toMatchObject({ status: "cancelled",
|
|
error: "Queued messages were already delivered to a completed run" });
|
|
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0].status).toBe("done");
|
|
}, 180_000);
|
|
|
|
it.each([
|
|
{ shape: "edited after the later run started", edit: true, startedBeforeComment: false },
|
|
{ shape: "posted while the later run was already running", edit: false, startedBeforeComment: true },
|
|
])("still delivers a saved comment $shape", async ({ edit, startedBeforeComment }) => {
|
|
// A run's recorded context can be refreshed after it starts, and an edit
|
|
// made after it started is unread: neither proves delivery.
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "process_lost" }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
const startedAt = startedBeforeComment ? new Date(Date.now() - 60_000) : new Date();
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, invocationSource: "automation",
|
|
triggerDetail: "system", status: "succeeded", startedAt, finishedAt: new Date(), retryOfRunId: f.sourceRunId,
|
|
contextSnapshot: { issueId: f.issueId, wakeReason: "transient_failure_retry",
|
|
executionContinuation: { messages: [{ id: f.commentId }] } } });
|
|
if (edit) await db.update(issueComments).set({ body: "Edited: also include the FOLLOWUP line.", updatedAt: new Date(Date.now() + 1_000) })
|
|
.where(eq(issueComments.id, f.commentId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
it("still delivers a saved comment that the later completed run never received", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "process_lost" }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, invocationSource: "automation",
|
|
triggerDetail: "system", status: "succeeded", startedAt: new Date(), finishedAt: new Date(), retryOfRunId: f.sourceRunId,
|
|
contextSnapshot: { issueId: f.issueId, wakeReason: "transient_failure_retry", executionContinuation: { messages: [] } } });
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
it("still gives a saved comment its own turn when a later non-retry run only saw it as task history", async () => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.update(heartbeatRuns).set({ status: "failed", errorCode: "process_lost" }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, invocationSource: "automation",
|
|
triggerDetail: "system", status: "succeeded", startedAt: new Date(), finishedAt: new Date(),
|
|
contextSnapshot: { issueId: f.issueId, wakeReason: "issue_commented", wakeCommentIds: [randomUUID()],
|
|
executionContinuation: { messages: [{ id: f.commentId }] } } });
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(1);
|
|
}, 180_000);
|
|
|
|
async function runningNativeSource(f: Awaited<ReturnType<typeof seedHeldComment>>) {
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
await db.update(heartbeatRuns).set({ status: "running", runtimeMode: "native", nativeIssueId: f.issueId, finishedAt: null,
|
|
startedAt: new Date(Date.now() - 30_000) }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issues).set({ executionRunId: f.sourceRunId, executionLockedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
}
|
|
|
|
async function readHistoryDuringTurn(f: Awaited<ReturnType<typeof seedHeldComment>>) {
|
|
const reader = new PaperclipRunnerToolAuthority(db, { companyId: f.companyId, agentId: f.agentId, issueId: f.issueId,
|
|
runId: f.sourceRunId });
|
|
try {
|
|
const history = await reader.execute({ tool: "get_task_history", callId: `history-${f.sourceRunId}`, arguments: {} });
|
|
expect((history as { comments: Array<{ id: string }> }).comments.map(comment => comment.id)).toContain(f.commentId);
|
|
expect((history as { arrivedDuringThisTurn?: string[] }).arrivedDuringThisTurn).toContain(`comment:${f.commentId}`);
|
|
} finally { reader.close(); }
|
|
}
|
|
|
|
const reportResult = (f: { companyId: string; agentId: string; commentId: string }, runId: string, cited: boolean) =>
|
|
db.insert(heartbeatRunEvents).values({ companyId: f.companyId, runId, agentId: f.agentId, seq: 100_000, eventType: "run.result.accepted",
|
|
stream: "system", level: "info", payload: { prpEvent: { payload: { result: { schema: "paperclip.run_result.v1",
|
|
reportedWorkDisposition: "done", summary: "Done.",
|
|
evidence: [{ ref: "document:plan" }, ...(cited ? [{ ref: `comment:${f.commentId} (applied)` }] : [])],
|
|
completionClaim: { contractRevision: "1", objectiveSatisfied: true, remainingWork: [],
|
|
criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: ["document:plan"] }] } } } } } });
|
|
|
|
async function closeSourceRun(f: Awaited<ReturnType<typeof seedHeldComment>>) {
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date(Date.now() + 1_000) })
|
|
.where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, f.issueId));
|
|
await closeByAgent(f);
|
|
}
|
|
|
|
const issueStatus = async (f: { issueId: string }) =>
|
|
(await db.select({ status: issues.status }).from(issues).where(eq(issues.id, f.issueId)))[0]?.status;
|
|
|
|
const postComment = (f: Awaited<ReturnType<typeof seedHeldComment>>) => heartbeat.wakeup(f.agentId, {
|
|
source: "automation", triggerDetail: "system", reason: "issue_commented",
|
|
payload: { issueId: f.issueId, commentId: f.commentId, mutation: "comment" },
|
|
requestedByActorType: "user", requestedByActorId: f.userId,
|
|
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: f.commentId, wakeCommentId: f.commentId,
|
|
source: "issue.comment", wakeReason: "issue_commented" },
|
|
});
|
|
|
|
it.each([
|
|
{ shape: "read unchanged and cited as handled", edit: false, cited: true, delivered: 0 },
|
|
{ shape: "read unchanged but not cited", edit: false, cited: false, delivered: 1 },
|
|
{ shape: "edited after the read", edit: true, cited: true, delivered: 1 },
|
|
])("settles a stale queued run of a message the closing run read mid-turn ($shape)", async ({ edit, cited, delivered }) => {
|
|
const f = await seedHeldComment();
|
|
await runningNativeSource(f);
|
|
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation",
|
|
triggerDetail: "system", reason: "issue_commented", status: "queued", requestedByActorType: "user", requestedByActorId: f.userId,
|
|
payload: { issueId: f.issueId, commentId: f.commentId, mutation: "comment" } }).returning();
|
|
const [stale] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, invocationSource: "automation",
|
|
triggerDetail: "system", status: "queued", wakeupRequestId: wake!.id,
|
|
contextSnapshot: { issueId: f.issueId, taskId: f.issueId, commentId: f.commentId, wakeCommentId: f.commentId,
|
|
wakeCommentIds: [f.commentId], wakeReason: "issue_commented", source: "issue.comment" } }).returning();
|
|
await db.update(agentWakeupRequests).set({ runId: stale!.id }).where(eq(agentWakeupRequests.id, wake!.id));
|
|
await readHistoryDuringTurn(f);
|
|
if (edit) await db.update(issueComments).set({ body: "Edited: also include the FOLLOWUP line twice.",
|
|
updatedAt: new Date(Date.now() + 500) }).where(eq(issueComments.id, f.commentId));
|
|
await reportResult(f, f.sourceRunId, cited);
|
|
await closeSourceRun(f);
|
|
await db.update(issues).set({ executionRunId: null, executionLockedAt: null }).where(eq(issues.id, f.issueId));
|
|
await settle(f); await settle(f);
|
|
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, stale!.id)))[0]).toMatchObject({
|
|
status: "cancelled", errorCode: "continuation_task_ownership_changed" });
|
|
await db.update(agentWakeupRequests).set({ updatedAt: new Date(Date.now() - 60_000) })
|
|
.where(and(eq(agentWakeupRequests.companyId, f.companyId), eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(delivered);
|
|
await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(delivered);
|
|
expect(await issueStatus(f)).toBe("done");
|
|
const report = await runOwnershipReconciliationPass(db, { state: createOwnershipReportState(), record: false });
|
|
expect(report.violations.filter(row => row.recordId === f.commentId)).toEqual([]);
|
|
}, 180_000);
|
|
|
|
it.each([
|
|
{ outcome: "succeeded", edit: "none", cited: true, consumed: true },
|
|
{ outcome: "succeeded", edit: "none", cited: false, consumed: false },
|
|
{ outcome: "succeeded", edit: "during the run", cited: true, consumed: false },
|
|
{ outcome: "succeeded", edit: "after the run finished", cited: true, consumed: false },
|
|
{ outcome: "failed", edit: "none", cited: true, consumed: false },
|
|
])("counts a mid-turn history read as delivery only for a run that $outcome on the current revision and cited it (edited: $edit, cited: $cited)", async ({ outcome, edit, cited, consumed }) => {
|
|
const f = await seedHeldComment();
|
|
await runningNativeSource(f);
|
|
await readHistoryDuringTurn(f);
|
|
await reportResult(f, f.sourceRunId, cited);
|
|
const editComment = (at: number) => db.update(issueComments).set({ body: "Edited after the read.", updatedAt: new Date(at) })
|
|
.where(eq(issueComments.id, f.commentId));
|
|
if (edit === "during the run") await editComment(Date.now() + 500);
|
|
await db.update(heartbeatRuns).set({ status: outcome, finishedAt: new Date(Date.now() + 1_000),
|
|
...(outcome === "failed" ? { errorCode: "adapter_failed" } : {}) }).where(eq(heartbeatRuns.id, f.sourceRunId));
|
|
if (edit === "after the run finished") await editComment(Date.now() + 60_000);
|
|
const delivered = await deliveredQueuedCommentIds(db, { companyId: f.companyId, issueId: f.issueId, agentId: f.agentId,
|
|
commentIds: [f.commentId] });
|
|
expect(delivered.has(f.commentId)).toBe(consumed);
|
|
}, 180_000);
|
|
|
|
it("holds a message behind a run another server is executing instead of queueing a run of its own", async () => {
|
|
const f = await seedHeldComment();
|
|
await runningNativeSource(f);
|
|
await postComment(f);
|
|
const queued = (await issueRuns(f)).filter(run => run.id !== f.sourceRunId && run.status === "queued");
|
|
expect(queued).toEqual([]);
|
|
const held = await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, f.companyId),
|
|
eq(agentWakeupRequests.status, "deferred_issue_execution")));
|
|
expect(held.map(row => row.payload?.commentId)).toEqual([f.commentId]);
|
|
}, 180_000);
|
|
|
|
it.each([
|
|
{ read: true, cited: true, delivered: 0 },
|
|
{ read: true, cited: false, delivered: 1 },
|
|
{ read: false, cited: false, delivered: 1 },
|
|
])("settles a held message as soon as its run closes the task (read mid-turn: $read, cited: $cited)", async ({ read, cited, delivered }) => {
|
|
const f = await seedHeldComment();
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId));
|
|
let closingRunId: string | null = null;
|
|
duringTurn = async runId => {
|
|
duringTurn = null;
|
|
closingRunId = runId;
|
|
await db.update(issueComments).set({ createdAt: new Date(), updatedAt: new Date() }).where(eq(issueComments.id, f.commentId));
|
|
await postComment(f);
|
|
if (!read) return;
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: f.issueId }).where(eq(heartbeatRuns.id, runId));
|
|
await readHistoryDuringTurn({ ...f, sourceRunId: runId });
|
|
await reportResult(f, runId, cited);
|
|
await db.update(heartbeatRuns).set({ runtimeMode: "legacy" }).where(eq(heartbeatRuns.id, runId));
|
|
await closeByAgent(f);
|
|
};
|
|
if (!read) await closeByAgent(f);
|
|
await heartbeat.wakeup(f.agentId, { source: "assignment", triggerDetail: "system", reason: "issue_assigned",
|
|
payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId, taskId: f.issueId, wakeReason: "issue_assigned" } });
|
|
await heartbeat.drainActiveRunExecutions();
|
|
await heartbeat.drainActiveRunExecutions();
|
|
expect(closingRunId).not.toBeNull();
|
|
expect(deliveries(f.commentId)).toHaveLength(delivered);
|
|
expect((await issueRuns(f)).filter(run => run.status === "cancelled")).toEqual([]);
|
|
await settle(f); await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(delivered);
|
|
expect(await issueStatus(f)).toBe("done");
|
|
const delivery = deliveries(f.commentId)[0]?.context.paperclipWake as { comments?: Array<{ id: string; readByEarlierTurn?: boolean }> } | undefined;
|
|
if (delivered) expect(delivery?.comments?.find(comment => comment.id === f.commentId)?.readByEarlierTurn ?? false).toBe(read);
|
|
}, 180_000);
|
|
|
|
it("still refuses work at a reached hard stop, which the board owns", async () => {
|
|
const f = await seedHeldComment({ hardStopReached: true });
|
|
await db.update(heartbeatRuns).set({ costAccountingPending: false }).where(eq(heartbeatRuns.id, f.pendingRunId));
|
|
// The comment's wake was promoted earlier; a reached hard stop refuses it terminally.
|
|
await db.insert(costEvents).values({ companyId: f.companyId, agentId: f.agentId, provider: "openai", model: "gpt",
|
|
costCents: 50, occurredAt: new Date() });
|
|
await settle(f);
|
|
expect(deliveries(f.commentId)).toHaveLength(0);
|
|
expect(await issueRuns(f).then(runs => runs.filter(run => run.status === "queued"))).toHaveLength(0);
|
|
}, 180_000);
|
|
});
|