Files
PaperClipAI/server/src/modules/wake-queue/adapters/queued-operator-stop-postgres.test.ts
T
Devin FoleyandPaperclip 6fae6f2cb8 fix(wake-queue): preserve pending work when a queued run is stopped (#16357)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The wake queue releases task locks and can promote saved follow-up
work when a run ends.
> - An operator can stop a queued run before dispatch selects its
runtime.
> - The legacy release path can promote the old follow-up after that
Stop.
> - For an obsolete routine cycle, this can conflict with the current
active cycle after cancellation has already committed.
> - This pull request preserves the deferred input without promoting it
at that release boundary.
> - The benefit is that a queued Stop stays stopped and does not return
an error from starting an obsolete follow-up.

## Linked Issues or Issue Description

Related: Refs #14300 and Refs #16339 describe other pre-dispatch
cancellation paths. Related PR #16238 changes former-owner wake
selection. This change addresses expected operator cancellation before
legacy runtime selection, without changing those contracts.

**What happened?**

Stopping an unstarted legacy run committed its cancellation, then tried
to promote a deferred wake. When another cycle of the same routine was
already active, the issue lock update failed against
`issues_open_routine_execution_uq`. The cancel request returned HTTP 500
even though the run was cancelled. Without a competing cycle, the Stop
launched the old follow-up.

**What did you expect to happen?**

Release the stopped run's own locks and preserve deferred input for a
later eligible instruction. Do not start the old queue as a side effect
of Stop.

**Steps to reproduce**

Create two cycles of the same routine. Keep the current cycle active.
Give the old cycle an unstarted legacy run with a verified bootstrap
record and a deferred wake. Cancel that run as a board operator, then
release its issue execution. The added database regression reproduces
the failed lock update before the fix.

**Paperclip version / commit**

Reproduced on parent commit `976cdc77ed564dc3e014621505890b647e554a7f`.

**Deployment mode**

A remote runtime exposed the symptom. The regression uses embedded
PostgreSQL and the real wake-queue adapter.

## What Changed

- Recognize an expected operator Stop only when the legacy run has no
runtime-selection time, start time, process-start time, or process
identity.
- Use the existing release boundary to preserve deferred input without
promoting it.
- Add database regressions for ordinary Stop, a competing routine cycle,
repeated release, provider cancellation, runtime selection, and
dispatched-run reconciliation.
- Document the behavior and update both recovery maps.

## Verification

- Before the fix, both queued Stop cases failed. The competing-cycle
case raised `issues_open_routine_execution_uq`. The three control cases
passed.
- `node node_modules/vitest/vitest.mjs run --project @paperclipai/server
server/src/modules/wake-queue`: 7 files and 214 tests passed, including
all five new database cases.
- `node scripts/check-maps.mjs` and `git diff --check` passed.
- `pnpm -r typecheck`, `pnpm build`, and all 10 map-script tests passed
locally. The complete CI suite is green on
`8e204eef87f1a593df5c183abef764e4b9eb5fd1`. The remaining serial local
full-suite run was stopped once that unchanged-head CI verification
passed; a local full-suite pass is not claimed. Apex review passed at
5/5 on the same head with no review threads.

## Risks

- The change is limited to recorded operator cancellation before legacy
runtime selection. Selected runtimes, dispatched work, provider
cancellation, and native stop acknowledgement keep their existing paths.
- Deferred work is retained. A later eligible instruction must still use
normal admission and all current gates.
- No schema migration or new authority is required. This regression is
not proof of a live provider stop.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. The exact serving snapshot ID and context window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-14 18:18:26 -07:00

96 lines
6.7 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { agentWakeupRequests, agents, companies, createDb, heartbeatRuns, issues } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../../__tests__/helpers/embedded-postgres.js";
import { createReleaseIssueExecution } from "../application/use-cases.js";
import { createPostgresWakeQueueAdapter } from "./postgres.js";
const support = await getEmbeddedPostgresTestSupport();
const describePostgres = support.supported ? describe : describe.skip;
describePostgres("queued operator stop release", () => {
let db: ReturnType<typeof createDb>;
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
beforeAll(async () => {
database = await startEmbeddedPostgresTestDatabase("paperclip-queued-operator-stop-");
db = createDb(database.connectionString);
}, 20_000);
afterAll(async () => {
await database?.cleanup();
});
async function seed(input: { activeCycle?: boolean; source?: string; started?: boolean; selected?: boolean }) {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
const originId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Stop regression", issuePrefix: `T${companyId.slice(0, 6)}`,
defaultResponsibleUserId: "operator", requireBoardApprovalForNewAgents: false });
await db.insert(agents).values({ id: agentId, companyId, name: "Worker", role: "engineer", status: "active",
adapterType: "codex_local", adapterConfig: {}, runtimeConfig: { heartbeat: { wakeOnDemand: true } } });
await db.insert(issues).values({ id: issueId, companyId, title: "Stopped cycle", status: "in_progress",
assigneeAgentId: agentId, responsibleUserId: "operator", originKind: "routine_execution", originId });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", status: "cancelled",
runtimeMode: "legacy", contextSnapshot: { issueId }, finishedAt: new Date(),
...(input.started ? { startedAt: new Date(), runtimeModeResolvedAt: new Date() } : input.selected ? { runtimeModeResolvedAt: new Date() } : {}),
resultJson: { ...(!input.started ? { executionRecovery: { kind: "bootstrap", providerWorkStarted: false },
startupCancellation: { requestedAt: new Date().toISOString(), beforeNativeSelection: false } } : {}), cancellation: { source: input.source ?? "operator", expected: input.source !== "provider",
initiator: { type: input.source === "provider" ? "provider" : "user", id: "operator" },
reason: "Stopped", recordedAt: new Date().toISOString() } } });
let activeCycleId: string | null = null;
let activeRunId: string | null = null;
if (input.activeCycle) {
activeCycleId = randomUUID(); activeRunId = randomUUID();
await db.insert(heartbeatRuns).values({ id: activeRunId, companyId, agentId, invocationSource: "on_demand", status: "running",
contextSnapshot: { issueId: activeCycleId }, startedAt: new Date() });
await db.insert(issues).values({ id: activeCycleId, companyId, title: "Current cycle", status: "in_progress",
assigneeAgentId: agentId, originKind: "routine_execution", originId, executionRunId: activeRunId });
} else {
await db.update(issues).set({ executionRunId: runId, checkoutRunId: runId }).where(eq(issues.id, issueId));
}
const [wake] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", reason: "issue_commented",
status: "deferred_issue_execution", payload: { issueId }, requestedByActorType: "user", requestedByActorId: "operator" }).returning();
const release = createReleaseIssueExecution({
issueLock: createPostgresWakeQueueAdapter(db, { resolveResponsibleUserId: async () => "operator",
getRoutineEnv: async () => ({ routineId: null, env: null, responsibleUserId: null }), resolveSessionBeforeForWakeup: async () => null }),
recovery: { escalateStrandedAssignedIssue: async () => { throw Error("Unexpected recovery"); },
escalateStrandedRecoveryIssueInPlace: async () => { throw Error("Unexpected recovery"); } },
});
return { companyId, agentId, issueId, runId, wakeId: wake.id, activeCycleId, activeRunId, release };
}
it.each([false, true])("preserves pending work after a pre-dispatch Stop (active cycle=%s)", async (activeCycle) => {
const f = await seed({ activeCycle });
const before = (await db.select().from(issues).where(eq(issues.id, f.issueId)))[0];
const result = await f.release({ companyId: f.companyId, runId: f.runId, now: new Date() });
expect(result).toEqual({ outcome: { kind: "released" }, postCommitEffects: [] });
expect((await db.select().from(issues).where(eq(issues.id, f.issueId)))[0]).toMatchObject({ status: "in_progress",
statusVersion: before.statusVersion, assigneeAgentId: f.agentId, executionRunId: null, checkoutRunId: null });
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId)))[0])
.toMatchObject({ status: "deferred_issue_execution", runId: null });
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(activeCycle ? 2 : 1);
if (f.activeCycleId) expect((await db.select().from(issues).where(eq(issues.id, f.activeCycleId)))[0])
.toMatchObject({ status: "in_progress", executionRunId: f.activeRunId });
expect(await f.release({ companyId: f.companyId, runId: f.runId, now: new Date() })).toEqual(result);
});
it("keeps ordinary provider cancellation promotion", async () => {
const f = await seed({ source: "provider" });
expect((await f.release({ companyId: f.companyId, runId: f.runId, now: new Date() })).outcome.kind).toBe("promoted");
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId)))[0].status).toBe("queued");
});
it("does not replace stop acknowledgement for a dispatched run", async () => {
const f = await seed({ started: true });
expect((await f.release({ companyId: f.companyId, runId: f.runId, now: new Date() })).outcome.kind).toBe("released");
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, f.wakeId)))[0])
.toMatchObject({ status: "deferred_issue_execution", runId: null });
});
it("keeps the existing release path when runtime selection has occurred", async () => {
const f = await seed({ selected: true });
expect((await f.release({ companyId: f.companyId, runId: f.runId, now: new Date() })).outcome.kind).toBe("promoted");
});
});