Files
PaperClipAI/server/src/services/native-runtime/native-stale-owner-completion.test.ts
T
DottaandClaude Opus 5.5 98e0f1fc0d fix(native-runtime): a stopped run cannot close a task another run continues; deliver a pre-restart comment at once (#20216)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When the board stops a run and then posts a comment asking the agent
to continue, that comment starts a new run, and only that new run should
decide when the task is done
> - In restart chaos wave `w-20261109T081456-s9901`, t16 (claude S1, K1)
closed the task as done without the line the user asked for, and t10
(codex S1, K6) never woke a run for the continue comment, so it needed a
Retry
> - In t16, the stopped run's saved "done" result was accepted 10 s
after the continue run started. In t10, a restart killed the server 21
ms after the comment committed, before its wake was written, and the
comment backstop then waited out its 30 s grace
> - This pull request stops a run from closing a task that another run
now holds, and makes the backstop deliver at once a comment that
predates the server process
> - The benefit is that a user's latest direction after a Stop is
neither overridden by the stopped run nor lost to a restart

## Linked Issues or Issue Description

Refs #19831. Restart chaos wave `w-20261109T081456-s9901` (master
`09e24dd12c`), tasks t16 and t10.

**t16, from the kept database (CHA-20):**
- 08:29:19: the board stopped run 1 (`6e33d123`). Its saved result
reported done, but finalization failed on the stop
(`finalization_failed_claim_preserved`, a retryable failure).
- 08:29:41: the user's continue comment started run 2 (`2cb71c5b`). Run
2 held the task's execution and saw the comment.
- 08:29:51: the finalization reconciler retried run 1's saved result and
committed `completion_claim_policy_accepted` for run 1. The task closed
as done.
- From then on, every Paperclip tool call from run 2 was refused
(`paperclip_runner_tool_binding_not_authorized`), so it could not add
the `CONTINUE-…` line. Its result (blocked) was then preserved as
`terminal_status_preserved`.
- Run 2 ending `succeeded` with `native_session_lease_lost` follows from
this: its result was accepted before the lease was lost. The comment was
delivered; the model saw it.

**t10 (CHA-14):**
- The Stop moved the task to blocked. The user's comment reopened it to
todo at 08:26:28.633, and K6 killed the server at 08:26:28.654, before
the route wrote the comment's wake. No wake request exists for that
comment.
- This is not caused by the Stop, #20066 or #20180: the wake is written
after the comment commits, and the kill landed in between.
- The I4 backstop ("a user comment without a wake") matched the comment,
but treated it as possibly still getting its normal wake for 30 s. The
pass interval then pushed the check to 02:27:16, and the harness clicked
Retry at 02:26:59.

## What Changed

- `status-arbiter.ts`:
  - New `heldByAnotherRun(issue, runId)`.
- `supersedeStaleOwnerCompletion` also keeps status, with reason
`completion_run_superseded`, when the issue's `executionRunId` is a
different run. A run that was reassigned away still gets
`completion_owner_superseded`.
- `native-run-finalizer.ts` and `native-status-reassessment.ts` pass the
run id to that guard.
- `status-decision-committer.ts` rechecks the same rule under the issue
lock: a done decision for a non-conversation issue held by another run
raises `NativeStatusRaceError`. The finalizer then re-arbitrates and
keeps status.
- `ownership-invariant-report.ts`: the comment-without-wake grace now
only applies to comments created after this server process started
(`getServerProcessStartedAt`). Separate live server replicas are not
supported, and the wake is written by the process that accepted the
comment, so an older comment is owed at once. The startup pass now
delivers it. `runOwnershipReconciliationPass` takes an optional
`processStartedAt`.

## Verification

Red/green:
- `native-stale-owner-completion.test.ts`: three new tests.
- The unit policy: another run holding the task keeps status; the same
run, or none, still closes it.
- "does not let a stopped run's saved done result close the task a newer
run is continuing" (the t16 shape).
- "does not close the task when a newer run takes it after arbitration
but before the status commit" (only the committer recheck covers this).
  - On master: 3 failed. With this PR: 13 passed.
- `ownership-reconciliation-report.test.ts`: "finds at once a young user
comment whose wake a restart lost after the board stopped the run" (the
t10 shape). It fails on master and passes with this PR; the same comment
in a process that accepted it still waits for its grace deadline. The
file: 27 passed.

Regression and static checks:
- 16 nearby native finalization, status projection, workspace finalizer
and arbiter suites: 439 passed.
- `tsc --noEmit` passes for `server`, and for the two test files.

## Risks

- A done result now waits when a different run holds the task. That run
reaches its own decision; the earlier result stays as history. If the
holder is stale, the existing execution-lock recovery clears it, and a
later reassessment can then close the task.
- After a restart, a user comment newer than the task's last run, with
no wake, is delivered at the startup pass rather than 30 s later. A
replica that is still live and writing the same wake is not a supported
topology.
- No schema change and no change to persisted run state.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 03:03:15 -06:00

254 lines
16 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { agents, companies, completionContracts, createDb, heartbeatRuns, issueWorkProducts, issues, nativeRunFinalizations,
nativeRunResults, statusDecisions, type Db } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
import { NativeRunCoordinatorStore } from "./native-run-coordinator-store.js";
import { finalizeNativeRun } from "./native-run-finalizer.js";
import { reconcileNativeFinalizations } from "./native-finalization-reconciler.js";
import { NATIVE_STATUS_ARBITER_POLICY_VERSION, supersedeStaleOwnerCompletion, type NativeStatusDecision } from "./status-arbiter.js";
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_STALE_OWNER_TEST_DATABASE_URL");
describe("stale owner completion policy", () => {
const done: NativeStatusDecision = { policyVersion: NATIVE_STATUS_ARBITER_POLICY_VERSION, statusAction: "done", toStatus: "done",
reasonCode: "completion_claim_policy_accepted", unblockDescriptor: null, effects: [{ kind: "release_checkout" }] };
const owner = { status: "in_progress", assigneeAgentId: "agent-a", assigneeUserId: null };
it("keeps the current owner's done decision", () => {
expect(supersedeStaleOwnerCompletion(done, owner, "agent-a")).toBe(done);
});
it.each([
["another agent", { ...owner, assigneeAgentId: "agent-b" }],
["a board user", { ...owner, assigneeAgentId: null, assigneeUserId: "user-1" }],
["nobody", { ...owner, assigneeAgentId: null }],
])("preserves status when the task now belongs to %s", (_label, issue) => {
expect(supersedeStaleOwnerCompletion(done, issue, "agent-a")).toEqual({ ...done, statusAction: "preserve",
toStatus: "in_progress", reasonCode: "completion_owner_superseded", effects: [] });
});
it("preserves status when another run of the same owner now holds the task", () => {
expect(supersedeStaleOwnerCompletion(done, { ...owner, executionRunId: "run-2" }, "agent-a", "run-1")).toEqual({ ...done,
statusAction: "preserve", toStatus: "in_progress", reasonCode: "completion_run_superseded", effects: [] });
expect(supersedeStaleOwnerCompletion(done, { ...owner, executionRunId: "run-1" }, "agent-a", "run-1")).toBe(done);
expect(supersedeStaleOwnerCompletion(done, { ...owner, executionRunId: null }, "agent-a", "run-1")).toBe(done);
});
it("leaves non-terminal decisions to their existing owner checks", () => {
const blocked: NativeStatusDecision = { ...done, statusAction: "blocked", toStatus: "blocked", reasonCode: "x" };
expect(supersedeStaleOwnerCompletion(blocked, { ...owner, assigneeAgentId: "agent-b" }, "agent-a")).toBe(blocked);
});
});
const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport();
describe.skipIf(!support.supported)("native completion after reassignment", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: Db;
beforeAll(async () => {
if (!external) database = await startEmbeddedPostgresTestDatabase("native-stale-owner-");
db = createDb(external ?? database!.connectionString);
}, 60_000);
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
/**
* A task whose original owner reported done in an accepted result. With
* pendingEvidence, the server arbiter waits for review of a work product.
*/
async function acceptedDoneResult(options: { pendingEvidence?: boolean } = {}) {
const companyId = randomUUID(), agentId = randomUUID(), successorId = randomUUID(), issueId = randomUUID();
const runId = randomUUID(), sessionId = randomUUID(), runnerId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Stale owner fixture", issuePrefix: companyId.slice(0, 8) });
await db.insert(agents).values([
{ id: agentId, companyId, name: "Original owner", adapterType: "paperclip_runner" },
{ id: successorId, companyId, name: "New owner", adapterType: "paperclip_runner" },
]);
await db.insert(issues).values({ id: issueId, companyId, title: "FizzBuzz variants", status: "in_progress",
assigneeAgentId: agentId, description: "Write fizzbuzz with tests and mark done." });
const [contract] = await db.insert(completionContracts).values({ companyId, issueId, revision: 1,
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: options.pendingEvidence ? "standard" : "low",
completionAuthority: options.pendingEvidence ? "server_arbiter" : "agent_claim_policy",
incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: { revision: "test", objective: "Write fizzbuzz", criteria: [{ id: "objective", requirement: "Complete the task" }] },
canonicalSha256: `stale-owner-${issueId}`, createdByActorType: "system", createdByActorId: "test" }).returning();
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, nativeSessionId: sessionId,
runnerInstanceId: runnerId, completionContractId: contract.id, completionContractSha256: contract.canonicalSha256,
status: "running", runtimeMode: "native", contextSnapshot: { issueId, source: "issue.assigned" } });
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
const workProductId = randomUUID();
if (options.pendingEvidence) await db.insert(issueWorkProducts).values({ id: workProductId, companyId, issueId,
type: "artifact", provider: "paperclip", title: "fizzbuzz.py", status: "ready_for_review", reviewState: "none" });
const evidenceRef = options.pendingEvidence ? `work_product:${workProductId}` : `task:${issueId}`;
const port = new NativeRunCoordinatorStore(db, { companyId, issueId, runId, agentId, normalizedSessionId: sessionId,
runnerSourceInstanceId: runnerId, completionContractId: contract.id, completionContractSha256: contract.canonicalSha256,
completionContractRevision: "test", completionContractCriterionIds: ["objective"] });
await port.completeRun({ result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "done",
summary: "Implemented fizzbuzz; 6 passed.", completionClaim: { contractRevision: "test", objectiveSatisfied: true,
criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [evidenceRef] }], remainingWork: [] },
evidence: [{ ref: evidenceRef }],
verification: [{ commandOrCheck: "pytest -q", status: "passed", detail: "6 passed", artifactRef: evidenceRef }], attentionRequests: [], artifacts: [] },
terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed",
reportedWorkDisposition: "done" }, turnId: "fizzbuzz-turn" });
return { companyId, agentId, successorId, issueId, runId, workProductId };
}
/** The board's reassignment: new owner, and the old run loses execution. */
async function reassign(f: Awaited<ReturnType<typeof acceptedDoneResult>>) {
await db.update(issues).set({ assigneeAgentId: f.successorId, executionRunId: null, checkoutRunId: null })
.where(eq(issues.id, f.issueId));
await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "issue_reassigned", finishedAt: new Date() })
.where(eq(heartbeatRuns.id, f.runId));
}
/** The board's Stop, then the user's continue comment starts a new run that holds the task. */
async function stopAndContinue(f: Awaited<ReturnType<typeof acceptedDoneResult>>) {
await db.update(heartbeatRuns).set({ status: "cancelled", errorCode: "cancelled", finishedAt: new Date() })
.where(eq(heartbeatRuns.id, f.runId));
await db.update(issues).set({ executionRunId: null, checkoutRunId: null }).where(eq(issues.id, f.issueId));
const continuationRunId = randomUUID();
await db.insert(heartbeatRuns).values({ id: continuationRunId, companyId: f.companyId, agentId: f.agentId,
nativeIssueId: f.issueId, status: "running", runtimeMode: "native",
contextSnapshot: { issueId: f.issueId, source: "issue.comment" } });
await db.update(issues).set({ executionRunId: continuationRunId, checkoutRunId: continuationRunId })
.where(eq(issues.id, f.issueId));
return continuationRunId;
}
function actBeforeStatusCommit(act: () => Promise<unknown>) {
let acted = false;
const racing = new Proxy(db, {
get(target, property, receiver) {
const value = Reflect.get(target, property, receiver);
if (property !== "update") return typeof value === "function" ? value.bind(target) : value;
return (table: unknown) => {
const builder = target.update(table as typeof nativeRunFinalizations);
if (table !== nativeRunFinalizations || acted) return builder;
const set = builder.set.bind(builder);
return Object.assign(builder, { set: (values: Record<string, unknown>) => {
const next = set(values as never);
if (values.phase !== "arbitrating" || !("assessmentId" in values)) return next;
const where = next.where.bind(next);
return Object.assign(next, { where: (...args: Parameters<typeof where>) => {
const query = where(...args);
return { then: (resolve: (value: unknown) => unknown, reject: (error: unknown) => unknown) =>
query.then(async (result) => { acted = true; await act(); return result; }).then(resolve, reject) };
} });
} });
};
},
}) as Db;
return { racing, acted: () => acted };
}
const issue = async (issueId: string) => (await db.select().from(issues).where(eq(issues.id, issueId)))[0];
const decision = async (runId: string) => {
const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId));
const [row] = await db.select().from(statusDecisions).where(eq(statusDecisions.id, coordinator.decisionId!));
return { coordinator, row };
};
it("closes the task when the reporting run still owns it", async () => {
const f = await acceptedDoneResult();
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: false });
expect(await issue(f.issueId)).toMatchObject({ status: "done", assigneeAgentId: f.agentId });
expect((await decision(f.runId)).row.reasonCode).toBe("completion_claim_policy_accepted");
});
it("does not let a reassigned-away run's done result close the new owner's task", async () => {
const f = await acceptedDoneResult();
const [acceptedBefore] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId));
await reassign(f);
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: false });
const current = await issue(f.issueId);
expect(current).toMatchObject({ status: "in_progress", assigneeAgentId: f.successorId });
const { coordinator, row } = await decision(f.runId);
expect(coordinator.phase).toBe("committed");
expect(row).toMatchObject({ toStatus: "in_progress", reasonCode: "completion_owner_superseded" });
// The original owner's accepted result is kept as history, unchanged.
expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId)))[0]).toEqual(acceptedBefore);
});
it("does not close the task when reassignment lands after arbitration but before the status commit", async () => {
const f = await acceptedDoneResult();
let reassigned = false;
// Reassign at the last finalizer write before the status transaction.
const racing = new Proxy(db, {
get(target, property, receiver) {
const value = Reflect.get(target, property, receiver);
if (property !== "update") return typeof value === "function" ? value.bind(target) : value;
return (table: unknown) => {
const builder = target.update(table as typeof nativeRunFinalizations);
if (table !== nativeRunFinalizations || reassigned) return builder;
const set = builder.set.bind(builder);
return Object.assign(builder, { set: (values: Record<string, unknown>) => {
const next = set(values as never);
if (values.phase !== "arbitrating" || !("assessmentId" in values)) return next;
const where = next.where.bind(next);
return Object.assign(next, { where: (...args: Parameters<typeof where>) => {
const query = where(...args);
return { then: (resolve: (value: unknown) => unknown, reject: (error: unknown) => unknown) =>
query.then(async (result) => { reassigned = true; await reassign(f); return result; }).then(resolve, reject) };
} });
} });
};
},
}) as Db;
await finalizeNativeRun({ db: racing, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: false });
expect(reassigned).toBe(true);
expect(await issue(f.issueId)).toMatchObject({ status: "in_progress", assigneeAgentId: f.successorId });
expect((await decision(f.runId)).row).toMatchObject({ reasonCode: "completion_owner_superseded" });
});
it("does not let a stopped run's saved done result close the task a newer run is continuing", async () => {
const f = await acceptedDoneResult();
const continuationRunId = await stopAndContinue(f);
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: false });
expect(await issue(f.issueId)).toMatchObject({ status: "in_progress", executionRunId: continuationRunId });
const { coordinator, row } = await decision(f.runId);
expect(coordinator.phase).toBe("committed");
expect(row).toMatchObject({ toStatus: "in_progress", reasonCode: "completion_run_superseded" });
});
it("does not close the task when a newer run takes it after arbitration but before the status commit", async () => {
const f = await acceptedDoneResult();
let continuationRunId: string | null = null;
const { racing, acted } = actBeforeStatusCommit(async () => { continuationRunId = await stopAndContinue(f); });
await finalizeNativeRun({ db: racing, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: false });
expect(acted()).toBe(true);
expect(await issue(f.issueId)).toMatchObject({ status: "in_progress", executionRunId: continuationRunId });
expect((await decision(f.runId)).row).toMatchObject({ reasonCode: "completion_run_superseded" });
});
async function approveEvidenceLater(f: Awaited<ReturnType<typeof acceptedDoneResult>>) {
await db.update(issueWorkProducts).set({ reviewState: "approved", updatedAt: new Date(Date.now() + 1_000) })
.where(eq(issueWorkProducts.id, f.workProductId));
}
it("closes the owner's task when later evidence satisfies the contract", async () => {
const f = await acceptedDoneResult({ pendingEvidence: true });
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: false });
expect((await issue(f.issueId)).status).not.toBe("done");
await approveEvidenceLater(f);
await reconcileNativeFinalizations(db, [f.runId]);
expect(await issue(f.issueId)).toMatchObject({ status: "done", assigneeAgentId: f.agentId });
expect((await decision(f.runId)).row.reasonCode).toBe("decision_superseded_by_new_evidence");
});
it("does not let later evidence from a reassigned-away run close the new owner's task", async () => {
const f = await acceptedDoneResult({ pendingEvidence: true });
await reassign(f);
await finalizeNativeRun({ db, runId: f.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: false });
const [acceptedBefore] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId));
await approveEvidenceLater(f);
await reconcileNativeFinalizations(db, [f.runId]);
expect(await issue(f.issueId)).toMatchObject({ status: "in_progress", assigneeAgentId: f.successorId });
expect((await decision(f.runId)).row).toMatchObject({ toStatus: "in_progress", reasonCode: "completion_owner_superseded" });
expect((await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, f.runId)))[0]).toEqual(acceptedBefore);
});
});