Files
PaperClipAI/server/src/services/execution-projection.test.ts
T
DottaandPaperclip 0e9b24c821 fix: resume subscription comment wakes and preserve retry status (#13433)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Concurrent task runs can share a managed AI subscription with one
credential lease.
> - #13438 added durable retries for busy subscriptions and task-lock
checks.
> - A comment wake can run for an agent who is not the task assignee.
Such a run never owns the task lock, so the new checks suppress its
retry.
> - This pull request preserves those comment wakes while keeping the
lock checks for assignee runs.
> - It also shows the subscription wait in task status and preserves the
failure count through the database projection.

## Linked Issues or Issue Description

Refs #13438.

**What happened?**

When a subscription is busy, a non-assignee comment wake is cancelled
without a successor. Repeated subscription waits also display an
inflated attempt count because the execution query omits the preserved
failure count.

**Expected behavior**

An eligible comment wake waits and resumes without claiming the
assignee's task lock. The task shows “Waiting for AI subscription”.
Waiting does not consume provider-failure retries. Assignee retries
still stop when the task lock is cleared or transferred.

**Steps to reproduce**

1. Configure an agent with a managed subscription and concurrent runs.
2. Hold its credential lease in one run.
3. Mention the agent on a task assigned to another actor.
4. Release the lease and inspect whether the comment wake has a
scheduled retry.

The test uses a real embedded Postgres database and a real credential
lease. Provider execution uses a fixture. The reassignment test applies
a database mutation after the real checkout. No live provider account is
needed.

**Paperclip version or commit**

Based on `f912ecaac` from #13438. Before the reconciliation fix, the
rebased regression at `2063cfd13` failed because the comment wake had no
scheduled retry. The original configuration failure was reproduced on
`5282cabde` before #13438 merged.

**Deployment mode**

Built from source with embedded Postgres for local verification.

## What Changed

- Record non-assignee comment-wake authority at admission while holding
the task and run locks. A later reassignment cannot grant this
exception. Preserve it through repeated subscription waits.
- Keep master’s lock checks for assignee retries, including the check
inside the scheduling transaction.
- Show the subscription wait and preserve its failure count in the
execution projection query.
- Limit pre-provider wait receipts to fresh executions. A persisted
native execution input must retain its recovery path.
- Add lease, retry, projection, cancellation, reassignment, pause,
revocation, service-recreation, and ownership regression coverage.
- Use master’s 60–120 second retry interval and document the resulting
behavior.

## Verification

- Red: after rebase, the comment-wake test failed with no scheduled
retry; the other 50 contention and retry-scheduling tests passed.
- Red/green: reassigning the task immediately after real checkout
reproduced an unwanted successor for both assignment and comment wakes.
Both tests pass after recording authority at admission.
- Green: all 285 targeted tests pass across 12 suites, including
#13438’s four cancellation-race phases, its hiring/connection suite,
run-dispatch integration tests, and both reassignment regressions.
- `pnpm -r typecheck` passes after rebase.
- `pnpm build` passes on the final revision.
- The full general and serialized test suites pass in [CI run
34899514681](https://github.com/paperclipai/paperclip/actions/runs/34899514681)
on `99f1e4c1d`. Full-suite verification ran in CI; local verification
used the 285 targeted tests.
- All 32 active PR checks pass, including build, typecheck, native
runner verification, canary dry run, and all browser shards. Two
Storybook checks are skipped by path filters.
- Greptile reviewed `99f1e4c1d` at 5/5 with no outstanding findings. All
review threads are resolved.
- `git diff --check` and `node scripts/check-module-boundaries.mjs`
pass.

## Risks

- The non-assignee exception requires authority recorded under admission
locks or its server-created subscription retry. Tests cover
caller-supplied flags, assignment and comment reassignment races,
repeated comment waits, and assignee lock protections.
- A wait uses master’s 60–120 second interval. A long-held lease can
produce multiple wait records.
- Existing native sessions can have provider effects. They must not
receive a fresh-execution receipt that permits replay.
- No schema migration or credential permission change is required.

## Model Used

OpenAI Codex, model `gpt-6-astra`, with repository search, code editing,
shell execution, and test tools. The session does not expose its
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-14 16:58:37 -05:00

226 lines
8.2 KiB
TypeScript

import { describe, expect, it } from "vitest";
import { projectExecution } from "./execution-projection.js";
type Run = Parameters<typeof projectExecution>[0];
type Coordinator = NonNullable<Parameters<typeof projectExecution>[1]>;
const now = new Date("2026-09-08T12:00:00Z");
const run = (values: Partial<Run> = {}) =>
({
status: "running",
runtimeMode: "native",
errorCode: null,
nextAction: null,
lastUsefulActionAt: now,
lastOutputAt: null,
startedAt: now,
scheduledRetryAt: null,
scheduledRetryAttempt: 1,
retryOfRunId: null,
executionControlDeadlineAt: null,
...values,
}) as Run;
const coordinator = (values: Partial<Coordinator> = {}) =>
({
phase: "observed",
attempt: 1,
failureCode: null,
failureDetail: {},
nextAttemptAt: null,
leaseExpiresAt: new Date(now.getTime() + 30_000),
controlDeadlineAt: null,
...values,
}) as Coordinator;
const project = (
r = run(),
c: Coordinator | undefined = coordinator(),
pending: { kind: string }[] = [],
) => projectExecution(r, c, pending, undefined, now);
describe("execution truth projection", () => {
it("shows subscription contention as a resource wait without failed provider attempts", () => {
expect(projectExecution(run({ runtimeMode: "legacy", status: "scheduled_retry", scheduledRetryReason: "ai_connection_busy",
scheduledRetryAttempt: 12, contextSnapshot: { failureRetriesBeforeAiConnectionWait: 0 } }), undefined, [], undefined, now))
.toMatchObject({ label: "Waiting for AI subscription", phase: "retry_scheduled", attempt: 1, recoveryOwner: null });
});
it("shows a workspace wait without presenting its deferral count as failed attempts", () => {
expect(projectExecution(run({ runtimeMode: "legacy", status: "scheduled_retry", scheduledRetryReason: "workspace_busy",
scheduledRetryAttempt: 12, contextSnapshot: { failureRetriesBeforeWorkspaceWait: 1 } }), undefined, [], undefined, now))
.toMatchObject({ label: "Waiting for workspace", phase: "retry_scheduled", attempt: 2, recoveryOwner: null });
});
it.each([
{ status: "resolved", previousRunId: "old", nextRunId: "next", continued: true },
{ status: "active", previousRunId: "old", nextRunId: "next", continued: false },
{ status: "resolved", previousRunId: "other", nextRunId: "next", continued: false },
{ status: "resolved", previousRunId: "old", nextRunId: "old", continued: false },
{ status: "resolved", previousRunId: "old", nextRunId: null, continued: false },
])("projects the recorded explicit successor without hiding unresolved recovery: $status/$previousRunId/$nextRunId", ({ status, previousRunId, nextRunId, continued }) => {
const source = run({ id: "old", status: "failed", errorCode: "adapter_failed" });
expect(projectExecution(source, coordinator({ phase: "terminal_failure" }), [], {
status,
cause: "native_continuation_requires_reconciliation",
nextAction: "Inspect the stopped execution.",
evidence: {
automaticRecovery: { policy: "preserve_without_replay_v1" },
explicitUserContinuation: { previousRunId, runId: nextRunId },
},
}, now)).toMatchObject(continued ? {
phase: "completed", label: "Continued in another run", successorRunId: "next",
cause: "native_continuation_requires_reconciliation", nextAction: null,
} : { phase: "recovery_needed", successorRunId: null });
expect(source.status).toBe("failed");
});
it("shows a reconciled continuation as queued until its durable delivery is recorded", () => {
const action = {
cause: "native_session_retry_exhausted",
nextAction: "Old recovery action",
status: "resolved" as const,
evidence: {
executionReconciliation: { runId: "old" },
continuationDelivery: "pending",
},
};
expect(
projectExecution(
run({ status: "failed" }),
coordinator({ phase: "terminal_failure" }),
[],
action,
now,
),
).toMatchObject({
phase: "queued",
label: "Continuation queued",
cause: null,
});
expect(
projectExecution(
run({ status: "failed" }),
coordinator({ phase: "terminal_failure" }),
[],
{
...action,
evidence: { ...action.evidence, continuationRunId: "next" },
},
now,
),
).toMatchObject({ phase: "completed", successorRunId: "next" });
expect(
projectExecution(
run({ status: "failed" }),
coordinator({ phase: "terminal_failure" }),
[],
{
...action,
evidence: { ...action.evidence, continuationDelivery: "invalidated" },
},
now,
),
).toMatchObject({ phase: "failed", label: "Continuation cancelled" });
});
it("requires current execution evidence, without treating output silence as failure", () => {
expect(project(run({ lastOutputAt: new Date(0) })).phase).toBe("working");
expect(
project(run(), coordinator({ leaseExpiresAt: new Date(0) })),
).toMatchObject({ phase: "reconnecting", label: "Confirming execution" });
expect(
project(
run({ runtimeMode: "legacy", processPid: process.pid }),
undefined,
).phase,
).toBe("working");
});
it("distinguishes provider work, finalization and timed retry", () => {
expect(
project(
run({ executionControlDeadlineAt: new Date(now.getTime() + 60_000) }),
).phase,
).toBe("finishing");
expect(
project(
run({ status: "failed" }),
coordinator({
phase: "retryable_failure",
nextAttemptAt: new Date(now.getTime() + 30_000),
attempt: 2,
}),
),
).toMatchObject({ phase: "retry_scheduled", attempt: 2, maxAttempts: 3 });
expect(
project(
run({ status: "failed" }),
coordinator({ phase: "retryable_failure", nextAttemptAt: new Date(0) }),
).phase,
).toBe("reconnecting");
});
it("includes the original legacy attempt in the displayed incident budget", () => {
expect(projectExecution(run({ runtimeMode: "legacy", status: "scheduled_retry", scheduledRetryAttempt: 1 }), undefined, [], undefined, now)).toMatchObject({ attempt: 2, maxAttempts: 3 });
});
it("shows pending interaction only after productive work has stopped", () => {
expect(
project(run(), coordinator(), [{ kind: "connection_intent" }]).phase,
).toBe("working");
expect(
project(run({ status: "succeeded" }), undefined, [
{ kind: "connection_intent" },
]).phase,
).toBe("waiting_for_access");
expect(
project(run({ status: "succeeded" }), undefined, [
{ kind: "ask_user_questions" },
]).phase,
).toBe("waiting_for_answer");
});
it("retains replacement lineage and exposes operator-owned failures for legacy runs", () => {
expect(
project(
run({ status: "failed" }),
coordinator({
phase: "terminal_failure",
failureDetail: { successorRunId: "replacement" },
}),
),
).toMatchObject({ phase: "completed", successorRunId: "replacement" });
expect(
projectExecution(
run({ runtimeMode: "legacy", status: "failed" }),
undefined,
[],
{
cause: "uncertain_external_action",
nextAction: "Reconcile the email delivery.",
},
now,
),
).toMatchObject({
phase: "recovery_needed",
recoveryOwner: "board",
nextAction: "Reconcile the email delivery.",
});
});
it("bounds recovery checking and surfaces its next action if verification never finishes", () => {
const c = coordinator({
phase: "terminal_failure",
failureCode: "native_provider_terminal_failed",
});
expect(project(run({ status: "failed", finishedAt: now }), c).label).toBe(
"Checking recovery",
);
expect(
project(
run({ status: "failed", finishedAt: new Date(now.getTime() - 60_000) }),
c,
).phase,
).toBe("recovery_needed");
expect(
project(
run({ status: "failed", finishedAt: now }),
coordinator({
...c,
failureDetail: { replacementDenied: "uncertain_provider_action" },
}),
).phase,
).toBe("recovery_needed");
});
});