mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 04:21:24 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a server swap or a failure interrupts an agent's task, recovery
continues the task automatically: the assigned-work sweep, the
scheduler's bounded retries and native session restart recovery.
> - Release-swap testing on staging found four gaps in that recovery.
The assigned-work report took 60–81 s. One lasting outage was retried by
two owners under two separate caps. A server with a fast clock could
take a session lease that another server still held. Each server
generation started its own 15-minute recovery wait, so the waits chained
for hours.
> - Each gap either slows recovery down or lets a task loop or stall
without any limit.
> - This pull request batches the report's reads, gives each task one
shared hourly cap for automatic attempts, uses the database clock to
judge a lease, and gives each interruption one deadline across server
generations.
> - The benefit is that recovery stays fast and bounded. A lasting
outage gets at most six automatic attempts an hour. A run whose provider
makes no progress ends after 15 minutes, and the fallback then continues
it in isolation.
## Linked Issues or Issue Description
**What happened?**
1. On staging, the assigned-work invariant report took 60–81 s with more
than 1,000 open agent tasks. For each task with nothing live or queued,
the report read the tree pause gate in sequence: one round trip, plus
one per ancestor while a pause hold existed. It also read the company's
agent list once per assignee.
2. A lasting repository catalog outage was retried by two owners. Each
sweep continuation started a fresh run. Each failure of that run got two
more bounded transient retries from the scheduler. One failure could
start up to 18 runs an hour.
3. `claimNativeSessionResumptions` compared the lease expiry with the
claiming server's own clock. A server whose clock ran fast could take a
lease that another server still held.
4. Each server generation's restart-recovery claim started its own
15-minute wait for the provider to recover. Swaps came every 10–14
minutes, so the waits chained for up to 12 generations and none of them
expired.
**Expected behavior**
The report reads pause holds in a fixed number of statements. One task
gets at most one hourly cap of automatic attempts across all owners. The
database clock judges a lease. An interruption with no provider progress
ends after one 15-minute deadline, whatever the number of server
generations.
**Steps to reproduce**
1. Seed 1,500 open tasks with 300 held roots and 400 queued wakes, then
run the invariant report. Before this change it used 2,279 statements.
2. Make the repository catalog unavailable for longer than an hour on an
assigned task. Count the runs.
3. Let a server whose clock is ten minutes fast claim a native session
resumption whose lease is still live.
4. Replace the server every 7 minutes while a native run's provider
makes no progress.
## What Changed
- `issue-tree-control.ts`: a new batched pause-hold read. For a set of
tasks it uses at most two statements: one to find any active hold, then
one recursive ancestor read. It has the same depth bound and cycle guard
as the per-issue gate. `assigned-work-sweep.ts` uses it, and reads the
org chart once per batch to judge each assignee.
- New `recovery/task-automatic-attempts.ts`: one per-task ledger of
automatic attempts in the last hour (sweep continuations plus the
scheduler's `transient_failure` retries). The sweep counts both for its
cap and its spacing (`assigned-work-sweep.ts`). The scheduler's
transient retry lane (`heartbeat/retries.ts`) reads the same ledger.
Once the sweep has continued a task in the hour, a transient retry that
would exceed the cap of six is reported as exhausted. If the sweep never
continued a task, the scheduler keeps its own chain bound.
- `native-finalization-reconciler.ts` (`claimNativeSessionResumptions`):
the compare-and-set also requires the other owner's lease to have lapsed
by `clock_timestamp()`.
- `native-restart-recovery.ts`: each restart-recovery claim records
`interruptionStartedAt` in its recovery-history entry. A later claim
keeps that value while the interrupted attempt shows no provider
progress, and resets it when there is progress. A claim made 15 minutes
or more after the interruption began, with no progress, ends the run as
no longer resumable. This uses the existing terminal path (run failed,
slots and task lock freed), so the fresh-continuation fallback continues
the task in isolation. The bound is
`NATIVE_INTERRUPTION_NO_PROGRESS_DEADLINE_MS` in
`fresh-continuation-fallback.ts`, set equal to the fallback's
exact-recovery cap. Coordinators saved by older releases have no start
time, so the first claim by this code starts the clock. The normal run
path is not changed.
- Review fixes:
- A graceful drain no longer counts as "no progress". The claim always
reads provider progress. The drain only decides whether the resume
spends an attempt, so a run that progressed before a deploy is never
ended by the deadline.
- The transient lane checks the hourly cap again inside the task-locked
transaction, after it reuses an existing successor and before it inserts
a new one. The sweep already reads the ledger under the same lock.
- A same-run resume claim that the database clock rejects (another
owner's lease is still live) no longer pushes that owner's
`nextAttemptAt` out.
- Code comments that the original commits added are removed
(CONTRIBUTING). Merged with current master (includes #21336):
`assigned-work-sweep.ts` keeps master's `BLOCK_ATTRIBUTION_WINDOW_MS`
import and adds the ledger import.
## Verification
Red-to-green tests, each red on the parent commit and green with the
fix:
- `server/src/__tests__/assigned-work-sweep-round-trips.test.ts` "reads
pause holds and assignees once per batch on 1,500 open tasks…": 1,500
tasks, 300 held roots plus 300 children, 400 queued wakes, 20 agents.
Red: 1,100 per-task gate reads and 2,279 statements. Green: 0 and 239.
Report verdict counts are asserted to be the same before and after.
- Same file, "the batched pause read gives the per-issue gate's answer:
deep chains, cycles, released holds, other companies": checks that the
batched read gives the same result as the per-issue gate.
- `server/src/__tests__/assigned-work-invariant.test.ts` "counts the
scheduler's transient retries of the same task against the one
continuation cap": red (repaired), green (deferred_cap).
- `server/src/services/heartbeat/retries.test.ts` "a lasting outage gets
at most the task's hourly cap of automatic attempts across the sweep and
the transient lane".
- `server/src/__tests__/native-session-resumption.test.ts` "judges
another owner's lease by the database clock, not a server clock that
runs fast".
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"resumes a run that progressed for 20 minutes before a graceful drain
suspended it" (red before: the deadline ended it) and "still ends a run
with no provider progress at the deadline when a graceful drain
suspended it".
- `server/src/services/heartbeat/retries.test.ts` "re-checks the task's
hourly cap under the task lock when the sweep records an attempt while
the retry is being prepared" (red before: attempt seven was scheduled).
- `server/src/__tests__/native-session-resumption.test.ts`: the
clock-skew test also asserts that `nextAttemptAt` is not pushed out (red
before).
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"ends a run whose generations show no progress at the deadline from the
first interruption, not per generation", "restarts the clock when a
generation's resume made provider progress", "starts the clock at the
first claim for a coordinator saved before the deadline existed".
Local runs on this branch (based on master 566843820c):
- `cd server && pnpm exec vitest run <file>`, one file at a time:
- `src/__tests__/assigned-work-sweep-round-trips.test.ts`: 2 passed
- `src/__tests__/assigned-work-invariant.test.ts`: 31 passed
- `src/services/heartbeat/retries.test.ts`: 30 passed
- `src/__tests__/native-session-resumption.test.ts`: 20 passed
-
`src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
217 passed, 14 skipped
- `src/__tests__/heartbeat-native-restart-suspension.test.ts`: 115
passed
- `cd server && pnpm run typecheck` (CI's layered typecheck, including
test files): clean.
- `cd server && pnpm exec tsc --noEmit`: clean.
- Timing note: local round trips take about 0.2 ms, so local
milliseconds do not show the gain (736 ms became 303–583 ms on a loaded
machine). At staging's ~50 ms per round trip, the ~2,040 sequential
statements removed are about 100 s.
- No existing test expectation was changed.
## Risks
- After this change, a task the sweep continued in the last hour can get
fewer transient retries. This is intended: one task gets at most six
automatic attempts an hour. A person's Retry is not counted in this
ledger.
- The interruption deadline ends a native run that shows no provider
progress for 15 minutes across server swaps. The fallback then continues
it in a fresh isolated workspace, so the task is not left for a person
to fix. A run that shows progress resets the clock.
- Coordinators saved by older releases start the clock at their first
claim by this code, so an upgrade does not expire any run immediately.
- No migration. No schema change. The new history field is inside
existing JSON.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1380 lines
52 KiB
TypeScript
1380 lines
52 KiB
TypeScript
import { notifyStrandedAgentStatus } from "./live-run-recovery-notifications.js";
|
|
import { promoteDeferredWorkIntents } from "./work-intent-outbox.js";
|
|
import { notifyStatusCardIssueChange } from "./status-card-notifications.js";
|
|
import { notifyRoutineExecutionStatus } from "./routine-execution-status-notifications.js";
|
|
import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js";
|
|
import { notifyIssueRecovery } from "./issue-recovery-notifications.js";
|
|
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
|
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
|
import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js";
|
|
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
|
|
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
|
|
import { and, asc, eq, inArray, isNull, notInArray, or, sql } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import {
|
|
agentWakeupRequests,
|
|
heartbeatRuns,
|
|
issueComments,
|
|
issueTreeHoldMembers,
|
|
issueTreeHolds,
|
|
issues,
|
|
} from "@paperclipai/db";
|
|
import {
|
|
ISSUE_STATUSES,
|
|
type IssueStatus,
|
|
type IssueTreeControlMode,
|
|
type IssueTreeControlPreview,
|
|
type IssueTreeHold,
|
|
type IssueTreeHoldMember,
|
|
type IssueTreeHoldReleasePolicy,
|
|
type IssueTreePreviewAgent,
|
|
type IssueTreePreviewIssue,
|
|
type IssueTreePreviewRun,
|
|
type IssueTreePreviewWarning,
|
|
} from "@paperclipai/shared";
|
|
import { conflict, notFound, unprocessable } from "../errors.js";
|
|
import type { IssuePostCommitAction } from "./issues.js";
|
|
import { notifyTerminalWorkspaceWork } from "./terminal-workspace-work.js";
|
|
import { readRunCancellation } from "./run-cancellation.js";
|
|
|
|
type IssueRow = typeof issues.$inferSelect;
|
|
type HoldRow = typeof issueTreeHolds.$inferSelect;
|
|
type HoldMemberRow = typeof issueTreeHoldMembers.$inferSelect;
|
|
export type ActiveIssueTreePauseHoldGate = {
|
|
holdId: string;
|
|
rootIssueId: string;
|
|
issueId: string;
|
|
isRoot: boolean;
|
|
mode: "pause";
|
|
reason: string | null;
|
|
releasePolicy: IssueTreeHoldReleasePolicy | null;
|
|
};
|
|
type ActorInput = {
|
|
actorType: "user" | "agent" | "system";
|
|
actorId: string;
|
|
agentId?: string | null;
|
|
userId?: string | null;
|
|
runId?: string | null;
|
|
};
|
|
type TreeIssue = IssueRow & { depth: number };
|
|
type ActiveRunRow = {
|
|
id: string;
|
|
issueId: string;
|
|
agentId: string;
|
|
status: "queued" | "running";
|
|
startedAt: Date | null;
|
|
createdAt: Date;
|
|
};
|
|
type ActiveCancelSnapshot = {
|
|
holdIds: string[];
|
|
member: IssueTreeHoldMember | null;
|
|
};
|
|
type TreeStatusUpdateResult = {
|
|
updatedIssueIds: string[];
|
|
updatedIssues: Array<{
|
|
id: string;
|
|
status: IssueStatus;
|
|
assigneeAgentId: string | null;
|
|
}>;
|
|
};
|
|
type RestoreTreeStatusResult = TreeStatusUpdateResult & {
|
|
releasedCancelHoldIds: string[];
|
|
restoreHold: IssueTreeHold | null;
|
|
};
|
|
|
|
const TERMINAL_ISSUE_STATUSES = new Set<IssueStatus>(["done", "cancelled"]);
|
|
export const ISSUE_TREE_RESUME_EXECUTABLE_STATUSES = ["todo", "in_progress", "in_review"];
|
|
|
|
/** The error code a dispatch gate records when an active pause hold refuses a run. */
|
|
export const ISSUE_PAUSED_ERROR_CODE = "issue_paused";
|
|
|
|
/**
|
|
* Recorded on a run a dispatch gate refused because a pause hold covered its
|
|
* task, so that hold's release can re-dispatch the refused work once.
|
|
*/
|
|
export function pauseSuppressionResult(hold: { holdId: string; rootIssueId?: string | null }) {
|
|
return { treeHoldSuppression: { holdId: hold.holdId, rootIssueId: hold.rootIssueId ?? null } };
|
|
}
|
|
|
|
function readPauseSuppressionHoldId(resultJson: Record<string, unknown> | null) {
|
|
const marker = resultJson?.treeHoldSuppression;
|
|
if (!marker || typeof marker !== "object" || Array.isArray(marker)) return null;
|
|
const holdId = (marker as Record<string, unknown>).holdId;
|
|
return typeof holdId === "string" && holdId.length > 0 ? holdId : null;
|
|
}
|
|
const ACTIVE_RUN_STATUSES = ["queued", "running"] as const;
|
|
const DEFAULT_RELEASE_POLICY: IssueTreeHoldReleasePolicy = { strategy: "manual" };
|
|
const MAX_PAUSE_HOLD_ANCESTOR_DEPTH = 100;
|
|
export const ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS: ReadonlySet<string> = new Set([
|
|
"issue_commented",
|
|
"issue_reopened_via_comment",
|
|
"issue_comment_mentioned",
|
|
] as const);
|
|
const ISSUE_TREE_CONTROL_INTERACTION_WAKE_SOURCES: Readonly<Record<string, ReadonlySet<string>>> = {
|
|
issue_commented: new Set(["issue.comment"]),
|
|
issue_reopened_via_comment: new Set(["issue.comment.reopen"]),
|
|
issue_comment_mentioned: new Set(["comment.mention"]),
|
|
};
|
|
|
|
type VerifiedInteractionActor = {
|
|
requestedByActorType?: string | null;
|
|
requestedByActorId?: string | null;
|
|
};
|
|
|
|
function readNonEmptyStringFromRecord(record: unknown, key: string) {
|
|
if (!record || typeof record !== "object") return null;
|
|
const value = (record as Record<string, unknown>)[key];
|
|
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
|
|
}
|
|
|
|
function readInteractionWakeCommentId(record: unknown) {
|
|
if (!record || typeof record !== "object") return null;
|
|
const value = (record as Record<string, unknown>).wakeCommentIds;
|
|
if (Array.isArray(value)) {
|
|
const latest = value
|
|
.filter((entry): entry is string => typeof entry === "string" && entry.trim().length > 0)
|
|
.at(-1);
|
|
if (latest) return latest.trim();
|
|
}
|
|
return readNonEmptyStringFromRecord(record, "wakeCommentId") ?? readNonEmptyStringFromRecord(record, "commentId");
|
|
}
|
|
|
|
function hasVerifiedInteractionSource(wakeReason: string, contextSnapshot: Record<string, unknown>) {
|
|
const source = readNonEmptyStringFromRecord(contextSnapshot, "source");
|
|
if (!source) return false;
|
|
return ISSUE_TREE_CONTROL_INTERACTION_WAKE_SOURCES[wakeReason]?.has(source) ?? false;
|
|
}
|
|
|
|
function actorMatchesComment(
|
|
actor: VerifiedInteractionActor,
|
|
comment: { authorAgentId: string | null; authorUserId: string | null },
|
|
) {
|
|
if (!actor.requestedByActorType) return false;
|
|
if (actor.requestedByActorType === "system") return true;
|
|
if (!actor.requestedByActorId) return false;
|
|
if (actor.requestedByActorType === "agent") return comment.authorAgentId === actor.requestedByActorId;
|
|
if (actor.requestedByActorType === "user") return comment.authorUserId === actor.requestedByActorId;
|
|
return false;
|
|
}
|
|
|
|
async function hasVerifiedInteractionWakeRequest(
|
|
dbOrTx: Pick<Db, "select">,
|
|
input: {
|
|
companyId: string;
|
|
agentId?: string | null;
|
|
runId?: string | null;
|
|
wakeupRequestId?: string | null;
|
|
issueId: string;
|
|
commentId: string;
|
|
comment: { authorAgentId: string | null; authorUserId: string | null };
|
|
},
|
|
) {
|
|
if (!input.runId && !input.wakeupRequestId) return false;
|
|
const predicates = [
|
|
eq(agentWakeupRequests.companyId, input.companyId),
|
|
sql`${agentWakeupRequests.payload} ->> 'issueId' = ${input.issueId}`,
|
|
sql`${agentWakeupRequests.payload} ->> 'commentId' = ${input.commentId}`,
|
|
];
|
|
if (input.agentId) predicates.push(eq(agentWakeupRequests.agentId, input.agentId));
|
|
if (input.runId && input.wakeupRequestId) {
|
|
const requestScope = or(
|
|
eq(agentWakeupRequests.runId, input.runId),
|
|
eq(agentWakeupRequests.id, input.wakeupRequestId),
|
|
);
|
|
if (requestScope) predicates.push(requestScope);
|
|
} else if (input.runId) {
|
|
predicates.push(eq(agentWakeupRequests.runId, input.runId));
|
|
} else if (input.wakeupRequestId) {
|
|
predicates.push(eq(agentWakeupRequests.id, input.wakeupRequestId));
|
|
}
|
|
|
|
const requests = await dbOrTx
|
|
.select({
|
|
requestedByActorType: agentWakeupRequests.requestedByActorType,
|
|
requestedByActorId: agentWakeupRequests.requestedByActorId,
|
|
})
|
|
.from(agentWakeupRequests)
|
|
.where(and(...predicates));
|
|
|
|
return requests.some((request) => actorMatchesComment(request, input.comment));
|
|
}
|
|
|
|
export async function isVerifiedIssueTreeControlInteractionWake(
|
|
dbOrTx: Pick<Db, "select">,
|
|
input: {
|
|
companyId: string;
|
|
issueId: string;
|
|
agentId?: string | null;
|
|
contextSnapshot: Record<string, unknown> | null | undefined;
|
|
requestedByActorType?: "user" | "agent" | "system" | string | null;
|
|
requestedByActorId?: string | null;
|
|
runId?: string | null;
|
|
wakeupRequestId?: string | null;
|
|
},
|
|
) {
|
|
const contextSnapshot = input.contextSnapshot ?? null;
|
|
const wakeReason =
|
|
readNonEmptyStringFromRecord(contextSnapshot, "wakeReason") ??
|
|
readNonEmptyStringFromRecord(contextSnapshot, "reason");
|
|
if (!wakeReason || !ISSUE_TREE_CONTROL_INTERACTION_WAKE_REASONS.has(wakeReason)) return false;
|
|
if (!contextSnapshot || !hasVerifiedInteractionSource(wakeReason, contextSnapshot)) return false;
|
|
|
|
const commentId = readInteractionWakeCommentId(contextSnapshot);
|
|
if (!commentId) return false;
|
|
|
|
const comment = await dbOrTx
|
|
.select({
|
|
id: issueComments.id,
|
|
authorAgentId: issueComments.authorAgentId,
|
|
authorUserId: issueComments.authorUserId,
|
|
})
|
|
.from(issueComments)
|
|
.where(
|
|
and(
|
|
eq(issueComments.companyId, input.companyId),
|
|
eq(issueComments.issueId, input.issueId),
|
|
eq(issueComments.id, commentId),
|
|
),
|
|
)
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!comment) return false;
|
|
|
|
const directActor = {
|
|
requestedByActorType: input.requestedByActorType,
|
|
requestedByActorId: input.requestedByActorId,
|
|
};
|
|
if (actorMatchesComment(directActor, comment)) return true;
|
|
|
|
return hasVerifiedInteractionWakeRequest(dbOrTx, {
|
|
companyId: input.companyId,
|
|
agentId: input.agentId,
|
|
runId: input.runId,
|
|
wakeupRequestId: input.wakeupRequestId,
|
|
issueId: input.issueId,
|
|
commentId,
|
|
comment,
|
|
});
|
|
}
|
|
|
|
function normalizeReleasePolicy(
|
|
releasePolicy: IssueTreeHoldReleasePolicy | null | undefined,
|
|
): IssueTreeHoldReleasePolicy {
|
|
return releasePolicy ?? DEFAULT_RELEASE_POLICY;
|
|
}
|
|
|
|
function coerceIssueStatus(status: string): IssueStatus {
|
|
return ISSUE_STATUSES.includes(status as IssueStatus) ? (status as IssueStatus) : "backlog";
|
|
}
|
|
|
|
function isTerminalIssue(status: string): status is IssueStatus {
|
|
return TERMINAL_ISSUE_STATUSES.has(coerceIssueStatus(status));
|
|
}
|
|
|
|
function toPreviewRun(row: ActiveRunRow): IssueTreePreviewRun {
|
|
return {
|
|
id: row.id,
|
|
issueId: row.issueId,
|
|
agentId: row.agentId,
|
|
status: row.status,
|
|
startedAt: row.startedAt,
|
|
createdAt: row.createdAt,
|
|
};
|
|
}
|
|
|
|
function toHold(row: HoldRow, members?: HoldMemberRow[]): IssueTreeHold {
|
|
return {
|
|
id: row.id,
|
|
companyId: row.companyId,
|
|
rootIssueId: row.rootIssueId,
|
|
mode: row.mode as IssueTreeControlMode,
|
|
status: row.status as IssueTreeHold["status"],
|
|
reason: row.reason,
|
|
releasePolicy: (row.releasePolicy as IssueTreeHoldReleasePolicy | null) ?? null,
|
|
createdByActorType: row.createdByActorType as IssueTreeHold["createdByActorType"],
|
|
createdByAgentId: row.createdByAgentId,
|
|
createdByUserId: row.createdByUserId,
|
|
createdByRunId: row.createdByRunId,
|
|
releasedAt: row.releasedAt,
|
|
releasedByActorType: row.releasedByActorType as IssueTreeHold["releasedByActorType"],
|
|
releasedByAgentId: row.releasedByAgentId,
|
|
releasedByUserId: row.releasedByUserId,
|
|
releasedByRunId: row.releasedByRunId,
|
|
releaseReason: row.releaseReason,
|
|
releaseMetadata: row.releaseMetadata ?? null,
|
|
createdAt: row.createdAt,
|
|
updatedAt: row.updatedAt,
|
|
...(members ? { members: members.map(toHoldMember) } : {}),
|
|
};
|
|
}
|
|
|
|
function toHoldMember(row: HoldMemberRow): IssueTreeHoldMember {
|
|
return {
|
|
id: row.id,
|
|
companyId: row.companyId,
|
|
holdId: row.holdId,
|
|
issueId: row.issueId,
|
|
parentIssueId: row.parentIssueId,
|
|
depth: row.depth,
|
|
issueIdentifier: row.issueIdentifier,
|
|
issueTitle: row.issueTitle,
|
|
issueStatus: coerceIssueStatus(row.issueStatus),
|
|
assigneeAgentId: row.assigneeAgentId,
|
|
assigneeUserId: row.assigneeUserId,
|
|
activeRunId: row.activeRunId,
|
|
activeRunStatus: row.activeRunStatus,
|
|
skipped: row.skipped,
|
|
skipReason: row.skipReason,
|
|
createdAt: row.createdAt,
|
|
};
|
|
}
|
|
|
|
function issueSkipReason(input: {
|
|
mode: IssueTreeControlMode;
|
|
issue: TreeIssue;
|
|
activePauseHoldIds: string[];
|
|
activeCancelSnapshot?: ActiveCancelSnapshot | null;
|
|
}): string | null {
|
|
const status = coerceIssueStatus(input.issue.status);
|
|
if (input.mode === "restore") {
|
|
if (input.activeCancelSnapshot?.member && status !== "cancelled") {
|
|
return "changed_after_cancel";
|
|
}
|
|
if (status !== "cancelled") return "not_cancelled";
|
|
if (!input.activeCancelSnapshot?.member) return "not_cancelled_by_tree_control";
|
|
const snapshotStatus = coerceIssueStatus(input.activeCancelSnapshot.member.issueStatus);
|
|
return isTerminalIssue(snapshotStatus) ? "terminal_status" : null;
|
|
}
|
|
if (isTerminalIssue(status)) {
|
|
return "terminal_status";
|
|
}
|
|
if (input.mode === "pause" && input.activePauseHoldIds.length > 0) {
|
|
return "already_held";
|
|
}
|
|
if (input.mode === "resume" && input.activePauseHoldIds.length === 0) {
|
|
return "not_held";
|
|
}
|
|
return null;
|
|
}
|
|
|
|
function canWakeOnResume(issue: IssueTreePreviewIssue): boolean {
|
|
return !!issue.assigneeAgentId && ISSUE_TREE_RESUME_EXECUTABLE_STATUSES.includes(issue.status);
|
|
}
|
|
|
|
function buildAffectedAgents(
|
|
issuesToPreview: IssueTreePreviewIssue[],
|
|
mode: IssueTreeControlMode,
|
|
): IssueTreePreviewAgent[] {
|
|
const byAgentId = new Map<string, IssueTreePreviewAgent>();
|
|
for (const issue of issuesToPreview) {
|
|
if (issue.skipped) continue;
|
|
if (mode === "resume" && !canWakeOnResume(issue)) continue;
|
|
const agentIds = new Set<string>();
|
|
if (issue.assigneeAgentId) agentIds.add(issue.assigneeAgentId);
|
|
if (mode !== "resume" && issue.activeRun) agentIds.add(issue.activeRun.agentId);
|
|
for (const agentId of agentIds) {
|
|
const current = byAgentId.get(agentId) ?? { agentId, issueCount: 0, activeRunCount: 0 };
|
|
current.issueCount += 1;
|
|
if (issue.activeRun?.agentId === agentId) current.activeRunCount += 1;
|
|
byAgentId.set(agentId, current);
|
|
}
|
|
}
|
|
return [...byAgentId.values()].sort((a, b) => a.agentId.localeCompare(b.agentId));
|
|
}
|
|
|
|
function buildWarnings(input: {
|
|
mode: IssueTreeControlMode;
|
|
issuesToPreview: IssueTreePreviewIssue[];
|
|
activeRuns: IssueTreePreviewRun[];
|
|
}): IssueTreePreviewWarning[] {
|
|
const affectedIssues = input.issuesToPreview.filter((issue) => !issue.skipped);
|
|
const affectedIssueIds = new Set(affectedIssues.map((issue) => issue.id));
|
|
const affectedRuns = input.activeRuns.filter((run) => affectedIssueIds.has(run.issueId));
|
|
const warnings: IssueTreePreviewWarning[] = [];
|
|
|
|
if (affectedIssues.length === 0) {
|
|
warnings.push({
|
|
code: "no_affected_issues",
|
|
message: "No issues in this subtree match the requested control action.",
|
|
});
|
|
}
|
|
|
|
const runningRunIssueIds = affectedRuns
|
|
.filter((run) => run.status === "running")
|
|
.map((run) => run.issueId);
|
|
if ((input.mode === "pause" || input.mode === "cancel") && runningRunIssueIds.length > 0) {
|
|
warnings.push({
|
|
code: "running_runs_present",
|
|
message: "Some affected issues have running heartbeat runs.",
|
|
issueIds: [...new Set(runningRunIssueIds)].sort(),
|
|
});
|
|
}
|
|
|
|
const queuedRunIssueIds = affectedRuns
|
|
.filter((run) => run.status === "queued")
|
|
.map((run) => run.issueId);
|
|
if ((input.mode === "pause" || input.mode === "cancel") && queuedRunIssueIds.length > 0) {
|
|
warnings.push({
|
|
code: "queued_runs_present",
|
|
message: "Some affected issues have queued heartbeat runs.",
|
|
issueIds: [...new Set(queuedRunIssueIds)].sort(),
|
|
});
|
|
}
|
|
|
|
if (input.mode === "resume" && affectedIssues.length === 0) {
|
|
warnings.push({
|
|
code: "no_active_pause_holds",
|
|
message: "No active pause holds were found in this subtree.",
|
|
});
|
|
}
|
|
|
|
if (input.mode === "resume") {
|
|
const idleIssueIds = affectedIssues.filter((issue) => !canWakeOnResume(issue)).map((issue) => issue.id);
|
|
if (idleIssueIds.length > 0) {
|
|
warnings.push({
|
|
code: "resume_tasks_remain_idle",
|
|
message: "Blocked, backlog, and unassigned tasks will be unpaused but will not start.",
|
|
issueIds: idleIssueIds.sort(),
|
|
});
|
|
}
|
|
}
|
|
|
|
if (input.mode === "restore") {
|
|
const changedIssueIds = input.issuesToPreview
|
|
.filter((issue) => issue.skipReason === "changed_after_cancel")
|
|
.map((issue) => issue.id);
|
|
if (changedIssueIds.length > 0) {
|
|
warnings.push({
|
|
code: "restore_conflicts_present",
|
|
message: "Some issues changed after subtree cancellation and will be skipped.",
|
|
issueIds: changedIssueIds,
|
|
});
|
|
}
|
|
}
|
|
|
|
return warnings;
|
|
}
|
|
|
|
function restoreStatusFromCancelSnapshot(status: IssueStatus): IssueStatus | null {
|
|
if (status === "in_progress") return "todo";
|
|
if (isTerminalIssue(status)) return null;
|
|
return status;
|
|
}
|
|
|
|
export function issueTreeControlService(db: Db) {
|
|
async function listTreeIssues(companyId: string, rootIssueId: string): Promise<TreeIssue[]> {
|
|
const root = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(and(eq(issues.id, rootIssueId), eq(issues.companyId, companyId)))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!root) {
|
|
throw notFound("Root issue not found");
|
|
}
|
|
|
|
const result: TreeIssue[] = [{ ...root, depth: 0 }];
|
|
const visited = new Set<string>([root.id]);
|
|
let frontier = [{ id: root.id, depth: 0 }];
|
|
|
|
while (frontier.length > 0) {
|
|
const parentIds = frontier.map((item) => item.id);
|
|
const depthByParentId = new Map(frontier.map((item) => [item.id, item.depth]));
|
|
const children = await db
|
|
.select()
|
|
.from(issues)
|
|
.where(and(eq(issues.companyId, companyId), inArray(issues.parentId, parentIds)))
|
|
.orderBy(asc(issues.createdAt), asc(issues.id));
|
|
|
|
const nextFrontier: typeof frontier = [];
|
|
for (const child of children) {
|
|
if (visited.has(child.id)) continue;
|
|
const depth = (depthByParentId.get(child.parentId ?? "") ?? 0) + 1;
|
|
visited.add(child.id);
|
|
result.push({ ...child, depth });
|
|
nextFrontier.push({ id: child.id, depth });
|
|
}
|
|
frontier = nextFrontier;
|
|
}
|
|
|
|
return result;
|
|
}
|
|
|
|
async function activeRunsForTree(companyId: string, treeIssues: TreeIssue[]) {
|
|
const issueIds = treeIssues.map((issue) => issue.id);
|
|
if (issueIds.length === 0) return [];
|
|
const runIds = treeIssues
|
|
.map((issue) => issue.executionRunId)
|
|
.filter((id): id is string => typeof id === "string" && id.length > 0);
|
|
const uniqueRunIds = [...new Set(runIds)];
|
|
const issueIdFromContext = sql<string | null>`${heartbeatRuns.contextSnapshot} ->> 'issueId'`;
|
|
const issueIdSet = new Set(issueIds);
|
|
|
|
const rows = await db
|
|
.select({
|
|
id: heartbeatRuns.id,
|
|
agentId: heartbeatRuns.agentId,
|
|
status: heartbeatRuns.status,
|
|
issueIdFromContext,
|
|
startedAt: heartbeatRuns.startedAt,
|
|
createdAt: heartbeatRuns.createdAt,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
eq(heartbeatRuns.companyId, companyId),
|
|
inArray(heartbeatRuns.status, [...ACTIVE_RUN_STATUSES]),
|
|
uniqueRunIds.length > 0
|
|
? or(inArray(heartbeatRuns.id, uniqueRunIds), inArray(issueIdFromContext, issueIds))
|
|
: inArray(issueIdFromContext, issueIds),
|
|
),
|
|
);
|
|
|
|
const issueIdByExecutionRunId = new Map(
|
|
treeIssues
|
|
.filter((issue) => issue.executionRunId)
|
|
.map((issue) => [issue.executionRunId as string, issue.id]),
|
|
);
|
|
return rows
|
|
.map((run) => {
|
|
if (run.status !== "queued" && run.status !== "running") return null;
|
|
const issueId = run.issueIdFromContext && issueIdSet.has(run.issueIdFromContext)
|
|
? run.issueIdFromContext
|
|
: issueIdByExecutionRunId.get(run.id) ?? null;
|
|
if (!issueId) return null;
|
|
return {
|
|
id: run.id,
|
|
issueId,
|
|
agentId: run.agentId,
|
|
status: run.status,
|
|
startedAt: run.startedAt,
|
|
createdAt: run.createdAt,
|
|
} satisfies ActiveRunRow;
|
|
})
|
|
.filter((run): run is ActiveRunRow => run !== null)
|
|
.sort((a, b) => a.issueId.localeCompare(b.issueId) || a.createdAt.getTime() - b.createdAt.getTime());
|
|
}
|
|
|
|
async function activeHoldsByIssueId(companyId: string, issueIds: string[]) {
|
|
const byIssueId = new Map<string, { all: string[]; pause: string[] }>();
|
|
if (issueIds.length === 0) return byIssueId;
|
|
const rows = await db
|
|
.select({
|
|
issueId: issueTreeHoldMembers.issueId,
|
|
holdId: issueTreeHolds.id,
|
|
mode: issueTreeHolds.mode,
|
|
})
|
|
.from(issueTreeHoldMembers)
|
|
.innerJoin(issueTreeHolds, eq(issueTreeHoldMembers.holdId, issueTreeHolds.id))
|
|
.where(
|
|
and(
|
|
eq(issueTreeHoldMembers.companyId, companyId),
|
|
eq(issueTreeHolds.status, "active"),
|
|
inArray(issueTreeHoldMembers.issueId, issueIds),
|
|
),
|
|
)
|
|
.orderBy(asc(issueTreeHolds.createdAt), asc(issueTreeHolds.id));
|
|
|
|
for (const row of rows) {
|
|
const current = byIssueId.get(row.issueId) ?? { all: [], pause: [] };
|
|
current.all.push(row.holdId);
|
|
if (row.mode === "pause") current.pause.push(row.holdId);
|
|
byIssueId.set(row.issueId, current);
|
|
}
|
|
return byIssueId;
|
|
}
|
|
|
|
async function activeCancelSnapshotsByIssueId(companyId: string, rootIssueId: string) {
|
|
const activeCancelHolds = await listHolds(companyId, rootIssueId, {
|
|
status: "active",
|
|
mode: "cancel",
|
|
includeMembers: true,
|
|
});
|
|
const byIssueId = new Map<string, ActiveCancelSnapshot>();
|
|
for (const hold of [...activeCancelHolds].reverse()) {
|
|
for (const member of hold.members ?? []) {
|
|
const current = byIssueId.get(member.issueId) ?? { holdIds: [], member: null };
|
|
if (!current.holdIds.includes(hold.id)) current.holdIds.push(hold.id);
|
|
if (!current.member && !member.skipped) current.member = member;
|
|
byIssueId.set(member.issueId, current);
|
|
}
|
|
}
|
|
return byIssueId;
|
|
}
|
|
|
|
async function activePauseHoldsForIssueIds(companyId: string, issueIds: string[]) {
|
|
if (issueIds.length === 0) return [];
|
|
return db
|
|
.select()
|
|
.from(issueTreeHolds)
|
|
.where(
|
|
and(
|
|
eq(issueTreeHolds.companyId, companyId),
|
|
eq(issueTreeHolds.status, "active"),
|
|
eq(issueTreeHolds.mode, "pause"),
|
|
inArray(issueTreeHolds.rootIssueId, issueIds),
|
|
),
|
|
)
|
|
.orderBy(asc(issueTreeHolds.createdAt), asc(issueTreeHolds.id));
|
|
}
|
|
|
|
async function getActivePauseHoldGate(
|
|
companyId: string,
|
|
issueId: string,
|
|
): Promise<ActiveIssueTreePauseHoldGate | null> {
|
|
const activePauseHolds = await db
|
|
.select({
|
|
id: issueTreeHolds.id,
|
|
rootIssueId: issueTreeHolds.rootIssueId,
|
|
reason: issueTreeHolds.reason,
|
|
releasePolicy: issueTreeHolds.releasePolicy,
|
|
})
|
|
.from(issueTreeHolds)
|
|
.where(
|
|
and(
|
|
eq(issueTreeHolds.companyId, companyId),
|
|
eq(issueTreeHolds.status, "active"),
|
|
eq(issueTreeHolds.mode, "pause"),
|
|
),
|
|
)
|
|
.orderBy(asc(issueTreeHolds.createdAt), asc(issueTreeHolds.id));
|
|
if (activePauseHolds.length === 0) return null;
|
|
|
|
const holdByRootIssueId = new Map(activePauseHolds.map((hold) => [hold.rootIssueId, hold]));
|
|
let currentIssueId: string | null = issueId;
|
|
const visited = new Set<string>();
|
|
|
|
while (
|
|
currentIssueId
|
|
&& !visited.has(currentIssueId)
|
|
&& visited.size < MAX_PAUSE_HOLD_ANCESTOR_DEPTH
|
|
) {
|
|
visited.add(currentIssueId);
|
|
const hold = holdByRootIssueId.get(currentIssueId);
|
|
if (hold) {
|
|
return {
|
|
holdId: hold.id,
|
|
rootIssueId: hold.rootIssueId,
|
|
issueId,
|
|
isRoot: hold.rootIssueId === issueId,
|
|
mode: "pause",
|
|
reason: hold.reason,
|
|
releasePolicy: (hold.releasePolicy as IssueTreeHoldReleasePolicy | null) ?? null,
|
|
};
|
|
}
|
|
|
|
const parent: { parentId: string | null } | null = await db
|
|
.select({ parentId: issues.parentId })
|
|
.from(issues)
|
|
.where(and(eq(issues.id, currentIssueId), eq(issues.companyId, companyId)))
|
|
.then((rows) => rows[0] ?? null);
|
|
currentIssueId = parent?.parentId ?? null;
|
|
}
|
|
|
|
return null;
|
|
}
|
|
|
|
async function listActivePauseHeldIssueIds(companyId: string, issueIds: string[]): Promise<Set<string>> {
|
|
if (issueIds.length === 0) return new Set();
|
|
const [anyHold] = await db.select({ id: issueTreeHolds.id }).from(issueTreeHolds).where(and(
|
|
eq(issueTreeHolds.companyId, companyId), eq(issueTreeHolds.status, "active"), eq(issueTreeHolds.mode, "pause"))).limit(1);
|
|
if (!anyHold) return new Set();
|
|
const result = await db.execute<{ issueId: string }>(sql`
|
|
with recursive chain(start_id, id, depth, path) as (
|
|
select requested.id, requested.id, 1, array[requested.id]
|
|
from unnest(${sql.param(issueIds)}::uuid[]) as requested(id)
|
|
union all
|
|
select chain.start_id, parent.parent_id, chain.depth + 1, chain.path || parent.parent_id
|
|
from chain
|
|
join issues parent on parent.id = chain.id and parent.company_id = ${companyId}
|
|
where parent.parent_id is not null and chain.depth < ${MAX_PAUSE_HOLD_ANCESTOR_DEPTH}
|
|
and not parent.parent_id = any(chain.path)
|
|
)
|
|
select distinct chain.start_id::text as "issueId" from chain
|
|
where exists (select 1 from issue_tree_holds hold where hold.company_id = ${companyId}
|
|
and hold.status = 'active' and hold.mode = 'pause' and hold.root_issue_id = chain.id)`);
|
|
const rows = Array.isArray(result) ? result : (result as { rows?: Array<{ issueId: string }> }).rows ?? [];
|
|
return new Set(rows.map(row => row.issueId));
|
|
}
|
|
|
|
async function preview(
|
|
companyId: string,
|
|
rootIssueId: string,
|
|
input: {
|
|
mode: IssueTreeControlMode;
|
|
releasePolicy?: IssueTreeHoldReleasePolicy | null;
|
|
},
|
|
): Promise<IssueTreeControlPreview> {
|
|
const treeIssues = await listTreeIssues(companyId, rootIssueId);
|
|
const issueIds = treeIssues.map((issue) => issue.id);
|
|
const [activeRunRows, holdsByIssueId, activeCancelSnapshots] = await Promise.all([
|
|
activeRunsForTree(companyId, treeIssues),
|
|
activeHoldsByIssueId(companyId, issueIds),
|
|
input.mode === "restore"
|
|
? activeCancelSnapshotsByIssueId(companyId, rootIssueId)
|
|
: Promise.resolve(new Map<string, ActiveCancelSnapshot>()),
|
|
]);
|
|
const runsByIssueId = new Map<string, ActiveRunRow>();
|
|
for (const run of activeRunRows) {
|
|
if (!runsByIssueId.has(run.issueId)) runsByIssueId.set(run.issueId, run);
|
|
}
|
|
const countsByStatus: Partial<Record<IssueStatus, number>> = {};
|
|
|
|
const issuesToPreview = treeIssues.map((issue) => {
|
|
const status = coerceIssueStatus(issue.status);
|
|
countsByStatus[status] = (countsByStatus[status] ?? 0) + 1;
|
|
const holdState = holdsByIssueId.get(issue.id) ?? { all: [], pause: [] };
|
|
const skipReason = issueSkipReason({
|
|
mode: input.mode,
|
|
issue,
|
|
activePauseHoldIds: holdState.pause,
|
|
activeCancelSnapshot: activeCancelSnapshots.get(issue.id) ?? null,
|
|
});
|
|
const run = runsByIssueId.get(issue.id);
|
|
return {
|
|
id: issue.id,
|
|
identifier: issue.identifier,
|
|
title: issue.title,
|
|
status,
|
|
parentId: issue.parentId,
|
|
depth: issue.depth,
|
|
assigneeAgentId: issue.assigneeAgentId,
|
|
assigneeUserId: issue.assigneeUserId,
|
|
activeRun: run ? toPreviewRun(run) : null,
|
|
activeHoldIds: holdState.all,
|
|
action: input.mode,
|
|
skipped: skipReason !== null,
|
|
skipReason,
|
|
} satisfies IssueTreePreviewIssue;
|
|
});
|
|
const skippedIssues = issuesToPreview.filter((issue) => issue.skipped);
|
|
const activeRuns = activeRunRows
|
|
.map(toPreviewRun)
|
|
.sort((a, b) => a.issueId.localeCompare(b.issueId) || a.id.localeCompare(b.id));
|
|
const affectedAgents = buildAffectedAgents(issuesToPreview, input.mode);
|
|
|
|
return {
|
|
companyId,
|
|
rootIssueId,
|
|
mode: input.mode,
|
|
generatedAt: new Date(),
|
|
releasePolicy: normalizeReleasePolicy(input.releasePolicy),
|
|
totals: {
|
|
totalIssues: issuesToPreview.length,
|
|
affectedIssues: issuesToPreview.length - skippedIssues.length,
|
|
skippedIssues: skippedIssues.length,
|
|
activeRuns: activeRuns.filter((run) => run.status === "running").length,
|
|
queuedRuns: activeRuns.filter((run) => run.status === "queued").length,
|
|
affectedAgents: affectedAgents.length,
|
|
},
|
|
countsByStatus,
|
|
issues: issuesToPreview,
|
|
skippedIssues,
|
|
activeRuns,
|
|
affectedAgents,
|
|
warnings: buildWarnings({ mode: input.mode, issuesToPreview, activeRuns }),
|
|
};
|
|
}
|
|
|
|
async function createHold(
|
|
companyId: string,
|
|
rootIssueId: string,
|
|
input: {
|
|
mode: IssueTreeControlMode;
|
|
reason?: string | null;
|
|
releasePolicy?: IssueTreeHoldReleasePolicy | null;
|
|
actor: ActorInput;
|
|
},
|
|
): Promise<{
|
|
hold: IssueTreeHold;
|
|
preview: IssueTreeControlPreview;
|
|
resumedPauseHoldIds?: string[];
|
|
}> {
|
|
if (input.mode === "cancel") {
|
|
const [conversation] = await db.select({ id: issues.id }).from(issues).where(and(
|
|
eq(issues.id, rootIssueId), eq(issues.companyId, companyId), sql`${issues.conversationAgentId} is not null`,
|
|
));
|
|
if (conversation) throw unprocessable("Stop the active reply instead of cancelling the persistent conversation");
|
|
}
|
|
const holdReleasePolicy = normalizeReleasePolicy(input.releasePolicy);
|
|
const holdPreview = await preview(companyId, rootIssueId, {
|
|
mode: input.mode,
|
|
releasePolicy: holdReleasePolicy,
|
|
});
|
|
|
|
if (input.mode === "resume") {
|
|
const issueIds = [...new Set(holdPreview.issues.map((issue) => issue.id))];
|
|
const activePauseHolds = await activePauseHoldsForIssueIds(companyId, issueIds);
|
|
const releaseReason = input.reason ?? "Subtree resume applied.";
|
|
|
|
const { hold: resumeHold } = await db.transaction(async (tx) => {
|
|
await notifyIssueRecovery(tx, { companyId });
|
|
await notifyDeferredExecutionEligibility(tx, { companyId });
|
|
const [createdHold] = await tx
|
|
.insert(issueTreeHolds)
|
|
.values({
|
|
companyId,
|
|
rootIssueId,
|
|
mode: input.mode,
|
|
status: "active",
|
|
reason: input.reason ?? null,
|
|
releasePolicy: holdReleasePolicy as unknown as Record<string, unknown>,
|
|
createdByActorType: input.actor.actorType,
|
|
createdByAgentId: input.actor.agentId ?? null,
|
|
createdByUserId: input.actor.userId ?? (input.actor.actorType === "user" ? input.actor.actorId : null),
|
|
createdByRunId: input.actor.runId ?? null,
|
|
})
|
|
.returning();
|
|
|
|
const memberRows = holdPreview.issues.map((issue) => ({
|
|
companyId,
|
|
holdId: createdHold.id,
|
|
issueId: issue.id,
|
|
parentIssueId: issue.parentId,
|
|
depth: issue.depth,
|
|
issueIdentifier: issue.identifier,
|
|
issueTitle: issue.title,
|
|
issueStatus: issue.status,
|
|
assigneeAgentId: issue.assigneeAgentId,
|
|
assigneeUserId: issue.assigneeUserId,
|
|
activeRunId: issue.activeRun?.id ?? null,
|
|
activeRunStatus: issue.activeRun?.status ?? null,
|
|
skipped: issue.skipped,
|
|
skipReason: issue.skipReason,
|
|
}));
|
|
|
|
const createdMembers = memberRows.length > 0
|
|
? await tx
|
|
.insert(issueTreeHoldMembers)
|
|
.values(memberRows)
|
|
.returning()
|
|
: [];
|
|
|
|
return { hold: toHold(createdHold, createdMembers) };
|
|
});
|
|
|
|
const resumedPauseHoldIds = activePauseHolds.map((hold) => hold.id);
|
|
if (resumedPauseHoldIds.length > 0) {
|
|
await Promise.all(
|
|
activePauseHolds.map((pauseHold) =>
|
|
releaseHold(companyId, pauseHold.rootIssueId, pauseHold.id, {
|
|
reason: releaseReason,
|
|
metadata: {
|
|
resumedByResumeHoldId: resumeHold.id,
|
|
resumeHoldMode: "tree_resume",
|
|
resumedPauseHoldId: pauseHold.id,
|
|
},
|
|
actor: input.actor,
|
|
}),
|
|
),
|
|
);
|
|
}
|
|
|
|
const releasedResumeHold = await releaseHold(companyId, rootIssueId, resumeHold.id, {
|
|
reason: releaseReason,
|
|
metadata: {
|
|
resumedPauseHoldIds,
|
|
resumeMode: "subtree",
|
|
...(input.releasePolicy ? { releasePolicy: holdReleasePolicy } : {}),
|
|
},
|
|
actor: input.actor,
|
|
});
|
|
|
|
return {
|
|
hold: releasedResumeHold,
|
|
preview: holdPreview,
|
|
resumedPauseHoldIds,
|
|
};
|
|
}
|
|
|
|
const { hold, members } = await db.transaction(async (tx) => {
|
|
await notifyIssueRecovery(tx, { companyId });
|
|
await notifyDeferredExecutionEligibility(tx, { companyId });
|
|
const [createdHold] = await tx
|
|
.insert(issueTreeHolds)
|
|
.values({
|
|
companyId,
|
|
rootIssueId,
|
|
mode: input.mode,
|
|
status: "active",
|
|
reason: input.reason ?? null,
|
|
releasePolicy: holdReleasePolicy as unknown as Record<string, unknown>,
|
|
createdByActorType: input.actor.actorType,
|
|
createdByAgentId: input.actor.agentId ?? null,
|
|
createdByUserId: input.actor.userId ?? (input.actor.actorType === "user" ? input.actor.actorId : null),
|
|
createdByRunId: input.actor.runId ?? null,
|
|
})
|
|
.returning();
|
|
|
|
const memberRows = holdPreview.issues.map((issue) => ({
|
|
companyId,
|
|
holdId: createdHold.id,
|
|
issueId: issue.id,
|
|
parentIssueId: issue.parentId,
|
|
depth: issue.depth,
|
|
issueIdentifier: issue.identifier,
|
|
issueTitle: issue.title,
|
|
issueStatus: issue.status,
|
|
assigneeAgentId: issue.assigneeAgentId,
|
|
assigneeUserId: issue.assigneeUserId,
|
|
activeRunId: issue.activeRun?.id ?? null,
|
|
activeRunStatus: issue.activeRun?.status ?? null,
|
|
skipped: issue.skipped,
|
|
skipReason: issue.skipReason,
|
|
}));
|
|
|
|
const createdMembers = memberRows.length > 0
|
|
? await tx.insert(issueTreeHoldMembers).values(memberRows).returning()
|
|
: [];
|
|
|
|
return { hold: createdHold, members: createdMembers };
|
|
});
|
|
|
|
return {
|
|
hold: toHold(hold, members),
|
|
preview: holdPreview,
|
|
};
|
|
}
|
|
|
|
async function cancelIssueStatusesForHold(
|
|
companyId: string,
|
|
rootIssueId: string,
|
|
holdId: string,
|
|
): Promise<TreeStatusUpdateResult> {
|
|
const hold = await getHold(companyId, holdId);
|
|
if (!hold) throw notFound("Issue tree hold not found");
|
|
if (hold.rootIssueId !== rootIssueId) {
|
|
throw unprocessable("Issue tree hold does not belong to the requested root issue");
|
|
}
|
|
if (hold.mode !== "cancel") {
|
|
throw unprocessable("Issue tree hold is not a cancel operation");
|
|
}
|
|
|
|
const issueIds = [...new Set((hold.members ?? [])
|
|
.filter((member) => !member.skipped)
|
|
.map((member) => member.issueId))];
|
|
if (issueIds.length === 0) return { updatedIssueIds: [], updatedIssues: [] };
|
|
|
|
const now = new Date();
|
|
const postCommitIssueActions: IssuePostCommitAction[] = [];
|
|
const { executeIssuePostCommitActions, issueService } = await import("./issues.js");
|
|
const svc = issueService(db);
|
|
const updated = await db.transaction(async (tx) => {
|
|
const eligibleIssues = await tx
|
|
.select({ id: issues.id })
|
|
.from(issues)
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, companyId),
|
|
inArray(issues.id, issueIds),
|
|
notInArray(issues.status, ["done", "cancelled"]),
|
|
),
|
|
);
|
|
|
|
const rows = [];
|
|
for (const issue of eligibleIssues) {
|
|
const updatedIssue = await svc.update(
|
|
issue.id,
|
|
{
|
|
status: "cancelled",
|
|
cancelledAt: now,
|
|
actorAgentId: hold.createdByAgentId,
|
|
actorUserId: hold.createdByUserId,
|
|
},
|
|
tx,
|
|
undefined,
|
|
postCommitIssueActions,
|
|
);
|
|
if (updatedIssue) rows.push(updatedIssue);
|
|
}
|
|
return rows;
|
|
});
|
|
await executeIssuePostCommitActions(db, postCommitIssueActions);
|
|
|
|
return {
|
|
updatedIssueIds: updated.map((issue) => issue.id),
|
|
updatedIssues: updated.map((issue) => ({
|
|
id: issue.id,
|
|
status: coerceIssueStatus(issue.status),
|
|
assigneeAgentId: issue.assigneeAgentId,
|
|
})),
|
|
};
|
|
}
|
|
|
|
async function restoreIssueStatusesForHold(
|
|
companyId: string,
|
|
rootIssueId: string,
|
|
restoreHoldId: string,
|
|
input: {
|
|
reason?: string | null;
|
|
actor: ActorInput;
|
|
},
|
|
): Promise<RestoreTreeStatusResult> {
|
|
const restoreHold = await getHold(companyId, restoreHoldId);
|
|
if (!restoreHold) throw notFound("Issue tree hold not found");
|
|
if (restoreHold.rootIssueId !== rootIssueId) {
|
|
throw unprocessable("Issue tree hold does not belong to the requested root issue");
|
|
}
|
|
if (restoreHold.mode !== "restore") {
|
|
throw unprocessable("Issue tree hold is not a restore operation");
|
|
}
|
|
|
|
const activeCancelHolds = await listHolds(companyId, rootIssueId, {
|
|
status: "active",
|
|
mode: "cancel",
|
|
includeMembers: true,
|
|
});
|
|
const cancelSnapshotByIssueId = new Map<string, IssueTreeHoldMember>();
|
|
for (const hold of [...activeCancelHolds].reverse()) {
|
|
for (const member of hold.members ?? []) {
|
|
if (!member.skipped && !cancelSnapshotByIssueId.has(member.issueId)) {
|
|
cancelSnapshotByIssueId.set(member.issueId, member);
|
|
}
|
|
}
|
|
}
|
|
|
|
const restoreIssueIds = [...new Set((restoreHold.members ?? [])
|
|
.filter((member) => !member.skipped)
|
|
.map((member) => member.issueId))];
|
|
const restoreStatusByIssueId = new Map<string, IssueStatus>();
|
|
for (const issueId of restoreIssueIds) {
|
|
const snapshot = cancelSnapshotByIssueId.get(issueId);
|
|
if (!snapshot) continue;
|
|
const restoredStatus = restoreStatusFromCancelSnapshot(coerceIssueStatus(snapshot.issueStatus));
|
|
if (restoredStatus) restoreStatusByIssueId.set(issueId, restoredStatus);
|
|
}
|
|
|
|
const issueIdsByStatus = new Map<IssueStatus, string[]>();
|
|
for (const [issueId, status] of restoreStatusByIssueId) {
|
|
const current = issueIdsByStatus.get(status) ?? [];
|
|
current.push(issueId);
|
|
issueIdsByStatus.set(status, current);
|
|
}
|
|
|
|
const now = new Date();
|
|
const releasedCancelHoldIds = activeCancelHolds.map((hold) => hold.id);
|
|
const updatedIssues = await db.transaction(async (tx) => {
|
|
await notifyDeferredExecutionEligibility(tx, { companyId });
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
|
|
await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers);
|
|
const restored: TreeStatusUpdateResult["updatedIssues"] = [];
|
|
for (const [status, issueIdsForStatus] of issueIdsByStatus) {
|
|
if (issueIdsForStatus.length === 0) continue;
|
|
for (const issueId of issueIdsForStatus) {
|
|
await notifyRoutineExecutionStatus(tx, issueId, companyId);
|
|
await notifyNativeStatusReassessment(tx, { companyId, issueId });
|
|
await notifyTerminalWorkspaceWork(tx, `issue:${issueId}`);
|
|
await notifyChatResponseAvailability(tx, { companyId, issueId });
|
|
if (status === "blocked" || status === "in_review") await notifyUnsafeWorkspaceExportRecovery(tx, { companyId, issueId });
|
|
}
|
|
await notifyStatusCardIssueChange(tx, companyId);
|
|
const rows = await tx
|
|
.update(issues)
|
|
.set({
|
|
status,
|
|
cancelledAt: null,
|
|
completedAt: null,
|
|
checkoutRunId: null,
|
|
executionRunId: null,
|
|
executionAgentNameKey: null,
|
|
executionLockedAt: null,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(issues.companyId, companyId),
|
|
inArray(issues.id, issueIdsForStatus),
|
|
eq(issues.status, "cancelled"),
|
|
),
|
|
)
|
|
.returning({
|
|
id: issues.id,
|
|
status: issues.status,
|
|
assigneeAgentId: issues.assigneeAgentId,
|
|
});
|
|
restored.push(...rows.map((issue) => ({
|
|
id: issue.id,
|
|
status: coerceIssueStatus(issue.status),
|
|
assigneeAgentId: issue.assigneeAgentId,
|
|
})));
|
|
}
|
|
|
|
if (releasedCancelHoldIds.length > 0) {
|
|
await tx
|
|
.update(issueTreeHolds)
|
|
.set({
|
|
status: "released",
|
|
releasedAt: now,
|
|
releasedByActorType: input.actor.actorType,
|
|
releasedByAgentId: input.actor.agentId ?? null,
|
|
releasedByUserId: input.actor.userId ?? (input.actor.actorType === "user" ? input.actor.actorId : null),
|
|
releasedByRunId: input.actor.runId ?? null,
|
|
releaseReason: input.reason ?? "Restored by subtree restore operation",
|
|
releaseMetadata: {
|
|
restoreHoldId,
|
|
restoredIssueIds: restored.map((issue) => issue.id),
|
|
},
|
|
updatedAt: now,
|
|
})
|
|
.where(and(eq(issueTreeHolds.companyId, companyId), inArray(issueTreeHolds.id, releasedCancelHoldIds)));
|
|
}
|
|
|
|
await tx
|
|
.update(issueTreeHolds)
|
|
.set({
|
|
status: "released",
|
|
releasedAt: now,
|
|
releasedByActorType: input.actor.actorType,
|
|
releasedByAgentId: input.actor.agentId ?? null,
|
|
releasedByUserId: input.actor.userId ?? (input.actor.actorType === "user" ? input.actor.actorId : null),
|
|
releasedByRunId: input.actor.runId ?? null,
|
|
releaseReason: input.reason ?? "Restore operation applied",
|
|
releaseMetadata: {
|
|
restoredIssueIds: restored.map((issue) => issue.id),
|
|
releasedCancelHoldIds,
|
|
},
|
|
updatedAt: now,
|
|
})
|
|
.where(and(eq(issueTreeHolds.companyId, companyId), eq(issueTreeHolds.id, restoreHoldId)));
|
|
|
|
return restored;
|
|
});
|
|
|
|
return {
|
|
updatedIssueIds: updatedIssues.map((issue) => issue.id),
|
|
updatedIssues,
|
|
releasedCancelHoldIds,
|
|
restoreHold: await getHold(companyId, restoreHoldId),
|
|
};
|
|
}
|
|
|
|
async function getHold(companyId: string, holdId: string) {
|
|
const hold = await db
|
|
.select()
|
|
.from(issueTreeHolds)
|
|
.where(and(eq(issueTreeHolds.id, holdId), eq(issueTreeHolds.companyId, companyId)))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!hold) return null;
|
|
const members = await db
|
|
.select()
|
|
.from(issueTreeHoldMembers)
|
|
.where(and(eq(issueTreeHoldMembers.companyId, companyId), eq(issueTreeHoldMembers.holdId, holdId)))
|
|
.orderBy(asc(issueTreeHoldMembers.depth), asc(issueTreeHoldMembers.createdAt), asc(issueTreeHoldMembers.issueId));
|
|
return toHold(hold, members);
|
|
}
|
|
|
|
async function listHolds(
|
|
companyId: string,
|
|
rootIssueId: string,
|
|
input?: {
|
|
status?: IssueTreeHold["status"];
|
|
mode?: IssueTreeControlMode;
|
|
includeMembers?: boolean;
|
|
},
|
|
) {
|
|
const whereClauses = [
|
|
eq(issueTreeHolds.companyId, companyId),
|
|
eq(issueTreeHolds.rootIssueId, rootIssueId),
|
|
];
|
|
if (input?.status) whereClauses.push(eq(issueTreeHolds.status, input.status));
|
|
if (input?.mode) whereClauses.push(eq(issueTreeHolds.mode, input.mode));
|
|
|
|
const holds = await db
|
|
.select()
|
|
.from(issueTreeHolds)
|
|
.where(and(...whereClauses))
|
|
.orderBy(asc(issueTreeHolds.createdAt), asc(issueTreeHolds.id));
|
|
if (!input?.includeMembers || holds.length === 0) {
|
|
return holds.map((hold) => toHold(hold));
|
|
}
|
|
|
|
const holdIds = holds.map((hold) => hold.id);
|
|
const members = await db
|
|
.select()
|
|
.from(issueTreeHoldMembers)
|
|
.where(
|
|
and(
|
|
eq(issueTreeHoldMembers.companyId, companyId),
|
|
inArray(issueTreeHoldMembers.holdId, holdIds),
|
|
),
|
|
)
|
|
.orderBy(asc(issueTreeHoldMembers.depth), asc(issueTreeHoldMembers.createdAt), asc(issueTreeHoldMembers.issueId));
|
|
|
|
const membersByHoldId = new Map<string, HoldMemberRow[]>();
|
|
for (const member of members) {
|
|
const existing = membersByHoldId.get(member.holdId) ?? [];
|
|
existing.push(member);
|
|
membersByHoldId.set(member.holdId, existing);
|
|
}
|
|
|
|
return holds.map((hold) => toHold(hold, membersByHoldId.get(hold.id) ?? []));
|
|
}
|
|
|
|
async function releaseHold(
|
|
companyId: string,
|
|
rootIssueId: string,
|
|
holdId: string,
|
|
input: {
|
|
reason?: string | null;
|
|
releasePolicy?: IssueTreeHoldReleasePolicy | null;
|
|
metadata?: Record<string, unknown> | null;
|
|
actor: ActorInput;
|
|
},
|
|
) {
|
|
const existing = await db
|
|
.select()
|
|
.from(issueTreeHolds)
|
|
.where(and(eq(issueTreeHolds.id, holdId), eq(issueTreeHolds.companyId, companyId)))
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!existing) throw notFound("Issue tree hold not found");
|
|
if (existing.rootIssueId !== rootIssueId) {
|
|
throw unprocessable("Issue tree hold does not belong to the requested root issue");
|
|
}
|
|
if (existing.status === "released") {
|
|
throw conflict("Issue tree hold is already released");
|
|
}
|
|
|
|
const [updated] = await db.transaction(async tx => {
|
|
await promoteDeferredWorkIntents(tx, { companyId, trigger: "tree_hold_released" });
|
|
return tx
|
|
.update(issueTreeHolds)
|
|
.set({
|
|
status: "released",
|
|
releasedAt: new Date(),
|
|
releasedByActorType: input.actor.actorType,
|
|
releasedByAgentId: input.actor.agentId ?? null,
|
|
releasedByUserId: input.actor.userId ?? (input.actor.actorType === "user" ? input.actor.actorId : null),
|
|
releasedByRunId: input.actor.runId ?? null,
|
|
releaseReason: input.reason ?? null,
|
|
releasePolicy: input.releasePolicy
|
|
? (normalizeReleasePolicy(input.releasePolicy) as unknown as Record<string, unknown>)
|
|
: existing.releasePolicy,
|
|
releaseMetadata: input.metadata ?? null,
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(and(eq(issueTreeHolds.id, holdId), eq(issueTreeHolds.companyId, companyId),
|
|
eq(issueTreeHolds.status, "active")))
|
|
.returning();
|
|
});
|
|
// A concurrent release won; it alone owns the resumed work.
|
|
if (!updated) throw conflict("Issue tree hold is already released");
|
|
|
|
const members = await db
|
|
.select()
|
|
.from(issueTreeHoldMembers)
|
|
.where(and(eq(issueTreeHoldMembers.companyId, companyId), eq(issueTreeHoldMembers.holdId, holdId)))
|
|
.orderBy(asc(issueTreeHoldMembers.depth), asc(issueTreeHoldMembers.createdAt), asc(issueTreeHoldMembers.issueId));
|
|
|
|
return toHold(updated, members);
|
|
}
|
|
|
|
/**
|
|
* A pause interrupts runs as an operator hold, not as a failure. That pending
|
|
* work is owed one re-dispatch once no pause covers the task. A run is owed
|
|
* when a pause hold's member snapshot names it and an operator Stop ended it,
|
|
* or when a dispatch gate refused it because a pause covered the task (a
|
|
* scheduled retry that came due, or a queued run reaching its claim); the
|
|
* gate records the covering hold on the run. The obligation belongs to the
|
|
* run, so overlapping holds keep it until the last release. Current state
|
|
* governs: the interrupted run must still be the task's latest run, the task
|
|
* must be executable by the same agent, and the claim marker makes the
|
|
* resume happen at most once. Execution blockers are the caller's.
|
|
*/
|
|
async function findOwedPauseResume(companyId: string, issueId: string) {
|
|
const [issue] = await db.select({
|
|
status: issues.status, assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId,
|
|
}).from(issues).where(and(eq(issues.companyId, companyId), eq(issues.id, issueId)));
|
|
if (!issue?.assigneeAgentId || issue.assigneeUserId
|
|
|| !ISSUE_TREE_RESUME_EXECUTABLE_STATUSES.includes(issue.status)) return null;
|
|
const [run] = await db.select({
|
|
id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, status: heartbeatRuns.status,
|
|
finishedAt: heartbeatRuns.finishedAt, resultJson: heartbeatRuns.resultJson, errorCode: heartbeatRuns.errorCode,
|
|
}).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, companyId),
|
|
sql`coalesce(${heartbeatRuns.issueId}::text, ${heartbeatRuns.contextSnapshot}->>'issueId') = ${issueId}`,
|
|
)).orderBy(sql`${heartbeatRuns.createdAt} desc`).limit(1);
|
|
if (!run || run.status !== "cancelled" || !run.finishedAt || run.agentId !== issue.assigneeAgentId
|
|
|| run.resultJson?.treeHoldResume !== undefined) return null;
|
|
// Only a cancellation recorded while a covering pause was active is owed.
|
|
const activeAtCancellation = and(
|
|
eq(issueTreeHolds.companyId, companyId), eq(issueTreeHolds.mode, "pause"),
|
|
sql`${issueTreeHolds.createdAt} <= ${run.finishedAt.toISOString()}::timestamptz`,
|
|
or(isNull(issueTreeHolds.releasedAt),
|
|
sql`${issueTreeHolds.releasedAt} >= ${run.finishedAt.toISOString()}::timestamptz`),
|
|
);
|
|
const memberHold = (snapshotRunId?: string) => db.select({ id: issueTreeHolds.id }).from(issueTreeHoldMembers)
|
|
.innerJoin(issueTreeHolds, eq(issueTreeHolds.id, issueTreeHoldMembers.holdId))
|
|
.where(and(
|
|
eq(issueTreeHoldMembers.companyId, companyId), eq(issueTreeHoldMembers.issueId, issueId),
|
|
snapshotRunId ? eq(issueTreeHoldMembers.activeRunId, snapshotRunId) : undefined,
|
|
eq(issueTreeHoldMembers.skipped, false), activeAtCancellation,
|
|
)).orderBy(asc(issueTreeHolds.createdAt)).limit(1);
|
|
let hold: { id: string } | undefined;
|
|
if (run.errorCode === ISSUE_PAUSED_ERROR_CODE) {
|
|
const suppressedBy = readPauseSuppressionHoldId(run.resultJson);
|
|
// The gate recorded the pause it observed. A run refused before gates
|
|
// recorded that hold falls back to a snapshot pause active at the time.
|
|
[hold] = suppressedBy
|
|
? await db.select({ id: issueTreeHolds.id }).from(issueTreeHolds).where(and(eq(issueTreeHolds.id, suppressedBy),
|
|
eq(issueTreeHolds.companyId, companyId), eq(issueTreeHolds.mode, "pause"))).limit(1)
|
|
: await memberHold();
|
|
} else if (readRunCancellation(run.resultJson)?.source === "operator") {
|
|
// A Stop is owed only when the pause snapshot named the stopped run.
|
|
[hold] = await memberHold(run.id);
|
|
}
|
|
if (!hold) return null;
|
|
if (await getActivePauseHoldGate(companyId, issueId)) return null;
|
|
return { issueId, agentId: run.agentId, interruptedRunId: run.id, holdId: hold.id };
|
|
}
|
|
|
|
async function cancelUnclaimedWakeupsForTree(companyId: string, rootIssueId: string, reason: string) {
|
|
const treeIssues = await listTreeIssues(companyId, rootIssueId);
|
|
const issueIds = treeIssues.map((issue) => issue.id);
|
|
if (issueIds.length === 0) return [];
|
|
const now = new Date();
|
|
return db.transaction(async tx => {
|
|
const affected = await tx.select({ agentId: agentWakeupRequests.agentId }).from(agentWakeupRequests).where(and(
|
|
eq(agentWakeupRequests.companyId, companyId), inArray(agentWakeupRequests.status, ["queued", "deferred_issue_execution"]),
|
|
isNull(agentWakeupRequests.runId), inArray(sql<string | null>`${agentWakeupRequests.payload} ->> 'issueId'`, issueIds),
|
|
)).for("update");
|
|
for (const agentId of new Set(affected.map(wake => wake.agentId))) await notifyStrandedAgentStatus(tx, { companyId, agentId });
|
|
await notifyChatInboundWakeNotices(tx, { companyId });
|
|
return tx
|
|
.update(agentWakeupRequests)
|
|
.set({
|
|
status: "cancelled",
|
|
finishedAt: now,
|
|
error: reason,
|
|
updatedAt: now,
|
|
})
|
|
.where(
|
|
and(
|
|
eq(agentWakeupRequests.companyId, companyId),
|
|
inArray(agentWakeupRequests.status, ["queued", "deferred_issue_execution"]),
|
|
isNull(agentWakeupRequests.runId),
|
|
inArray(sql<string | null>`${agentWakeupRequests.payload} ->> 'issueId'`, issueIds),
|
|
),
|
|
)
|
|
.returning({
|
|
id: agentWakeupRequests.id,
|
|
agentId: agentWakeupRequests.agentId,
|
|
reason: agentWakeupRequests.reason,
|
|
payload: agentWakeupRequests.payload,
|
|
});
|
|
});
|
|
}
|
|
|
|
return {
|
|
listTreeIssues,
|
|
preview,
|
|
createHold,
|
|
cancelIssueStatusesForHold,
|
|
restoreIssueStatusesForHold,
|
|
getHold,
|
|
listHolds,
|
|
getActivePauseHoldGate,
|
|
listActivePauseHeldIssueIds,
|
|
releaseHold,
|
|
findOwedPauseResume,
|
|
cancelUnclaimedWakeupsForTree,
|
|
};
|
|
}
|