mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 13:24:55 +01:00
fix(server): schedule accepted native response recovery (#17696)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs store accepted results and their finalization decisions. > - A completed run can still owe its public answer after an interruption. > - Periodic recovery scans for that answer even when no new work exists. > - This pull request wakes recovery from committed changes and saved retry deadlines. > - The existing owner still verifies the result, source, and current publication authority. > - Settled or dormant response work no longer needs its own periodic scan. ## Linked Issues or Issue Description Refs #17690 (merged). This is slice 48 of the sleep migration stack, now based on master. Related: #17661 handles GitHub terminal publication; this change handles native accepted-response presentation. Governed-review presentation remains a separate duty. **What happened?** Native finalization recovery repeatedly checks already accepted results for a missing answer. Removing that check without complete notifications would lose responses when evidence arrives late, an agent resumes, or admission settles after execution. **Expected behavior** Recover the answer from its saved result when a supported change makes it eligible. Retain real retry deadlines. Stop querying when the answer is selected or cannot currently be published. Reconstruct work after restart. **Steps to reproduce** 1. Commit a native accepted result with no selected response comment. 2. Run finalization recovery while its source is unavailable or locked. 3. Restore that source or release the lock. The answer must appear once without repeating provider execution. ## What Changed - Honor recorded non-create decisions and skip-comment policy in both discovery and recovery. Apply durable conversation/result/continuation guards before recovery locks. Preserve genuine unanswered and legacy recovery. - Scope routine response wakes to the affected issue/run. Coalesced input names its active run; pending-effect settlement names its issue. Sixteen existing global authority/configuration hooks use one explicit company invalidation boundary. No new transaction wrappers or tables are added. - Add an accepted-response worker using the existing delivery coordinator and scoped deadline helper. Preserve saved retry deadlines and pending-effect fences under the owner's lock. - Share the finalization notification boundary. Wake on finalization settlement, exact accepted-result/final-message evidence, agent availability, retry admission, and existing chat authority changes. - Keep lifecycle policy in services. A narrow pre-write status hook lets lifecycle retain ownership of its transaction and status transitions. - Register the worker during enabled maintenance startup. Remove its authority subscription on shutdown. Periodic replay skips this duty on both committed and fresh-commit paths; ordinary finalization and explicit recovery retain their behavior. - Add a partial index and expression statistics for unpresented native runs. Migration 0412 supports concurrent replay and adds no queue table. - Add real PostgreSQL and lifecycle coverage. Preserve the implementation and producer audit in the plan document. ## Verification - Additional real Photon callback acceptance on the published production code disproves the security warning: ten active assertions cause zero response-recovery hints/SELECTs, while three concurrent assertions during an actual attention-to-active recovery cause exactly one wake/SELECT. The existing locked status guard already enforces this boundary. No production change was needed; the resolved finding now has a successful current-head security check. The regression will be retained with the next slice. - Eligibility repair source `892f36e5f66ba1af123034936a7160c64c3ab883` passes **1489 distinct server cases**: 1139 in six complete recovery/finalizer/monitor/summary/workspace suites (81.02 seconds), plus 350 selected chat cases (297.68s). Another 1463 chat cases are outside the pattern; the complete 1813-case file is not claimed. All eleven new regressions are included in that count. Five dedicated fresh PostgreSQL 17 databases migrate through unchanged 0412, with no fixture-source adaptations. Database build, server types/build, boundaries and full-SHA migration ordering pass. All seven test source hashes match. - The exact published baseline first fails eight behavioral assertions while the three genuine-answer cases pass. The same corrected test source passes all eleven after repair: recorded none/reuse remain suppressed; five ineligible history families take zero recovery row locks across repeated issue hints; missing/legacy/create-null answers still publish once; fresh periodic finalization defers presentation to its worker. Earlier fixture setup and variable/compiler failures remain explained in the audit, rather than erased by later success. - Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821 server cases**: 1496 in 13 complete producer/lifecycle/authority suites (110.63 seconds), plus 325 selected chat cases (278.99 seconds). Coverage includes all native-session cases, queue admission, status decisions, membership/board access, the complete GitHub workflow, accepted-response lifecycle and real authority restoration. The other 1477 chat cases are outside this selection; the complete 1802-case file is not claimed for this repair run. - Actual PostgreSQL regressions first reproduce a coalesced-input hint missing its run scope and two routine hint paths producing 12 response-history SELECTs each after three notifications. The same assertions pass with run-scoped coalescing and zero response-history SELECTs for unscoped routine hints. Actual membership restoration still publishes the waiting response once. Assertion time limits are unchanged; this measures real driver SQL and persisted producer work. - Eleven dedicated fresh PostgreSQL 17 databases migrate through 0412. One embedded-only wake-queue fixture import temporarily uses its own external database and restores byte for byte. All 14 selected source files match pre-run hashes. This qualifies external-database behavior rather than reproducing embedded CI locally. - Master advanced independently with #18126's Pi archive retry fix. Normal merge `2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both complete binary patches. All accepted server/database/UI/test/CLI trees stay identical; the three independent runner installer/manifest files are retained. All 13 installer tests pass, including stalled header/body cancellation. Combined server types/build, boundaries and full-SHA migration order pass. A subsequent normal merge of #18128 at `e8b78cce5bb557b086bee9ca930eeaab1d09b244` retains its three test-only file changes with both complete patches preserved. The changed HTTPS backfill case passes on a fresh PostgreSQL fixture (1 selected case; 178 outside the pattern), and combined types/build, boundaries and migration order pass again. All 14 repair source hashes remain unchanged. The independent browser fixture changes are preserved; no local browser run is claimed. - The earlier source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passed the 4171-case full baseline before this scope repair. That scope is preserved in the audit rather than represented as a current full rerun. All 0412 database bytes remain unchanged: original concurrent index/statistics/ANALYZE, prior executable history, journal entries and retained snapshots. Five snapshots 0408 through 0412 remain. The populated 0411-to0412 upgrade preserved 410 prior receipts and 38 companies, appended one receipt and none on replay; index/statistic checks passed. - The earlier P2 security failure, Greptile's former conditional dependency on #17702, and the subsequent actual Ponytail eligibility findings are recorded in `doc/plans/2026-10-25-native-response-presentation-sleep-audit.md`. This PR now includes the issue-scoped pending-effect notification fix itself, so it is independently correct before #17702. Fresh published-head CI, Greptile 5/5 and a clean security scan remain required. No deployed-sleep, external-wake or fleet-savings claim is made. ## Risks - The writer contract is process-local. Independent SQL writers and other replicas are not covered by these notifications. - Unpresented backlog reconstruction still has memory and time cost proportional to candidate count. Pages contain at most 100 rows; execution uses the shared fair batch. Selected history is excluded. Health-only signals do not trigger reconstruction. - Publication policy remains with its current owner. Digest, company/run binding, source authorization, selected-answer suppression, and ownership fences remain required. - The rest of native recovery and governed-review presentation still run separately. This PR does not relax Cloud sleep eligibility or add external wake coverage. - The stack parent has merged and its actual ancestry is reconciled. This publication still needs fresh current-head CI and review. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution, and PostgreSQL tests. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
53 files changed
+1960
-108
No files matched your search
@@ -1,6 +1,8 @@
|
||||
import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js";
|
||||
import { nativeResponsePresentationWork } from "../services/native-runtime/native-response-presentation-work.js";
|
||||
import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js";
|
||||
import { registerGitHubReviewProjectionWork } from "../services/chat-channels/github/review-projection-work.js";
|
||||
import { retainGitHubReviewReceipt } from "../services/chat-channels/github/task-preparation.js";
|
||||
import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js";
|
||||
import { settleUnrecoverableExecutions } from "../services/execution-recovery-resolution.js";
|
||||
import { recordRunPresentationDecision } from "../services/heartbeat/run-completion.js";
|
||||
import { environmentService } from "../services/environments.js";
|
||||
@@ -16,13 +18,13 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
import { claimTaskRetryInput, releaseUnadmittedTaskRetryInput } from "../services/chat-channels/task-retry-input.js";
|
||||
import { restoreNativeChatReviewPresentationInTransaction } from "../services/native-runtime/native-chat-review-presentation.js";
|
||||
import { notifyChatEndpointConfiguration, notifyChatCompanyConfiguration } from "../services/chat-channels/configuration-work-notifications.js";
|
||||
import { configureAgentLifecycle, createAgentLifecycle, processAgentLifecycle } from "../services/agent-lifecycle.js";
|
||||
import { deleteCompany } from "../services/company-deletion.js";
|
||||
import { adapterExecutionControls, createAdapterExecutionControl } from "../services/adapter-execution-control.js";
|
||||
import { chatRunSourceMessageId } from "../services/chat-channels/run-origin.js";
|
||||
import * as durableChatWakeup from "../services/durable-chat-wakeup.js";
|
||||
import { conflict } from "../errors.js";
|
||||
import * as githubTaskPreparation from "../services/chat-channels/github/task-preparation.js";
|
||||
import { configureAgentLifecycle, createAgentLifecycle } from "../services/agent-lifecycle.js";
|
||||
import { budgetPolicies } from "@paperclipai/db";
|
||||
import { budgetService } from "../services/budgets.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
@@ -150,6 +152,8 @@ import {
|
||||
issues,
|
||||
nativeRunResults,
|
||||
nativeRunFinalizations,
|
||||
statusDecisions,
|
||||
statusDecisionEffects,
|
||||
environmentLeases,
|
||||
environments,
|
||||
principalPermissionGrants,
|
||||
@@ -267,6 +271,7 @@ import type {
|
||||
import {
|
||||
finalizeNativeRun,
|
||||
repairCommittedNativeChatResponse,
|
||||
recoverCommittedNativeChatResponse,
|
||||
} from "../services/native-runtime/native-run-finalizer.js";
|
||||
import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js";
|
||||
import { reconcileNativeFinalizations } from "../services/native-runtime/native-finalization-reconciler.js";
|
||||
@@ -66562,6 +66567,19 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
const input = { companyId: context.fixture.companyId, issueId: context.issue.id,
|
||||
agentId: context.fixture.assignedAgentId, failedRunId: context.runId, initiatedByUserId: "owner-user" };
|
||||
const stage = () => db.transaction(tx => context.service.prepareFailedChatRunRetry(tx, input));
|
||||
const observedResponseAdmissions: string[] = [];
|
||||
const responseHints: string[] = [];
|
||||
await coordinator.register(DELIVERY_QUEUES.nativeResponsePresentation, {
|
||||
retryMs: 30_000,
|
||||
onSettled: hints => { responseHints.push(...hints ?? []); },
|
||||
run: async () => {
|
||||
const admitted = await db.select({ id: chatActions.id }).from(chatActions).where(and(
|
||||
eq(chatActions.companyId, context.fixture.companyId), eq(chatActions.kind, "failed_run_retry"), eq(chatActions.status, "processed"),
|
||||
));
|
||||
observedResponseAdmissions.push(...admitted.map(row => row.id));
|
||||
},
|
||||
hasPending: async () => false,
|
||||
}).ready;
|
||||
context.wakeup.mockClear();
|
||||
try {
|
||||
let actionId: string;
|
||||
@@ -66607,6 +66625,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
expect(action?.result.attemptCount).toBe(mode === "retry" ? 2 : 1);
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
await vi.waitFor(() => expect(observedResponseAdmissions).toContain(actionId), { timeout: 5000 });
|
||||
expect(responseHints.map(hint => JSON.parse(hint))).toContainEqual({ companyId: context.fixture.companyId, issueId: context.issue.id });
|
||||
expect(context.wakeup).toHaveBeenCalledTimes(mode === "retry" ? 2 : 1);
|
||||
const receipts = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, actionId));
|
||||
expect(receipts).toHaveLength(1);
|
||||
@@ -74255,6 +74275,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
context: Awaited<ReturnType<typeof failedChatRetryFixture>>,
|
||||
) => Promise<void>,
|
||||
summary?: (issueId: string) => string,
|
||||
finalize = true,
|
||||
) {
|
||||
const context = await failedChatRetryFixture(
|
||||
provider,
|
||||
@@ -74409,7 +74430,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
.from(nativeRunResults)
|
||||
.where(eq(nativeRunResults.runId, context.runId));
|
||||
const accepted = { resultId: acceptedRow.id };
|
||||
await finalizeNativeRun({
|
||||
if (finalize) await finalizeNativeRun({
|
||||
db,
|
||||
runId: context.runId,
|
||||
workspaceFinalizeStatus: "succeeded",
|
||||
@@ -74418,7 +74439,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
.select()
|
||||
.from(nativeRunFinalizations)
|
||||
.where(eq(nativeRunFinalizations.runId, context.runId));
|
||||
expect(coordinator.phase).toBe("committed");
|
||||
if (finalize) expect(coordinator.phase).toBe("committed");
|
||||
else expect(coordinator.phase).not.toBe("committed");
|
||||
await db
|
||||
.update(heartbeatRuns)
|
||||
.set({
|
||||
@@ -74616,6 +74638,11 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
})
|
||||
: context.repair(),
|
||||
).resolves.toBe(false);
|
||||
await expect(recoverCommittedNativeChatResponse(db, {
|
||||
companyId: mutation === "cross_company" ? randomUUID() : context.fixture.companyId,
|
||||
issueId: context.issue.id,
|
||||
runId: mutation === "cross_run" ? randomUUID() : context.runId,
|
||||
})).resolves.toBe(false);
|
||||
await expect(
|
||||
db
|
||||
.select()
|
||||
@@ -75257,6 +75284,519 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["none", "reuse"] as const)("preserves a recorded %s presentation instead of recovering an unsolicited answer", async commentAction => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
const presentationDecision = { commentAction, commentId: null, reasonCodes: ["interaction_owns_response"] };
|
||||
try {
|
||||
await db.update(heartbeatRuns).set({ resultJson: { presentationDecision } }).where(eq(heartbeatRuns.id, context.runId));
|
||||
await expect(recoverCommittedNativeChatResponse(db, { companyId: context.fixture.companyId,
|
||||
issueId: context.issue.id, runId: context.runId })).resolves.toBe(false);
|
||||
expect(await nativeResponsePresentationWork(db).load({ companyId: context.fixture.companyId, runId: context.runId })).toEqual([]);
|
||||
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toEqual([]);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([]);
|
||||
expect((await readHeartbeatRunsById(context.runId))[0].resultJson).toEqual({ presentationDecision });
|
||||
} finally { await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it.each(["create", "legacy", "missing"] as const)("recovers a genuine unanswered %s presentation", async action => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
try {
|
||||
await db.update(heartbeatRuns).set({ resultJson: action === "missing" ? {} : {
|
||||
presentationDecision: { commentId: null, ...(action === "create" ? { commentAction: "create" } : {}) },
|
||||
} }).where(eq(heartbeatRuns.id, context.runId));
|
||||
const work = nativeResponsePresentationWork(db);
|
||||
const entries = await work.load({ companyId: context.fixture.companyId, runId: context.runId });
|
||||
expect(entries).toHaveLength(1);
|
||||
await expect(work.check(entries[0])).resolves.toBeNull();
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
|
||||
await expect(context.repair()).resolves.toBe(false);
|
||||
} finally { await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it.each(["conversation_done", "attention_request", "same_agent_reason", "monitor_reason", "skip_comment"] as const)(
|
||||
"does not lock durably ineligible %s history on repeated issue milestones", async kind => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
await context.service.shutdown();
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded",
|
||||
...(kind === "skip_comment" ? { contextSnapshot: { skipIssueComment: true } } : {}) }).where(eq(heartbeatRuns.id, context.runId));
|
||||
const [accepted] = await db.select().from(nativeRunResults).where(eq(nativeRunResults.id, context.accepted.resultId));
|
||||
const envelope = accepted.resultJson as { result: PrpStructuredRunResult; terminal: PrpTerminalState };
|
||||
if (kind === "conversation_done") {
|
||||
envelope.result = { ...envelope.result, reportedWorkDisposition: "done" };
|
||||
envelope.terminal = { ...envelope.terminal, reportedWorkDisposition: "done" };
|
||||
await db.update(issues).set({ conversationAgentId: context.fixture.assignedAgentId, conversationUserId: "owner-user", conversationState: "active", assigneeUserId: null })
|
||||
.where(eq(issues.id, context.issue.id));
|
||||
} else if (kind === "attention_request") {
|
||||
envelope.result = { ...envelope.result, attentionRequests: [{ kind: "review", summary: "Needs a decision", ownerClass: "human" }] };
|
||||
} else if (kind === "same_agent_reason" || kind === "monitor_reason") {
|
||||
envelope.result = { ...envelope.result, continuation: { ...envelope.result.continuation,
|
||||
kind: kind === "same_agent_reason" ? "same_agent" : "monitor" } as PrpStructuredRunResult["continuation"] };
|
||||
await db.update(statusDecisions).set({ reasonCode: "external_chat_response_waiting" })
|
||||
.where(eq(statusDecisions.id, context.coordinator.decisionId!));
|
||||
}
|
||||
await db.update(nativeRunResults).set({ resultJson: envelope }).where(eq(nativeRunResults.id, accepted.id));
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
const work = nativeResponsePresentationWork(db);
|
||||
const worker = work.register(coordinator, () => true);
|
||||
const debug = db.$client.options.debug;
|
||||
const locks: string[] = [];
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 })
|
||||
.toEqual({ active: 0, next: null });
|
||||
db.$client.options.debug = (_connection, query) => {
|
||||
if (query.startsWith("select") && query.includes("for update") && /"(?:issues|heartbeat_runs|native_run_finalizations)"/.test(query)) locks.push(query);
|
||||
};
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await db.transaction(tx => notifyChatRunMilestones(tx, { companyId: context.fixture.companyId, issueId: context.issue.id }));
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 })
|
||||
.toEqual({ active: 0, next: null });
|
||||
}
|
||||
expect(locks).toEqual([]);
|
||||
expect(await work.load({ companyId: context.fixture.companyId, issueId: context.issue.id })).toEqual([]);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([]);
|
||||
expect(errors).toEqual([]);
|
||||
} finally { db.$client.options.debug = debug; worker.stop(); await coordinator.stop(); }
|
||||
});
|
||||
|
||||
it("leaves fresh-commit accepted-response presentation to the worker when periodic replay skips it", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram", "coordinator", false, undefined, undefined, false);
|
||||
try {
|
||||
await finalizeNativeRun({ db, runId: context.runId, workspaceFinalizeStatus: "succeeded",
|
||||
projectRunStatus: true, skipNativeResponsePresentation: true });
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toEqual([]);
|
||||
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toEqual([]);
|
||||
const work = nativeResponsePresentationWork(db);
|
||||
const entries = await work.load({ companyId: context.fixture.companyId, runId: context.runId });
|
||||
expect(entries).toHaveLength(1);
|
||||
await expect(work.check(entries[0])).resolves.toBeNull();
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
|
||||
} finally { await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it.each(["reconciler", "finalizer"])("leaves accepted-response presentation to its worker during periodic %s replay", async caller => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
try {
|
||||
if (caller === "reconciler") {
|
||||
await reconcileNativeFinalizations(db, [context.runId], { skipNativeResponsePresentation: true });
|
||||
} else {
|
||||
await finalizeNativeRun({ db, runId: context.runId, workspaceFinalizeStatus: "succeeded",
|
||||
projectRunStatus: true, skipNativeResponsePresentation: true });
|
||||
}
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(0);
|
||||
const work = nativeResponsePresentationWork(db);
|
||||
const entries = await work.load({ companyId: context.fixture.companyId, runId: context.runId });
|
||||
expect(entries).toHaveLength(1);
|
||||
await expect(work.check(entries[0])).resolves.toBeNull();
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
|
||||
await expect(context.repair()).resolves.toBe(false);
|
||||
} finally { await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it("recovers accepted-response work on startup and leaves selected history quiet", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
const work = nativeResponsePresentationWork(db);
|
||||
const worker = work.register(coordinator, () => true);
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
|
||||
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
|
||||
const comments = await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId));
|
||||
expect(comments).toHaveLength(1);
|
||||
expect(comments[0].body).toBe(context.result.summary);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
|
||||
expect(await work.load({ companyId: context.fixture.companyId })).toEqual([]);
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled();
|
||||
} finally { reads.mockRestore(); writes.mockRestore(); }
|
||||
expect(errors).toEqual([]);
|
||||
} finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it("reactivates dormant accepted-response work when the live chat authority registers", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
await context.service.shutdown();
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
const worker = nativeResponsePresentationWork(db).register(coordinator, () => true);
|
||||
let replacement: ReturnType<typeof createService> | undefined;
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
|
||||
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
replacement = createService(new FakeChatSdkRuntime(), fakeTelegramFetch() as typeof fetch);
|
||||
await expect.poll(async () => (await readChatPublicationsByEndpointId(context.endpoint.id)).length, { timeout: 5000 }).toBe(1);
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
|
||||
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
|
||||
expect(errors).toEqual([]);
|
||||
} finally { worker.stop(); await coordinator.stop(); await replacement?.service.shutdown(); }
|
||||
});
|
||||
|
||||
it.each(["issue", "run"])("does not reload dormant accepted-response history for an unrelated %s wake", async scope => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
await context.service.shutdown();
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
const work = nativeResponsePresentationWork(db);
|
||||
const worker = work.register(coordinator, () => true);
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
|
||||
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
|
||||
expect(await work.load({ companyId: context.fixture.companyId, runId: context.runId })).toHaveLength(1);
|
||||
const previousDebug = db.$client.options.debug;
|
||||
const queries: string[] = [];
|
||||
db.$client.options.debug = (_connection, query) => {
|
||||
if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query);
|
||||
};
|
||||
try {
|
||||
await db.transaction(tx => notifyChatRunMilestones(tx, { companyId: context.fixture.companyId,
|
||||
...(scope === "issue" ? { issueId: randomUUID() } : { runId: randomUUID() }) }));
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
|
||||
{ timeout: 5000 }).toEqual({ active: 0, next: null });
|
||||
expect(queries).toHaveLength(1);
|
||||
} finally { db.$client.options.debug = previousDebug; }
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
expect(errors).toEqual([]);
|
||||
} finally { worker.stop(); await coordinator.stop(); }
|
||||
});
|
||||
|
||||
it.each(["milestone", "finalization"] as const)("ignores repeated unscoped routine %s hints without rebuilding dormant accepted-response history", async source => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
await context.service.shutdown();
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
const work = nativeResponsePresentationWork(db);
|
||||
const worker = work.register(coordinator, () => true);
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 })
|
||||
.toEqual({ active: 0, next: null });
|
||||
expect(await work.load({ companyId: context.fixture.companyId, runId: context.runId })).toHaveLength(1);
|
||||
const debug = db.$client.options.debug;
|
||||
const queries: string[] = [];
|
||||
db.$client.options.debug = (_connection, query) => {
|
||||
if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query);
|
||||
};
|
||||
try {
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await db.transaction(tx => source === "milestone"
|
||||
? notifyChatRunMilestones(tx, { companyId: context.fixture.companyId })
|
||||
: notifyNativeFinalizationWork(tx, { companyId: context.fixture.companyId }));
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 })
|
||||
.toEqual({ active: 0, next: null });
|
||||
}
|
||||
expect(queries).toEqual([]);
|
||||
} finally { db.$client.options.debug = debug; }
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
expect(errors).toEqual([]);
|
||||
} finally { worker.stop(); await coordinator.stop(); }
|
||||
});
|
||||
|
||||
it("wakes dormant accepted-response history from the real company membership restoration owner", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
const [membership] = await db.select().from(companyMemberships).where(and(
|
||||
eq(companyMemberships.companyId, context.fixture.companyId), eq(companyMemberships.principalId, "owner-user")));
|
||||
await db.update(companyMemberships).set({ status: "suspended" }).where(eq(companyMemberships.id, membership.id));
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
const worker = nativeResponsePresentationWork(db).register(coordinator, () => true);
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 10_000 })
|
||||
.toEqual({ active: 0, next: null });
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
await accessService(db).updateMember(context.fixture.companyId, membership.id, { status: "active" });
|
||||
await expect.poll(async () => ({ count: (await readChatPublicationsByEndpointId(context.endpoint.id)).length,
|
||||
active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }), { timeout: 5000 })
|
||||
.toEqual({ count: 1, active: 0, next: null });
|
||||
expect(errors).toEqual([]);
|
||||
} finally { worker.stop(); await coordinator.stop(); await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it.each(["commit", "rollback"])("keeps accepted-response work behind the outer %s boundary", async outcome => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, context.runId));
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
const worker = nativeResponsePresentationWork(db).register(coordinator, () => true);
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
|
||||
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
|
||||
const transaction = db.transaction(async tx => {
|
||||
await tx.transaction(async nested => {
|
||||
await notifyChatRunMilestones(nested, { companyId: context.fixture.companyId, issueId: context.issue.id, runId: randomUUID() });
|
||||
await nested.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, context.runId));
|
||||
});
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
expect(idleWorkSnapshot().active).toBeGreaterThan(0);
|
||||
if (outcome === "rollback") throw new Error("response fixture rollback");
|
||||
});
|
||||
if (outcome === "rollback") await expect(transaction).rejects.toThrow("response fixture rollback");
|
||||
else await transaction;
|
||||
await expect.poll(async () => ({
|
||||
publications: (await readChatPublicationsByEndpointId(context.endpoint.id)).length,
|
||||
active: idleWorkSnapshot().active,
|
||||
next: coordinator.nextWakeAt(),
|
||||
}), { timeout: 5000 }).toEqual({ publications: outcome === "commit" ? 1 : 0, active: 0, next: null });
|
||||
expect(errors).toEqual([]);
|
||||
} finally { worker.stop(); await coordinator.stop(); await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it("retains the first accepted-response wake through idle drain without passive SQL", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, context.runId));
|
||||
let canRun = true;
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun, canReconcile: () => true, onError: error => errors.push(error) });
|
||||
const worker = nativeResponsePresentationWork(db).register(coordinator, () => canRun);
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
|
||||
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
|
||||
canRun = false;
|
||||
await db.transaction(async tx => {
|
||||
await notifyChatRunMilestones(tx, { companyId: context.fixture.companyId, issueId: context.issue.id });
|
||||
await tx.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, context.runId));
|
||||
});
|
||||
await expect.poll(() => idleWorkSnapshot().active, { timeout: 5000 }).toBe(0);
|
||||
expect(coordinator.nextWakeAt()).not.toBeNull();
|
||||
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled();
|
||||
} finally { reads.mockRestore(); writes.mockRestore(); }
|
||||
canRun = true; worker.wake();
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 5000 });
|
||||
expect(errors).toEqual([]);
|
||||
} finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it("retains a failed accepted-response attempt across drain and retries after resume", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
let release!: () => void, observed!: () => void;
|
||||
const hold = new Promise<void>(resolve => { release = resolve; });
|
||||
const locked = new Promise<void>(resolve => { observed = resolve; });
|
||||
const holder = db.transaction(async tx => {
|
||||
await tx.select().from(chatActions).where(eq(chatActions.id, context.action.id)).for("update");
|
||||
observed();
|
||||
await hold;
|
||||
});
|
||||
await locked;
|
||||
let canRun = true;
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => canRun, canReconcile: () => true, onError: error => errors.push(error) });
|
||||
const worker = nativeResponsePresentationWork(db).register(coordinator, () => canRun);
|
||||
try {
|
||||
await worker.ready;
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(errors).toHaveLength(1);
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
}, { timeout: 5000 });
|
||||
expect(coordinator.nextWakeAt()).toBeGreaterThan(Date.now());
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
canRun = false;
|
||||
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
|
||||
try {
|
||||
await vi.advanceTimersByTimeAsync(120_000);
|
||||
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled();
|
||||
} finally { reads.mockRestore(); writes.mockRestore(); }
|
||||
release(); await holder;
|
||||
canRun = true; worker.wake();
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 5000 });
|
||||
expect(errors).toHaveLength(1);
|
||||
} finally { release(); await holder; worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it("reconstructs a future accepted-response deadline without early publication", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
const at = new Date(Date.now() + 60_000);
|
||||
await db.update(nativeRunFinalizations).set({ nextAttemptAt: at }).where(eq(nativeRunFinalizations.runId, context.runId));
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
const worker = nativeResponsePresentationWork(db).register(coordinator, () => true);
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
|
||||
{ timeout: 10_000 }).toEqual({ active: 0, next: at.getTime() });
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
await vi.advanceTimersByTimeAsync(Math.max(0, at.getTime() - Date.now()));
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 5000 });
|
||||
expect(errors).toEqual([]);
|
||||
} finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it("retains an accepted-response deadline that elapses during the locked recovery check", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
const work = nativeResponsePresentationWork(db);
|
||||
const [entry] = await work.load({ companyId: context.fixture.companyId });
|
||||
expect(entry?.id).toBe(context.runId);
|
||||
vi.useFakeTimers({ toFake: ["Date"] });
|
||||
const due = new Date(Date.now() + 1000);
|
||||
const transaction = db.transaction.bind(db);
|
||||
const intercept = vi.spyOn(db, "transaction").mockImplementationOnce((async (callback, options) => {
|
||||
await transaction(async tx => {
|
||||
await tx.update(nativeRunFinalizations).set({ nextAttemptAt: due }).where(eq(nativeRunFinalizations.runId, context.runId));
|
||||
});
|
||||
const result = await transaction(callback, options);
|
||||
vi.setSystemTime(due.getTime() + 1);
|
||||
return result;
|
||||
}) as Db["transaction"]);
|
||||
try {
|
||||
const deferred = await work.check(entry);
|
||||
intercept.mockRestore();
|
||||
expect(deferred).toEqual({ ...entry, dueAt: due.getTime() });
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
await expect(work.check(deferred!)).resolves.toBeNull();
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
|
||||
} finally { intercept.mockRestore(); vi.useRealTimers(); await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it("wakes dormant accepted-response recovery after agent resume", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
await db.update(agents).set({ status: "paused", lifecycleState: "paused", lifecycleHolds: ["manual"], lifecycleOperation: null })
|
||||
.where(eq(agents.id, context.binding.agentId));
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
const work = nativeResponsePresentationWork(db);
|
||||
const worker = work.register(coordinator, () => true);
|
||||
const lifecycle = configureAgentLifecycle(db, {
|
||||
requiredPluginIds: async () => [], runHost: async () => "complete", runPlugin: async () => "complete",
|
||||
});
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
|
||||
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
|
||||
expect(await work.load({ companyId: context.fixture.companyId, agentId: randomUUID() })).toEqual([]);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
await createAgentLifecycle(db).resumeAgent(context.binding.agentId);
|
||||
await processAgentLifecycle(db, context.binding.agentId);
|
||||
await expect.poll(async () => (await db.select().from(agents).where(eq(agents.id, context.binding.agentId)))[0].status).toBe("idle");
|
||||
await expect.poll(async () => (await readChatPublicationsByEndpointId(context.endpoint.id)).length, { timeout: 5000 }).toBe(1);
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
|
||||
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
|
||||
expect(errors).toEqual([]);
|
||||
} finally { await lifecycle.stop(); worker.stop(); await coordinator.stop(); await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it("wakes accepted-response recovery from the real coordinator retry release", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
const store = new NativeRunCoordinatorStore(db, context.binding);
|
||||
await store.claimFinalization({ leaseOwner: "response-recovery-fixture", leaseTtlMs: 1000 });
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
const worker = nativeResponsePresentationWork(db).register(coordinator, () => true);
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
|
||||
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
|
||||
await vi.advanceTimersByTimeAsync(2000);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
await store.markFinalizationRetry({ leaseOwner: "response-recovery-fixture", failureCode: "response_fixture_retry", retryAfterMs: 1000 });
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 5000 });
|
||||
expect(errors).toEqual([]);
|
||||
} finally { worker.stop(); await coordinator.stop(); vi.useRealTimers(); await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it("wakes accepted-response recovery when its authenticated accepted-result journal arrives late", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram", "port");
|
||||
await db.delete(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, context.runId), eq(heartbeatRunEvents.eventType, "run.result.accepted")));
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
const worker = nativeResponsePresentationWork(db).register(coordinator, () => true);
|
||||
try {
|
||||
await worker.ready;
|
||||
await expect.poll(() => ({ active: idleWorkSnapshot().active, next: coordinator.nextWakeAt() }),
|
||||
{ timeout: 10_000 }).toEqual({ active: 0, next: null });
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
const port = new PaperclipControlPlanePort(db, {
|
||||
companyId: context.binding.companyId, issueId: context.binding.issueId,
|
||||
runId: context.runId, agentId: context.binding.agentId, sessionId: context.binding.normalizedSessionId,
|
||||
completionContractId: context.binding.completionContractId,
|
||||
completionContractSha256: context.binding.completionContractSha256,
|
||||
sourceInstanceId: context.binding.runnerSourceInstanceId, controlPlaneSourceInstanceId: `control-${context.runId}`,
|
||||
});
|
||||
await port.appendEvent({
|
||||
schema: "paperclip.prp.event.v1", sourceEventId: `accepted-${context.runId}`, sourceSeq: 1,
|
||||
sourceInstanceId: `control-${context.runId}`, sourceKind: "control_plane",
|
||||
runId: context.runId, normalizedSessionId: context.binding.normalizedSessionId,
|
||||
turnId: `turn-${context.runId}`, eventType: "run.result.accepted", schemaVersion: 1,
|
||||
priority: 0, emittedAt: new Date().toISOString(), payload: { result: context.result },
|
||||
});
|
||||
await expect.poll(async () => ({
|
||||
publications: (await readChatPublicationsByEndpointId(context.endpoint.id)).length,
|
||||
active: idleWorkSnapshot().active, next: coordinator.nextWakeAt(),
|
||||
}), { timeout: 5000 }).toEqual({ publications: 1, active: 0, next: null });
|
||||
expect(errors).toEqual([]);
|
||||
} finally { worker.stop(); await coordinator.stop(); await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it("defers scheduled accepted-response recovery until the persisted retry is due", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
const input = { companyId: context.fixture.companyId, issueId: context.issue.id, runId: context.runId, onlyWhenDue: true };
|
||||
try {
|
||||
await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(Date.now() + 60_000) })
|
||||
.where(eq(nativeRunFinalizations.runId, context.runId));
|
||||
await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(false);
|
||||
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(0);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
await db.update(nativeRunFinalizations).set({ nextAttemptAt: new Date(0) })
|
||||
.where(eq(nativeRunFinalizations.runId, context.runId));
|
||||
await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(true);
|
||||
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(1);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
|
||||
} finally { await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it.each(["pending", "failed"])("defers scheduled accepted-response recovery behind %s status effects", async deliveryState => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
const input = { companyId: context.fixture.companyId, issueId: context.issue.id, runId: context.runId, onlyWhenDue: true };
|
||||
try {
|
||||
const [effect] = await db.select().from(statusDecisionEffects).where(eq(statusDecisionEffects.decisionId, context.coordinator.decisionId!));
|
||||
expect(effect).toBeDefined();
|
||||
await db.update(statusDecisionEffects).set({ deliveryState }).where(eq(statusDecisionEffects.id, effect.id));
|
||||
await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(false);
|
||||
expect(await db.select().from(issueComments).where(eq(issueComments.createdByRunId, context.runId))).toHaveLength(0);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(0);
|
||||
await db.update(statusDecisionEffects).set({ deliveryState: "delivered" }).where(eq(statusDecisionEffects.id, effect.id));
|
||||
await expect(recoverCommittedNativeChatResponse(db, input)).resolves.toBe(true);
|
||||
expect(await readChatPublicationsByEndpointId(context.endpoint.id)).toHaveLength(1);
|
||||
} finally { await context.service.shutdown(); }
|
||||
});
|
||||
|
||||
it("leaves a contended accepted response retryable without reclassifying or partially publishing it", async () => {
|
||||
const context = await committedChatResponseRecoveryFixture("telegram");
|
||||
let release!: () => void;
|
||||
@@ -75279,6 +75819,9 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
try {
|
||||
await locked;
|
||||
await expect(context.repair()).resolves.toBe(false);
|
||||
await expect(recoverCommittedNativeChatResponse(db, {
|
||||
companyId: context.fixture.companyId, issueId: context.issue.id, runId: context.runId,
|
||||
})).rejects.toThrow();
|
||||
await expect(
|
||||
db
|
||||
.select()
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
import { isNativeResponseEvidence } from "../services/native-response-presentation-notifications.js";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { authorizeCommittedChatResponse, onCommittedChatResponseAuthorityRegistered, registerCommittedChatResponseAuthority } from "../services/durable-chat-wakeup.js";
|
||||
|
||||
describe("committed response authority lifecycle", () => {
|
||||
it("reactivates only its database and unsubscribes without removing a replacement listener", () => {
|
||||
const db = {} as Db, otherDb = {} as Db;
|
||||
const first = vi.fn(), second = vi.fn(), other = vi.fn();
|
||||
const disposeFirst = onCommittedChatResponseAuthorityRegistered(db, first);
|
||||
const disposeOther = onCommittedChatResponseAuthorityRegistered(otherDb, other);
|
||||
const removeAuthority = registerCommittedChatResponseAuthority(db, async () => {});
|
||||
expect(first).toHaveBeenCalledTimes(1);
|
||||
expect(other).not.toHaveBeenCalled();
|
||||
disposeFirst();
|
||||
const disposeSecond = onCommittedChatResponseAuthorityRegistered(db, second);
|
||||
disposeFirst();
|
||||
const removeReplacement = registerCommittedChatResponseAuthority(db, async () => {});
|
||||
expect(first).toHaveBeenCalledTimes(1);
|
||||
expect(second).toHaveBeenCalledTimes(1);
|
||||
removeAuthority();
|
||||
expect(second).toHaveBeenCalledTimes(1);
|
||||
disposeSecond(); disposeOther(); removeReplacement();
|
||||
const removeLast = registerCommittedChatResponseAuthority(db, async () => {});
|
||||
expect(second).toHaveBeenCalledTimes(1);
|
||||
removeLast();
|
||||
});
|
||||
|
||||
it("retains the replacement authority when the old service shuts down", async () => {
|
||||
const db = {} as Db;
|
||||
const first = vi.fn(async () => {}), second = vi.fn(async () => {});
|
||||
const removeFirst = registerCommittedChatResponseAuthority(db, first);
|
||||
const removeSecond = registerCommittedChatResponseAuthority(db, second);
|
||||
try {
|
||||
removeFirst();
|
||||
const input = { companyId: "company", issueId: "issue", agentId: "agent", runId: "run", resultId: "result" };
|
||||
await authorizeCommittedChatResponse(db, db, input);
|
||||
expect(first).not.toHaveBeenCalled();
|
||||
expect(second).toHaveBeenCalledExactlyOnceWith(db, input);
|
||||
} finally { removeSecond(); }
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
describe("accepted response evidence selection", () => {
|
||||
it.each([
|
||||
{ type: "item.delta", kind: "agentMessage", channel: "final", expected: false },
|
||||
{ type: "item.completed", kind: "agentMessage", channel: "analysis", expected: false },
|
||||
{ type: "item.completed", kind: "commandExecution", channel: "final", expected: false },
|
||||
{ type: "item.completed", kind: "agentMessage", channel: "final", expected: true },
|
||||
{ type: "run.result.accepted", kind: "agentMessage", channel: "final", expected: true },
|
||||
])("signals only response evidence: $type/$kind/$channel", ({ type, kind, channel, expected }) => {
|
||||
expect(isNativeResponseEvidence(type, { payload: { kind, channel, text: "Saved evidence" } })).toBe(expected);
|
||||
});
|
||||
});
|
||||
@@ -116,6 +116,7 @@ const {
|
||||
reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })),
|
||||
registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerNativeResponsePresentationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })),
|
||||
registerNativePendingEffectWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerUnsafeWorkspaceExportWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerAutomaticCompletionReviewWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
@@ -558,6 +559,17 @@ describe("startServer feedback export wiring", () => {
|
||||
process.env.BETTER_AUTH_SECRET = "test-secret";
|
||||
});
|
||||
|
||||
it.each([false, true])("registers response recovery and its shutdown subscription only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
expect(heartbeatServiceMock.registerNativeResponsePresentationWork).toHaveBeenCalledTimes(enabled ? 1 : 0);
|
||||
if (enabled) {
|
||||
expect(heartbeatServiceMock.registerNativeResponsePresentationWork.mock.calls[0][1]()).toBe(true);
|
||||
const worker = heartbeatServiceMock.registerNativeResponsePresentationWork.mock.results[0].value;
|
||||
expect(fakeServer.on).toHaveBeenCalledWith("close", worker.stop);
|
||||
}
|
||||
});
|
||||
|
||||
it.each([false, true])("registers native pending effects only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
@@ -1005,7 +1017,7 @@ describe("startServer feedback export wiring", () => {
|
||||
intervalCallback?.();
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2);
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true });
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true });
|
||||
expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1);
|
||||
expect(retiredDetector).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
|
||||
@@ -4,7 +4,7 @@ import type { Db } from "@paperclipai/db";
|
||||
import { companies, companyMemberships, instanceUserRoles } from "@paperclipai/db";
|
||||
import type { DeploymentMode } from "@paperclipai/shared";
|
||||
import { ensureHumanRoleDefaultGrants } from "./services/principal-access-compatibility.js";
|
||||
import { notifyChatRunMilestones } from "./services/chat-run-milestone-notifications.js";
|
||||
import { notifyChatCompanyAuthorityChange } from "./services/chat-run-milestone-notifications.js";
|
||||
|
||||
const LOCAL_BOARD_USER_ID = "local-board";
|
||||
const CLAIM_TTL_MS = 1000 * 60 * 60 * 24;
|
||||
@@ -125,7 +125,7 @@ export async function claimBoardOwnership(
|
||||
.then((rows) => rows[0] ?? null);
|
||||
|
||||
if (!existing) {
|
||||
await notifyChatRunMilestones(tx, { companyId: company.id });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId: company.id });
|
||||
await tx.insert(companyMemberships).values({
|
||||
companyId: company.id,
|
||||
principalType: "user",
|
||||
@@ -137,7 +137,7 @@ export async function claimBoardOwnership(
|
||||
}
|
||||
|
||||
if (existing.status !== "active") {
|
||||
await notifyChatRunMilestones(tx, { companyId: company.id });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId: company.id });
|
||||
await tx
|
||||
.update(companyMemberships)
|
||||
.set({ status: "active", membershipRole: "owner", updatedAt: new Date() })
|
||||
|
||||
+10
-1
@@ -1403,6 +1403,7 @@ async function startServerWithDatabaseTeardown(
|
||||
hasPending: () => caseHandoffDeliveries.hasPending(),
|
||||
});
|
||||
let unregisterTerminalWorkspaceProbe: (() => void) | undefined;
|
||||
let unsubscribeNativeResponsePresentation: (() => void) | undefined;
|
||||
let unsubscribeNativeWorkspaceSyncRetention: (() => void) | undefined;
|
||||
const fastResponses = app.locals.fastResponses as ReturnType<typeof import("./services/fast-responses.js").fastResponseService>;
|
||||
const fastResponseWorker = deliveryWork.register(DELIVERY_QUEUES.fastResponse, {
|
||||
@@ -1824,6 +1825,13 @@ async function startServerWithDatabaseTeardown(
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) {
|
||||
const worker = heartbeat.registerNativeResponsePresentationWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
|
||||
unsubscribeNativeResponsePresentation = worker.stop;
|
||||
server.on("close", worker.stop);
|
||||
await worker.ready;
|
||||
}
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativePendingEffectWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerUnsafeWorkspaceExportWork(deliveryWork,
|
||||
@@ -1964,7 +1972,7 @@ async function startServerWithDatabaseTeardown(
|
||||
// Periodically reap orphaned runs (5-min staleness threshold) and make sure
|
||||
// persisted queued work is still being driven forward.
|
||||
runPeriodicRecovery(() => heartbeat
|
||||
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true })
|
||||
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true })
|
||||
.then(() => heartbeat.promoteDueScheduledRetries())
|
||||
.then(async (promotion) => {
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
@@ -2160,6 +2168,7 @@ async function startServerWithDatabaseTeardown(
|
||||
await skillVersionRetention.dispose();
|
||||
await skillSourceSync.stop();
|
||||
unsubscribeChatCompletions();
|
||||
unsubscribeNativeResponsePresentation?.();
|
||||
await deliveryWork.stop();
|
||||
await workspaceRecoveryWork.stop();
|
||||
unregisterTerminalWorkspaceProbe?.();
|
||||
|
||||
@@ -30,7 +30,7 @@ import { captureRunIdentity } from "../services/run-identity.js";
|
||||
import { boardAuthService } from "../services/board-auth.js";
|
||||
import { retryIdempotentDatabaseOperation } from "../database-retry.js";
|
||||
import { beginIdleTrackedWork } from "../services/task-admission.js";
|
||||
import { notifyChatRunMilestones } from "../services/chat-run-milestone-notifications.js";
|
||||
import { notifyChatCompanyAuthorityChange } from "../services/chat-run-milestone-notifications.js";
|
||||
|
||||
export {
|
||||
isTransientDbConnectionError,
|
||||
@@ -665,7 +665,7 @@ async function resolveCloudTenantActorOnce(
|
||||
// Serialize authority restoration and its notice while retaining the
|
||||
// guarded upsert that leaves unchanged memberships untouched.
|
||||
if (!existing || existing.status !== "active" || existing.membershipRole === "viewer") {
|
||||
await notifyChatRunMilestones(tx, { companyId });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId });
|
||||
}
|
||||
return tx
|
||||
.insert(companyMemberships)
|
||||
|
||||
@@ -673,7 +673,7 @@ export function createPostgresRunDispatchAdapter(
|
||||
await notifyInstructionCollection(tx, { companyId: input.companyId, runId: input.runId });
|
||||
await notifyCancelledWorkspacePreservation(tx, { companyId: input.companyId, runId: input.runId });
|
||||
await notifyScheduledRetryCancellation(tx, { companyId: input.companyId, runId: input.runId });
|
||||
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: issueId ?? undefined });
|
||||
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: issueId ?? undefined, runId: input.runId });
|
||||
const [row] = await tx
|
||||
.update(heartbeatRuns)
|
||||
.set({
|
||||
@@ -719,7 +719,7 @@ export function createPostgresRunDispatchAdapter(
|
||||
await notifyCancelledWorkspacePreservation(tx, { companyId: input.companyId, runId: input.runId });
|
||||
await notifyScheduledRetryCancellation(tx, { companyId: input.companyId, runId: input.runId });
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined });
|
||||
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined });
|
||||
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId ?? undefined, runId: input.runId });
|
||||
const [row] = await tx
|
||||
.update(heartbeatRuns)
|
||||
.set({
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { DELIVERY_QUEUES } from "../../../services/delivery-work-notifications.js";
|
||||
import { instanceSettingsService } from "../../../services/instance-settings.js";
|
||||
import { randomBytes, randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
@@ -16,6 +17,7 @@ import {
|
||||
issueRecoveryActions,
|
||||
issues,
|
||||
nativeRunFinalizations,
|
||||
subscribeDatabaseWork,
|
||||
} from "@paperclipai/db";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
@@ -1222,6 +1224,30 @@ describeEmbeddedPostgres("wake-queue postgres adapter", () => {
|
||||
expect(wakeRow?.payload).not.toHaveProperty("foo");
|
||||
});
|
||||
|
||||
it("scopes a real coalesced-input response wake to its active run after commit", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const agentId = await seedAgent({ companyId });
|
||||
const runId = await seedRun({ companyId, agentId, status: "running" });
|
||||
const hints: Array<Record<string, unknown>> = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeResponsePresentation, (event, values) => {
|
||||
if (event === "settled") for (const value of values ?? []) hints.push(JSON.parse(value));
|
||||
});
|
||||
try {
|
||||
await db.transaction(async tx => {
|
||||
await createWakeAdmissionWriter().coalesceIntoActiveExecutionRun(createAdmissionTransactionScope(companyId, tx as unknown as Db), {
|
||||
companyId, activeExecutionRunId: runId, mergedContextSnapshot: { commentId: "fresh-input" },
|
||||
agentId, source: "on_demand", triggerDetail: null, payload: null,
|
||||
requestedByActorType: "user", requestedByActorId: null, idempotencyKey: null,
|
||||
});
|
||||
expect(hints).toEqual([]);
|
||||
});
|
||||
expect(hints).toEqual([{ companyId, runId }]);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0].contextSnapshot)
|
||||
.toMatchObject({ commentId: "fresh-input" });
|
||||
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.runId, runId))).toHaveLength(1);
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
|
||||
// Review test (b), the coalesce write: a foreign company affects no row
|
||||
// on the active execution run either.
|
||||
it("refuses to coalesce into a run for a company that does not own it, and leaves the run and the wake table untouched", async () => {
|
||||
|
||||
@@ -956,7 +956,7 @@ export function createWakeAdmissionWriter(): WakeAdmissionWriter {
|
||||
async coalesceIntoActiveExecutionRun(scope, input) {
|
||||
const tx = requireAdmissionTx(scope, input.companyId);
|
||||
const now = new Date();
|
||||
await notifyChatRunMilestones(tx, { companyId: input.companyId });
|
||||
await notifyChatRunMilestones(tx, { companyId: input.companyId, runId: input.activeExecutionRunId });
|
||||
const mergedRun = await tx
|
||||
.update(heartbeatRuns)
|
||||
.set({ contextSnapshot: input.mergedContextSnapshot, updatedAt: now })
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js";
|
||||
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
|
||||
import { notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js";
|
||||
import { notifyVoiceChange } from "./voice/voice-recovery-notifications.js";
|
||||
import { notifyChatEndpointWork } from "./chat-work-notifications.js";
|
||||
import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js";
|
||||
@@ -558,7 +558,7 @@ export function accessService(db: Db) {
|
||||
|
||||
if (existing.principalType === "user" && nextStatus === "active" && nextMembershipRole !== "viewer"
|
||||
&& (existing.status !== "active" || existing.membershipRole === "viewer")) {
|
||||
await notifyChatRunMilestones(tx, { companyId });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId });
|
||||
}
|
||||
|
||||
if (
|
||||
@@ -894,7 +894,7 @@ export function accessService(db: Db) {
|
||||
const existingMembership = existingByCompany.get(companyId);
|
||||
if (existingMembership) {
|
||||
if (existingMembership.status !== "active") {
|
||||
if (existingMembership.membershipRole !== "viewer") await notifyChatRunMilestones(tx, { companyId });
|
||||
if (existingMembership.membershipRole !== "viewer") await notifyChatCompanyAuthorityChange(tx, { companyId });
|
||||
await tx
|
||||
.update(companyMemberships)
|
||||
.set({
|
||||
@@ -906,7 +906,7 @@ export function accessService(db: Db) {
|
||||
}
|
||||
continue;
|
||||
}
|
||||
await notifyChatRunMilestones(tx, { companyId });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId });
|
||||
await tx.insert(companyMemberships).values({
|
||||
companyId,
|
||||
principalType: "user",
|
||||
@@ -936,7 +936,7 @@ export function accessService(db: Db) {
|
||||
if (existing.status !== status || existing.membershipRole !== membershipRole) {
|
||||
if (principalType === "user" && status === "active" && membershipRole !== "viewer"
|
||||
&& (existing.status !== "active" || existing.membershipRole === "viewer")) {
|
||||
await notifyChatRunMilestones(tx, { companyId });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId });
|
||||
}
|
||||
const updated = await tx
|
||||
.update(companyMemberships)
|
||||
@@ -950,7 +950,7 @@ export function accessService(db: Db) {
|
||||
}
|
||||
|
||||
if (principalType === "user" && status === "active" && membershipRole !== "viewer") {
|
||||
await notifyChatRunMilestones(tx, { companyId });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId });
|
||||
}
|
||||
return tx
|
||||
.insert(companyMemberships)
|
||||
@@ -1146,7 +1146,7 @@ export function accessService(db: Db) {
|
||||
|
||||
if (existing.principalType === "user" && nextStatus === "active" && nextMembershipRole !== "viewer"
|
||||
&& (existing.status !== "active" || existing.membershipRole === "viewer")) {
|
||||
await notifyChatRunMilestones(tx, { companyId });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId });
|
||||
}
|
||||
|
||||
if (
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { isAgentStatusInvokable } from "@paperclipai/shared";
|
||||
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
|
||||
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
|
||||
import { logger } from "../middleware/logger.js";
|
||||
import { companies, type Db } from "@paperclipai/db";
|
||||
@@ -35,8 +36,9 @@ export function createAgentLifecycleEffects(_hooks: BudgetServiceHooks = {}): li
|
||||
return true;
|
||||
},
|
||||
notifyWork: tx => notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle),
|
||||
async beforeStatusChange(tx, companyId, _agentId, before, after) {
|
||||
async beforeStatusChange(tx, companyId, agentId, before, after) {
|
||||
if (!isAgentStatusInvokable(before) && isAgentStatusInvokable(after)) {
|
||||
await notifyNativeResponsePresentation(tx, { companyId, agentId });
|
||||
await notifyChatInboundWakeNotices(tx, { companyId });
|
||||
}
|
||||
},
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
|
||||
import { notifyChatCompanyAuthorityChange } from "../chat-run-milestone-notifications.js";
|
||||
import { chatConnectionTransport } from "./provider-catalog.js";
|
||||
import type { ConnectionManagementContext } from "../connection-management.js";
|
||||
import { slackSetupState } from "./slack/setup.js";
|
||||
@@ -458,7 +458,7 @@ function providerSetupState(
|
||||
if ((input.allowDirectMessages === true && !existing.endpoint.allowDirectMessages)
|
||||
|| (input.allowGroupChats === true && !existing.endpoint.allowGroupChats)
|
||||
|| (input.allowUnlinkedPeople === true && !existing.endpoint.allowUnlinkedPeople)) {
|
||||
await notifyChatRunMilestones(tx, { companyId: existing.endpoint.companyId });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId: existing.endpoint.companyId });
|
||||
}
|
||||
if (input.executionDefaults !== undefined) {
|
||||
await assertChatExecutionDefaultsAccess(db, {
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeResponsePresentation } from "../native-response-presentation-notifications.js";
|
||||
import { notifyChatInboundWakeNotices } from "../chat-inbound-wake-notice-notifications.js";
|
||||
import { createFailedRetryNotices } from "./failed-retry-notices.js";
|
||||
import { safeChatWakeAuthorityReason, type ChatWakeAuthorityReason } from "./inbound/wakeup-authorization-error.js";
|
||||
@@ -1784,6 +1785,9 @@ export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOpti
|
||||
const settle = async (status: string, result: Record<string, unknown>) => {
|
||||
const [current] = await db.transaction(async (tx) => {
|
||||
await notifyChatActionWork(tx, "failed_run_retry");
|
||||
if (status === "processed") await notifyNativeResponsePresentation(tx, {
|
||||
companyId: claimed.companyId, issueId: String(claimed.payload.issueId),
|
||||
});
|
||||
return tx
|
||||
.update(chatActions)
|
||||
.set({ status, result, updatedAt: new Date() })
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
|
||||
import { notifyChatCompanyAuthorityChange } from "../chat-run-milestone-notifications.js";
|
||||
import { githubIdentityFields } from "./github/identity-links.js";
|
||||
import { createSlackIdentityLinks, type SlackIdentityEffectStager } from "./slack/identity-links.js";
|
||||
import { createHash, randomBytes } from "node:crypto";
|
||||
@@ -320,7 +320,7 @@ export function createChatIdentityLinks(db: Db, options: ChatIdentityLinkOptions
|
||||
},
|
||||
);
|
||||
}
|
||||
await notifyChatRunMilestones(tx, { companyId: currentLink.companyId });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId: currentLink.companyId });
|
||||
const confirmedLink = await tx
|
||||
.update(chatIdentityLinks)
|
||||
.set({
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyChatRunMilestones } from "../chat-run-milestone-notifications.js";
|
||||
import { notifyChatRunMilestones, notifyChatCompanyAuthorityChange } from "../chat-run-milestone-notifications.js";
|
||||
import { notifyChatEndpointConfiguration } from "./configuration-work-notifications.js";
|
||||
import { notifyChatConnectionHealth } from "./connection-health-notifications.js";
|
||||
import type { ConnectionManagementContext } from "../connection-management.js";
|
||||
@@ -287,7 +287,7 @@ export function createChatResourceReconciliation(db: Db, options: ResourceReconc
|
||||
after: { enabled: finalEnabled.get(row.id)!, ...(JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id)) ? { executionDefaults: finalDefaults.get(row.id) } : {}) },
|
||||
}));
|
||||
if (changes.some(change => !change.before.enabled && change.after.enabled)) {
|
||||
await notifyChatRunMilestones(tx, { companyId: endpoint.companyId });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId: endpoint.companyId });
|
||||
}
|
||||
if (bulk) {
|
||||
// One atomic database update, independent of pagination/search and
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyChatRunMilestones } from "../../chat-run-milestone-notifications.js";
|
||||
import { notifyChatCompanyAuthorityChange } from "../../chat-run-milestone-notifications.js";
|
||||
import { chatIdentityLinks, chatExternalPrincipals, companyMemberships, toolConnections, type Db } from "@paperclipai/db";
|
||||
import { and, eq, ne, sql } from "drizzle-orm";
|
||||
import { conflict, forbidden, notFound } from "../../../errors.js";
|
||||
@@ -47,7 +47,7 @@ export function createSlackSetupServices(db: Db, options: SetupServicesOptions)
|
||||
const links = await tx.select().from(chatIdentityLinks).where(and(eq(chatIdentityLinks.companyId, record.endpoint.companyId), eq(chatIdentityLinks.principalId, principal.id))).for("update");
|
||||
if (links.some(link => link.status === "linked" && link.paperclipUserId !== actor.userId) || links.some(link => link.endpointId === endpointId && link.status === "revoked")) throw conflict("This Slack account has an existing link", { code: "chat_identity_link_conflict" });
|
||||
if (links.some(link => link.endpointId === endpointId && link.status === "linked")) return;
|
||||
await notifyChatRunMilestones(tx, { companyId: record.endpoint.companyId });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId: record.endpoint.companyId });
|
||||
await tx.insert(chatIdentityLinks).values({ companyId: record.endpoint.companyId, endpointId, principalId: principal.id,
|
||||
paperclipUserId: actor.userId, status: "linked", confirmedAt: new Date() })
|
||||
.onConflictDoUpdate({ target: [chatIdentityLinks.endpointId, chatIdentityLinks.principalId], set: {
|
||||
|
||||
@@ -24,7 +24,7 @@ import {
|
||||
import { HttpError, badRequest, conflict, forbidden, notFound, unprocessable } from "../errors.js";
|
||||
import { toolAccessService } from "./tool-access.js";
|
||||
import { logActivity } from "./activity-log.js";
|
||||
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
|
||||
import { notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js";
|
||||
import { githubBotRequest } from "./chat-github-client.js";
|
||||
import { environmentService } from "./environments.js";
|
||||
import { instanceSettingsService } from "./instance-settings.js";
|
||||
@@ -165,7 +165,7 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) {
|
||||
const allowUnlinkedPeople = config.people.some((p) => p.kind === "guest");
|
||||
if ((!currentEndpoint.allowUnlinkedPeople && allowUnlinkedPeople) ||
|
||||
(allowUnlinkedPeople && currentEndpoint.sponsorUserId !== config.responsibleUserId)) {
|
||||
await notifyChatRunMilestones(tx, { companyId: bot.companyId });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId: bot.companyId });
|
||||
}
|
||||
const values = {
|
||||
companyId: bot.companyId,
|
||||
@@ -335,7 +335,7 @@ export function githubChatManagementService(db: Db, fetchImpl = fetch) {
|
||||
updatedAt: new Date(),
|
||||
};
|
||||
if (prior?.status !== "linked") {
|
||||
await notifyChatRunMilestones(tx, { companyId: bot.companyId });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId: bot.companyId });
|
||||
}
|
||||
await tx
|
||||
.insert(chatIdentityLinks)
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
import { nativeResponsePresentationWork } from "./native-runtime/native-response-presentation-work.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
|
||||
import { randomBytes, randomUUID } from "node:crypto";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
agents, agentWakeupRequests, authUsers, chatConversations, chatEndpoints, chatMessageLinks,
|
||||
@@ -16,14 +18,18 @@ import { createHeartbeatQueueTestDependencies } from "../__tests__/helpers/heart
|
||||
import { createHeartbeatQueue } from "./heartbeat/queue.js";
|
||||
import { instanceSettingsService } from "./instance-settings.js";
|
||||
|
||||
const support = await getEmbeddedPostgresTestSupport();
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_MONITOR_CONTINUATION_TEST_DATABASE_URL");
|
||||
const support = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
|
||||
if (!support.supported) console.warn(`Monitor publication database unavailable: ${support.reason}`);
|
||||
(support.supported ? describe : describe.skip)("monitor conversation continuation", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("paperclip-monitor-publications-");
|
||||
db = createDb(temporary.connectionString);
|
||||
if (external) db = createDb(external);
|
||||
else {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("paperclip-monitor-publications-");
|
||||
db = createDb(temporary.connectionString);
|
||||
}
|
||||
await db.insert(authUsers).values({ id: "requester", name: "Requester", email: "requester@example.test", createdAt: new Date(), updatedAt: new Date() });
|
||||
});
|
||||
afterAll(async () => { await temporary?.cleanup(); });
|
||||
@@ -235,7 +241,7 @@ if (!support.supported) console.warn(`Monitor publication database unavailable:
|
||||
expect(await f.bindings()).toEqual([]);
|
||||
});
|
||||
|
||||
it("recovers the accepted continuation explanation after interrupted finalization without another provider result", async () => {
|
||||
it.each([false, true])("recovers the accepted continuation explanation without another provider result (missing presentation: %s)", async missingPresentation => {
|
||||
const f = await fixture();
|
||||
const executionPayload = randomBytes(128 * 1024).toString("base64");
|
||||
await db.update(heartbeatRuns).set({
|
||||
@@ -281,6 +287,20 @@ if (!support.supported) console.warn(`Monitor publication database unavailable:
|
||||
expect(settledRun.stdoutExcerpt).toBe(executionPayload);
|
||||
const priorComments = await db.select({ body: issueComments.body }).from(issueComments).where(eq(issueComments.createdByRunId, f.source.id));
|
||||
expect(priorComments).toEqual([{ body: summary }]);
|
||||
expect(await nativeResponsePresentationWork(db).load({ companyId: f.company.id, runId: f.source.id })).toEqual([]);
|
||||
if (missingPresentation) {
|
||||
await db.transaction(async tx => {
|
||||
await tx.delete(chatPublications).where(eq(chatPublications.issueId, f.issue.id));
|
||||
await tx.delete(issueComments).where(eq(issueComments.createdByRunId, f.source.id));
|
||||
await tx.update(heartbeatRuns).set({ resultJson: sql`${heartbeatRuns.resultJson} - 'presentationDecision'` })
|
||||
.where(eq(heartbeatRuns.id, f.source.id));
|
||||
});
|
||||
const work = nativeResponsePresentationWork(db);
|
||||
const entries = await work.load({ companyId: f.company.id, runId: f.source.id });
|
||||
expect(entries).toHaveLength(1);
|
||||
await expect(work.check(entries[0])).resolves.toBeNull();
|
||||
expect(await work.load({ companyId: f.company.id, runId: f.source.id })).toEqual([]);
|
||||
}
|
||||
expect(await repair()).toBe(false);
|
||||
expect(await repair()).toBe(false);
|
||||
const comments = await db.select().from(issueComments).where(eq(issueComments.createdByRunId, f.source.id));
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
|
||||
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
|
||||
import { notifyChatConfirmationWork } from "./chat-confirmation-notifications.js";
|
||||
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
|
||||
@@ -6,9 +7,13 @@ import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notificatio
|
||||
|
||||
/** Lineage changes can authorize descendants, so invalidate the issue rather
|
||||
* than just the run whose row changed. Configuration may invalidate a company. */
|
||||
export type ChatRunMilestoneScope = { companyId: string; issueId?: string };
|
||||
export type ChatRunMilestoneScope = { companyId: string; issueId?: string; runId?: string };
|
||||
|
||||
export async function notifyChatRunMilestones(tx: object, scope: ChatRunMilestoneScope) {
|
||||
if (scope.issueId || scope.runId) {
|
||||
await notifyNativeResponsePresentation(tx, { companyId: scope.companyId, issueId: scope.issueId,
|
||||
runId: scope.issueId ? undefined : scope.runId });
|
||||
}
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubReviewChecks, JSON.stringify({ companyId: scope.companyId }));
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatGitHubTerminalResponses, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }));
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.chatRunMilestones, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId }));
|
||||
@@ -21,3 +26,8 @@ export async function notifyChatRunMilestones(tx: object, scope: ChatRunMileston
|
||||
// Native presentation and run ownership can unblock a parked acknowledgement.
|
||||
await notifyChatResponseAvailability(tx, scope);
|
||||
}
|
||||
|
||||
export async function notifyChatCompanyAuthorityChange(tx: object, scope: { companyId: string }) {
|
||||
await notifyChatRunMilestones(tx, scope);
|
||||
await notifyNativeResponsePresentation(tx, scope);
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
import { notifyChatResponseAvailability } from "./chat-response-availability.js";
|
||||
import { notifyChatRunMilestones } from "./chat-run-milestone-notifications.js";
|
||||
import { notifyChatRunMilestones, notifyChatCompanyAuthorityChange } from "./chat-run-milestone-notifications.js";
|
||||
import { notifyChatConnectionConfiguration } from "./chat-channels/configuration-work-notifications.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
@@ -47,7 +47,7 @@ export async function notifyChatVerificationWork(tx: object): Promise<void> {
|
||||
export async function notifyChatEndpointWork(tx: object, connectionId: string, companyId: string): Promise<void> {
|
||||
// Configuration/status transitions can restore post-control publication
|
||||
// authority. Ordinary health checks use the narrower configuration signals.
|
||||
await notifyChatRunMilestones(tx, { companyId });
|
||||
await notifyChatCompanyAuthorityChange(tx, { companyId });
|
||||
await notifyChatDeliveryWork(tx);
|
||||
await notifyChatPublicationWork(tx);
|
||||
for (const queue of Object.values(CHAT_ACTION_QUEUES)) await notifyDeliveryWork(tx, queue);
|
||||
|
||||
@@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
|
||||
/** Topics for existing durable queues; register intent before their writes. */
|
||||
export const DELIVERY_QUEUES = {
|
||||
nativeResponsePresentation: "native-response-presentation",
|
||||
nativePendingEffects: "native-pending-effects",
|
||||
unsafeWorkspaceExportRecovery: "unsafe-workspace-export-recovery",
|
||||
automaticCompletionReviews: "automatic-completion-reviews",
|
||||
|
||||
@@ -21,6 +21,21 @@ const committedChatResponseAuthorities = new WeakMap<
|
||||
CommittedChatResponseAuthority
|
||||
>();
|
||||
|
||||
const committedChatResponseAuthorityListeners = new WeakMap<Db, Set<() => void>>();
|
||||
|
||||
export function onCommittedChatResponseAuthorityRegistered(db: Db, listener: () => void): () => void {
|
||||
let listeners = committedChatResponseAuthorityListeners.get(db);
|
||||
if (!listeners) {
|
||||
listeners = new Set();
|
||||
committedChatResponseAuthorityListeners.set(db, listeners);
|
||||
}
|
||||
listeners.add(listener);
|
||||
return () => {
|
||||
listeners.delete(listener);
|
||||
if (listeners.size === 0 && committedChatResponseAuthorityListeners.get(db) === listeners) committedChatResponseAuthorityListeners.delete(db);
|
||||
};
|
||||
}
|
||||
|
||||
export class CommittedChatResponseAuthorizationError extends HttpError {
|
||||
constructor() {
|
||||
super(
|
||||
@@ -41,6 +56,7 @@ export function registerCommittedChatResponseAuthority(
|
||||
authority: CommittedChatResponseAuthority,
|
||||
): () => void {
|
||||
committedChatResponseAuthorities.set(db, authority);
|
||||
for (const listener of committedChatResponseAuthorityListeners.get(db) ?? []) listener();
|
||||
return () => {
|
||||
if (committedChatResponseAuthorities.get(db) === authority) {
|
||||
committedChatResponseAuthorities.delete(db);
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyNativePendingEffects } from "./native-pending-effect-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "./native-finalization-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRunChange } from "./unsafe-workspace-export-notifications.js";
|
||||
import { notifyCancelledWorkspacePreservation } from "./cancelled-workspace-preservation-notifications.js";
|
||||
import { notifyInstructionCollection } from "./instruction-collection-notifications.js";
|
||||
@@ -147,7 +147,7 @@ export async function reconcileAbandonedExecutionControl(
|
||||
}
|
||||
// A persisted result belongs to the existing finalizer, not a replacement provider.
|
||||
if (coordinator?.resultId) {
|
||||
await notifyNativePendingEffects(tx, { companyId: run.companyId, runId: run.id });
|
||||
await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id });
|
||||
await tx
|
||||
.update(nativeRunFinalizations)
|
||||
.set({
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { isNativeResponseEvidence, notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
|
||||
import { isSafeNativeChatProgressEventType } from "./safe-native-chat-progress.js";
|
||||
import { notifyChatNativeProgress } from "./chat-native-progress-notifications.js";
|
||||
import { boundRunIssueIdSql } from "./run-task-binding.js";
|
||||
@@ -72,7 +73,8 @@ export async function appendHeartbeatRunEvent(
|
||||
input: AppendHeartbeatRunEventInput,
|
||||
): Promise<AppendHeartbeatRunEventResult> {
|
||||
const projectsProgress = isSafeNativeChatProgressEventType(input.eventType);
|
||||
if (!input.nativeSource && !input.retryExhaustion && !projectsProgress) {
|
||||
const projectsResponse = isNativeResponseEvidence(input.eventType, input.payload?.prpEvent);
|
||||
if (!input.nativeSource && !input.retryExhaustion && !projectsProgress && !projectsResponse) {
|
||||
const allocated = db.$with("allocated_run_event", {}).as(sql`
|
||||
update ${heartbeatRuns}
|
||||
set next_event_seq = ${heartbeatRuns.nextEventSeq} + 1,
|
||||
@@ -101,7 +103,8 @@ export async function appendHeartbeatRunEvent(
|
||||
.select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId,
|
||||
// Only progress needs the task binding. Other native/recovery events
|
||||
// must not read a potentially large context snapshot to append a log.
|
||||
...(projectsProgress ? { runtimeMode: heartbeatRuns.runtimeMode, issueId: boundRunIssueIdSql() } : {}) })
|
||||
...(projectsProgress || projectsResponse ? { runtimeMode: heartbeatRuns.runtimeMode } : {}),
|
||||
...(projectsProgress ? { issueId: boundRunIssueIdSql() } : {}) })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, input.runId))
|
||||
.for("update")
|
||||
@@ -175,6 +178,9 @@ export async function appendHeartbeatRunEvent(
|
||||
if (projectsProgress && run.runtimeMode === "native" && run.issueId) {
|
||||
await notifyChatNativeProgress(tx, { companyId: run.companyId, issueId: run.issueId });
|
||||
}
|
||||
if (projectsResponse && run.runtimeMode === "native") {
|
||||
await notifyNativeResponsePresentation(tx, { companyId: run.companyId, runId: input.runId });
|
||||
}
|
||||
const seq = await allocateHeartbeatRunEventSeq(
|
||||
tx as unknown as Db,
|
||||
input.runId,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { registerNativeResponsePresentationWork } from "./native-runtime/native-response-presentation-work.js";
|
||||
import { registerNativePendingEffectWork } from "./native-runtime/native-pending-effect-work.js";
|
||||
import { registerUnsafeWorkspaceExportWork } from "./native-runtime/native-workspace-export-recovery.js";
|
||||
import { registerAutomaticCompletionReviewWork } from "./native-runtime/automatic-completion-reviews.js";
|
||||
@@ -4089,6 +4090,8 @@ export function heartbeatService(
|
||||
triggerIssueMonitor,
|
||||
|
||||
reportRunActivity: clearDetachedRunWarning,
|
||||
registerNativeResponsePresentationWork: (coordinator: Parameters<typeof registerNativeResponsePresentationWork>[1], canRun: () => boolean) =>
|
||||
registerNativeResponsePresentationWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null),
|
||||
registerNativePendingEffectWork: (coordinator: Parameters<typeof registerNativePendingEffectWork>[1], canRun: () => boolean) =>
|
||||
registerNativePendingEffectWork(db, coordinator, () => canRun() && readTaskDrain(new Date()) === null),
|
||||
registerUnsafeWorkspaceExportWork: (coordinator: Parameters<typeof registerUnsafeWorkspaceExportWork>[1], canRun: () => boolean) =>
|
||||
|
||||
@@ -976,7 +976,8 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
.where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.status, "queued")))
|
||||
.for("update");
|
||||
if (!currentRun) return null;
|
||||
await notifyChatRunMilestones(tx, { companyId: run.companyId });
|
||||
await notifyChatRunMilestones(tx, { companyId: run.companyId, runId: run.id,
|
||||
issueId: run.nativeIssueId ?? run.issueId ?? readNonEmptyString(parseObject(run.contextSnapshot).issueId) ?? undefined });
|
||||
await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
||||
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
|
||||
await notifyUnsafeWorkspaceExportRunChange(tx, run);
|
||||
@@ -1625,7 +1626,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
)).for("update");
|
||||
if (!lockedRun || lockedRun.status !== "queued") return null;
|
||||
await withExternalAdmissionGuard(claimTx, run.companyId, run.agentId, () => assertNoExternalOverlap(claimTx, run.companyId, run.agentId));
|
||||
await notifyChatRunMilestones(claimTx, { companyId: run.companyId, issueId: issueId ?? undefined });
|
||||
await notifyChatRunMilestones(claimTx, { companyId: run.companyId, runId: run.id, issueId: issueId ?? undefined });
|
||||
const claimedRun = await claimTx.update(heartbeatRuns).set(claimValues).where(and(
|
||||
eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.status, "queued"),
|
||||
)).returning().then((rows) => rows[0] ?? null);
|
||||
@@ -1726,7 +1727,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende
|
||||
const [candidate] = await tx.select({ companyId: heartbeatRuns.companyId, contextSnapshot: heartbeatRuns.contextSnapshot })
|
||||
.from(heartbeatRuns).where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running"))).for("update");
|
||||
if (!candidate) return;
|
||||
await notifyChatRunMilestones(tx, { companyId: candidate.companyId,
|
||||
await notifyChatRunMilestones(tx, { companyId: candidate.companyId, runId,
|
||||
issueId: readNonEmptyString(parseObject(candidate.contextSnapshot).issueId) ?? undefined });
|
||||
const released = await tx
|
||||
.update(heartbeatRuns)
|
||||
|
||||
@@ -1947,7 +1947,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
|
||||
let settledProcessesDiscovered = false;
|
||||
|
||||
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean }) {
|
||||
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean }) {
|
||||
if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db);
|
||||
const staleThresholdMs = opts?.staleThresholdMs ?? 0;
|
||||
const now = new Date();
|
||||
@@ -1969,6 +1969,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
skipAutomaticCompletionReviews: opts?.skipAutomaticCompletionReviews,
|
||||
skipUnsafeWorkspaceExports: opts?.skipUnsafeWorkspaceExports,
|
||||
skipNativePendingEffects: opts?.skipNativePendingEffects,
|
||||
skipNativeResponsePresentation: opts?.skipNativeResponsePresentation,
|
||||
environmentRuntime,
|
||||
pluginWorkerManager: dependencies.pluginWorkerManager,
|
||||
onWorkspaceSettled: settleRecoveredNativeWorkspace,
|
||||
|
||||
@@ -351,7 +351,7 @@ export async function recordRunPresentationDecision(db: Db, input: {
|
||||
companyId: string; runId: string; issueId?: string; decision: RunPresentationDecision;
|
||||
}) {
|
||||
await db.transaction(async tx => {
|
||||
await notifyChatRunMilestones(tx, { companyId: input.companyId, issueId: input.issueId });
|
||||
await notifyChatRunMilestones(tx, { companyId: input.companyId, runId: input.runId, issueId: input.issueId });
|
||||
await tx.update(heartbeatRuns).set({
|
||||
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) || ${JSON.stringify({ presentationDecision: input.decision })}::jsonb`,
|
||||
updatedAt: new Date(),
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
||||
import { randomBytes } from "node:crypto";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { agents, agentWakeupRequests, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRunEvents, heartbeatRuns } from "@paperclipai/db";
|
||||
import { agents, agentWakeupRequests, companies, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRunEvents, heartbeatRuns, issues, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
@@ -241,6 +241,23 @@ describe.skipIf(!support.supported)("heartbeat lifecycle database wiring", () =>
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
|
||||
it.each(["native", "legacy", "unbound"] as const)("keeps %s run-status response recovery scoped to its task or run", async binding => {
|
||||
const { run } = await fixture();
|
||||
const [issue] = await db.insert(issues).values({ companyId: run.companyId, title: "Scope owner" }).returning();
|
||||
await db.update(heartbeatRuns).set(binding === "native" ? { nativeIssueId: issue.id }
|
||||
: binding === "legacy" ? { contextSnapshot: { issueId: issue.id } } : { contextSnapshot: {} }).where(eq(heartbeatRuns.id, run.id));
|
||||
const hints: unknown[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeResponsePresentation, (event, scopes) => {
|
||||
if (event === "settled") for (const hint of scopes ?? []) hints.push(JSON.parse(hint));
|
||||
});
|
||||
try {
|
||||
await createHeartbeatLifecycle(db, callbacks(db)).setRunStatusIfRunning(run.id, "succeeded", { finishedAt: new Date() });
|
||||
expect(hints).toEqual([binding === "unbound" ? { companyId: run.companyId, runId: run.id }
|
||||
: { companyId: run.companyId, issueId: issue.id }]);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("succeeded");
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
|
||||
it("does not request accounting for an unchanged running status", async () => {
|
||||
const { run } = await fixture();
|
||||
const settled = vi.fn();
|
||||
|
||||
@@ -1017,7 +1017,7 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
|
||||
return recordHeartbeatRunRuntimeProgress(currentRun, update, issueId);
|
||||
}
|
||||
|
||||
async function writeRunStatus<T>(status: string, previous: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "status" | "nativeIssueId" | "contextSnapshot"> | null,
|
||||
async function writeRunStatus<T>(status: string, previous: Pick<typeof heartbeatRuns.$inferSelect, "id" | "companyId" | "status" | "nativeIssueId" | "issueId" | "contextSnapshot"> | null,
|
||||
write: (tx: Db) => Promise<T>) {
|
||||
const companyId = previous?.companyId;
|
||||
// Register on the outer transaction before the terminal write. An unknown
|
||||
@@ -1031,7 +1031,8 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
|
||||
if (previous) await notifyCancelledWorkspacePreservation(tx, { companyId: previous.companyId, runId: previous.id });
|
||||
if (isHeartbeatRunTerminalStatus(status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, companyId);
|
||||
await notifyExecutionRunWork(tx);
|
||||
if (companyId) await notifyChatRunMilestones(tx, { companyId });
|
||||
if (previous) await notifyChatRunMilestones(tx, { companyId: previous.companyId, runId: previous.id,
|
||||
issueId: previous.nativeIssueId ?? previous.issueId ?? readNonEmptyString(parseObject(previous.contextSnapshot).issueId) ?? undefined });
|
||||
return write(tx);
|
||||
});
|
||||
}
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export type NativeFinalizationWorkScope = { companyId: string; runId?: string; issueId?: string };
|
||||
|
||||
export async function notifyNativeFinalizationWork(transaction: object, scope: NativeFinalizationWorkScope) {
|
||||
if (scope.issueId || scope.runId) await notifyNativeResponsePresentation(transaction, scope);
|
||||
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativePendingEffects,
|
||||
JSON.stringify({ companyId: scope.companyId, runId: scope.runId }));
|
||||
}
|
||||
@@ -1,8 +0,0 @@
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export type NativePendingEffectScope = { companyId: string; runId?: string };
|
||||
|
||||
export async function notifyNativePendingEffects(transaction: object, scope: NativePendingEffectScope) {
|
||||
await notifyDeliveryWork(transaction, DELIVERY_QUEUES.nativePendingEffects,
|
||||
JSON.stringify({ companyId: scope.companyId, runId: scope.runId }));
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
import { readCompletedAssistantMessageCandidate } from "./heartbeat-run-summary.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export type NativeResponsePresentationScope = { companyId: string; issueId?: string; runId?: string; agentId?: string };
|
||||
|
||||
export async function notifyNativeResponsePresentation(tx: object, scope: NativeResponsePresentationScope) {
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeResponsePresentation,
|
||||
JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId, agentId: scope.agentId }));
|
||||
}
|
||||
|
||||
export function isNativeResponseEvidence(eventType: string, prpEvent: unknown): boolean {
|
||||
return eventType === "run.result.accepted" || eventType === "item.completed" &&
|
||||
readCompletedAssistantMessageCandidate({ seq: 0, prpEvent })?.channel === "final";
|
||||
}
|
||||
@@ -644,6 +644,7 @@ type NativeReconciliationOptions = {
|
||||
skipAutomaticCompletionReviews?: boolean;
|
||||
skipUnsafeWorkspaceExports?: boolean;
|
||||
skipNativePendingEffects?: boolean;
|
||||
skipNativeResponsePresentation?: boolean;
|
||||
pluginWorkerManager?: WorkspaceWorkers;
|
||||
environmentRuntime?: EnvironmentRuntimeService;
|
||||
onWorkspaceSettled?: (input: {
|
||||
@@ -861,7 +862,7 @@ async function reconcileNativeFinalizationsPass(
|
||||
)).limit(1).then((entries) => entries[0] ?? null)
|
||||
: null;
|
||||
const currentDecisionJson = record(currentDecision?.decisionJson);
|
||||
if (row.coordinatorPhase === "committed") {
|
||||
if (row.coordinatorPhase === "committed" && !options.skipNativeResponsePresentation) {
|
||||
// A later decision can supersede this run's task status, but cannot
|
||||
// erase its accepted, still-authorized response. Repair presentation
|
||||
// before the status-only early return below, without rerunning work.
|
||||
@@ -1123,6 +1124,7 @@ async function reconcileNativeFinalizationsPass(
|
||||
runId: row.runId,
|
||||
workspaceFinalizeStatus,
|
||||
projectRunStatus: true,
|
||||
skipNativeResponsePresentation: options.skipNativeResponsePresentation,
|
||||
preserveProviderAttempt: Boolean(
|
||||
readNativeWorkspaceSyncReference(
|
||||
row.nativeWorkspaceSync,
|
||||
@@ -1181,6 +1183,7 @@ async function reconcileNativeFinalizationsPass(
|
||||
runId: row.runId,
|
||||
workspaceFinalizeStatus: barrier.status as "succeeded" | "failed",
|
||||
projectRunStatus: true,
|
||||
skipNativeResponsePresentation: options.skipNativeResponsePresentation,
|
||||
preserveProviderAttempt: Boolean(
|
||||
readNativeWorkspaceSyncReference(
|
||||
row.nativeWorkspaceSync,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import type { NativeInteractionResponseEnvelope } from "../../vendor/paperclip-runner/index.js";
|
||||
@@ -633,7 +633,7 @@ export async function routePersistedNativeResultAttention(input: {
|
||||
const delegation = effects.find((effect) => effect.kind === "create_delegated_issue");
|
||||
const finalizationError = effects.find((effect) => effect.kind === "record_finalization_error");
|
||||
await input.db.transaction(async tx => {
|
||||
await notifyNativePendingEffects(tx, { companyId: binding.companyId, runId: input.runId });
|
||||
await notifyNativeFinalizationWork(tx, { companyId: binding.companyId, runId: input.runId });
|
||||
await tx.update(nativeRunFinalizations).set({
|
||||
phase: finalizationError ? "retryable_failure" : "committed",
|
||||
assessmentId: existingAssessment.id,
|
||||
|
||||
@@ -2,7 +2,7 @@ import { and, asc, eq, exists, gt, inArray, isNotNull, ne } from "drizzle-orm";
|
||||
import { heartbeatRuns, issues, nativeRunFinalizations, statusDecisionEffects, type Db } from "@paperclipai/db";
|
||||
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
|
||||
import type { NativePendingEffectScope } from "../native-pending-effect-notifications.js";
|
||||
import type { NativeFinalizationWorkScope } from "../native-finalization-notifications.js";
|
||||
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
|
||||
import { dispatchPendingNativeStatusEffects, nativePendingEffectDueAt } from "./status-decision-committer.js";
|
||||
|
||||
@@ -15,7 +15,7 @@ export function pendingNativeStatusEffectsCondition(db: Db) {
|
||||
)));
|
||||
}
|
||||
|
||||
function findPendingEffects(db: Db, scope?: NativePendingEffectScope, after?: string) {
|
||||
function findPendingEffects(db: Db, scope?: NativeFinalizationWorkScope, after?: string) {
|
||||
return db.select({
|
||||
id: nativeRunFinalizations.runId, companyId: nativeRunFinalizations.companyId,
|
||||
issueId: nativeRunFinalizations.issueId, decisionId: nativeRunFinalizations.decisionId,
|
||||
@@ -36,7 +36,7 @@ function findPendingEffects(db: Db, scope?: NativePendingEffectScope, after?: st
|
||||
|
||||
export function registerNativePendingEffectWork(db: Db, coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
|
||||
type Entry = { id: string; companyId: string; dueAt: number };
|
||||
async function load(scope?: NativePendingEffectScope, signal?: AbortSignal): Promise<Entry[]> {
|
||||
async function load(scope?: NativeFinalizationWorkScope, signal?: AbortSignal): Promise<Entry[]> {
|
||||
const entries: Entry[] = [];
|
||||
let after: string | undefined;
|
||||
while (canRun() && !signal?.aborted) {
|
||||
@@ -49,8 +49,8 @@ export function registerNativePendingEffectWork(db: Db, coordinator: ReturnType<
|
||||
}
|
||||
return registerScopedDeadlineWork(coordinator, {
|
||||
queue: DELIVERY_QUEUES.nativePendingEffects, batchSize: 25,
|
||||
decodeScope: (hint: string): NativePendingEffectScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
|
||||
matches: (entry: Entry, scope?: NativePendingEffectScope) => !scope || entry.companyId === scope.companyId && (!scope.runId || entry.id === scope.runId),
|
||||
decodeScope: (hint: string): NativeFinalizationWorkScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
|
||||
matches: (entry: Entry, scope?: NativeFinalizationWorkScope) => !scope || entry.companyId === scope.companyId && (!scope.runId || entry.id === scope.runId),
|
||||
load,
|
||||
check: async (entry: Entry, signal?: AbortSignal) => {
|
||||
if (!canRun() || signal?.aborted) throw new Error("Native pending effect recovery stopped");
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
import { and, asc, eq, gt, inArray, isNotNull, isNull, notExists, or, sql } from "drizzle-orm";
|
||||
import { heartbeatRuns, issues, nativeRunFinalizations, nativeRunResults, statusDecisions, statusDecisionEffects, type Db } from "@paperclipai/db";
|
||||
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
|
||||
import type { NativeResponsePresentationScope } from "../native-response-presentation-notifications.js";
|
||||
import { onCommittedChatResponseAuthorityRegistered } from "../durable-chat-wakeup.js";
|
||||
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
|
||||
import { recoverCommittedNativeChatResponse } from "./native-run-finalizer.js";
|
||||
|
||||
export function registerNativeResponsePresentationWork(db: Db,
|
||||
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
|
||||
return nativeResponsePresentationWork(db).register(coordinator, canRun);
|
||||
}
|
||||
|
||||
type Entry = { id: string; companyId: string; issueId: string; agentId: string; dueAt: number };
|
||||
|
||||
export function nativeResponsePresentationWork(db: Db) {
|
||||
function candidates(scope?: NativeResponsePresentationScope, after?: string, runId?: string) {
|
||||
return db.select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId,
|
||||
issueId: nativeRunFinalizations.issueId, agentId: heartbeatRuns.agentId, nextAttemptAt: nativeRunFinalizations.nextAttemptAt,
|
||||
}).from(nativeRunFinalizations)
|
||||
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, nativeRunFinalizations.runId),
|
||||
eq(heartbeatRuns.companyId, nativeRunFinalizations.companyId), eq(heartbeatRuns.nativeIssueId, nativeRunFinalizations.issueId)))
|
||||
.innerJoin(issues, and(eq(issues.id, nativeRunFinalizations.issueId), eq(issues.companyId, nativeRunFinalizations.companyId)))
|
||||
.innerJoin(nativeRunResults, and(eq(nativeRunResults.id, nativeRunFinalizations.resultId),
|
||||
eq(nativeRunResults.companyId, nativeRunFinalizations.companyId), eq(nativeRunResults.issueId, nativeRunFinalizations.issueId),
|
||||
eq(nativeRunResults.runId, heartbeatRuns.id), eq(nativeRunResults.completionContractId, heartbeatRuns.completionContractId)))
|
||||
.innerJoin(statusDecisions, and(eq(statusDecisions.id, nativeRunFinalizations.decisionId),
|
||||
eq(statusDecisions.companyId, nativeRunFinalizations.companyId), eq(statusDecisions.issueId, nativeRunFinalizations.issueId),
|
||||
eq(statusDecisions.runId, heartbeatRuns.id), eq(statusDecisions.assessmentId, nativeRunFinalizations.assessmentId)))
|
||||
.where(and(
|
||||
eq(heartbeatRuns.runtimeMode, "native"), inArray(heartbeatRuns.status, ["succeeded", "failed", "timed_out"]),
|
||||
isNotNull(heartbeatRuns.finishedAt), eq(nativeRunFinalizations.phase, "committed"), isNull(nativeRunFinalizations.leaseOwner),
|
||||
eq(nativeRunResults.schemaStatus, "accepted"),
|
||||
sql`jsonb_typeof(${heartbeatRuns.resultJson} #> '{presentationDecision,commentId}') is distinct from 'string'`,
|
||||
sql`coalesce(${heartbeatRuns.resultJson} #> '{presentationDecision,commentAction}', '"create"'::jsonb) = '"create"'::jsonb`,
|
||||
sql`${heartbeatRuns.contextSnapshot}->'skipIssueComment' is distinct from 'true'::jsonb`,
|
||||
sql`coalesce(${statusDecisions.decisionJson}->'externalChatReviewPresentation', 'null'::jsonb) in ('null'::jsonb, 'false'::jsonb, '0'::jsonb, '""'::jsonb)`,
|
||||
sql`${nativeRunResults.resultJson} #>> '{result,schema}' = 'paperclip.run_result.v1'`,
|
||||
sql`${nativeRunResults.resultJson} #>> '{terminal,runTerminalState}' = 'succeeded'`,
|
||||
sql`${nativeRunResults.resultJson} #>> '{terminal,turnTerminalState}' = 'completed'`,
|
||||
or(
|
||||
and(eq(heartbeatRuns.status, "succeeded"),
|
||||
sql`(coalesce(${issues.conversationAgentId}::text, '') = '' or coalesce(${issues.conversationUserId}, '') = '')`,
|
||||
sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'done'`,
|
||||
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'done'`),
|
||||
and(sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'yielded'`,
|
||||
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'yielded'`,
|
||||
sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,summary}') = 'string'
|
||||
and ${nativeRunResults.resultJson} #>> '{result,summary}' ~ '[^[:space:]]'`,
|
||||
sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,continuation,idempotencyKey}') = 'string'
|
||||
and ${nativeRunResults.resultJson} #>> '{result,continuation,idempotencyKey}' ~ '[^[:space:]]'`,
|
||||
or(
|
||||
and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'response_wake'`,
|
||||
sql`${nativeRunResults.resultJson} #> '{result,attentionRequests}' = '[]'::jsonb`,
|
||||
sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,continuation,summary}') = 'string'
|
||||
and ${nativeRunResults.resultJson} #>> '{result,continuation,summary}' ~ '[^[:space:]]'`),
|
||||
and(eq(heartbeatRuns.status, "succeeded"), or(
|
||||
and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'same_agent'`,
|
||||
eq(statusDecisions.reasonCode, "live_continuation_registered")),
|
||||
and(sql`${nativeRunResults.resultJson} #>> '{result,continuation,kind}' = 'monitor'`,
|
||||
inArray(statusDecisions.reasonCode, ["scheduled_monitor_waiting", "scheduled_monitor_dispatched", "scheduled_monitor_preserved"])),
|
||||
)),
|
||||
)),
|
||||
),
|
||||
notExists(db.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and(
|
||||
eq(statusDecisionEffects.companyId, nativeRunFinalizations.companyId),
|
||||
eq(statusDecisionEffects.issueId, nativeRunFinalizations.issueId),
|
||||
eq(statusDecisionEffects.decisionId, nativeRunFinalizations.decisionId),
|
||||
inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]),
|
||||
))),
|
||||
scope ? and(eq(heartbeatRuns.companyId, scope.companyId), scope.issueId ? eq(nativeRunFinalizations.issueId, scope.issueId) : undefined) : undefined,
|
||||
scope?.runId ? eq(heartbeatRuns.id, scope.runId) : undefined,
|
||||
scope?.agentId ? eq(heartbeatRuns.agentId, scope.agentId) : undefined,
|
||||
after ? gt(heartbeatRuns.id, after) : undefined, runId ? eq(heartbeatRuns.id, runId) : undefined,
|
||||
)).orderBy(asc(heartbeatRuns.id)).limit(100);
|
||||
}
|
||||
async function load(scope?: NativeResponsePresentationScope, signal?: AbortSignal, canRun = () => true): Promise<Entry[]> {
|
||||
const entries: Entry[] = [];
|
||||
let after: string | undefined;
|
||||
while (canRun() && !signal?.aborted) {
|
||||
const rows = await candidates(scope, after);
|
||||
for (const row of rows) entries.push({ id: row.id, companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, dueAt: row.nextAttemptAt?.getTime() ?? 0 });
|
||||
if (rows.length < 100) return entries;
|
||||
after = rows[rows.length - 1].id;
|
||||
}
|
||||
throw new Error("Native response presentation discovery stopped");
|
||||
}
|
||||
async function check(entry: Entry, signal?: AbortSignal): Promise<Entry | null> {
|
||||
if (signal?.aborted) throw new Error("Native response presentation recovery stopped");
|
||||
const scope = { companyId: entry.companyId, issueId: entry.issueId };
|
||||
const [current] = await candidates(scope, undefined, entry.id);
|
||||
if (!current) return null;
|
||||
if (current.nextAttemptAt && current.nextAttemptAt.getTime() > Date.now()) return { ...entry, dueAt: current.nextAttemptAt.getTime() };
|
||||
if (signal?.aborted) throw new Error("Native response presentation recovery stopped");
|
||||
const recoveryStartedAt = Date.now();
|
||||
await recoverCommittedNativeChatResponse(db, { ...scope, runId: entry.id, onlyWhenDue: true });
|
||||
const [remaining] = await candidates(scope, undefined, entry.id);
|
||||
return remaining?.nextAttemptAt && remaining.nextAttemptAt.getTime() > recoveryStartedAt
|
||||
? { ...entry, dueAt: remaining.nextAttemptAt.getTime() } : null;
|
||||
}
|
||||
function register(coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
|
||||
const worker = registerScopedDeadlineWork(coordinator, {
|
||||
queue: DELIVERY_QUEUES.nativeResponsePresentation,
|
||||
decodeScope: (hint: string): NativeResponsePresentationScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
|
||||
matches: (entry: Entry, scope?: NativeResponsePresentationScope) => !scope || entry.companyId === scope.companyId && (!scope.issueId || entry.issueId === scope.issueId) && (!scope.runId || entry.id === scope.runId) && (!scope.agentId || entry.agentId === scope.agentId),
|
||||
load: (scope, signal) => load(scope, signal, canRun),
|
||||
check: (entry, signal) => {
|
||||
if (!canRun()) throw new Error("Native response presentation recovery paused");
|
||||
return check(entry, signal);
|
||||
},
|
||||
}, canRun);
|
||||
const unsubscribe = onCommittedChatResponseAuthorityRegistered(db, () => worker.invalidate());
|
||||
return { ...worker, stop: unsubscribe };
|
||||
}
|
||||
return { load, check, register };
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
|
||||
import { isNativeResponseEvidence, notifyNativeResponsePresentation } from "../native-response-presentation-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
|
||||
import { isSafeNativeChatProgressEventType } from "../safe-native-chat-progress.js";
|
||||
import { notifyChatNativeProgress } from "../chat-native-progress-notifications.js";
|
||||
@@ -297,6 +298,9 @@ export class NativeRunCoordinatorStore {
|
||||
if (isSafeNativeChatProgressEventType(event.eventType)) {
|
||||
await notifyChatNativeProgress(tx, { companyId: this.#binding.companyId, issueId: this.#binding.issueId });
|
||||
}
|
||||
if (isNativeResponseEvidence(event.eventType, canonicalPayload)) {
|
||||
await notifyNativeResponsePresentation(tx, this.#binding);
|
||||
}
|
||||
const cursor = run.nextEventSeq;
|
||||
const [inserted] = await tx
|
||||
.insert(heartbeatRunEvents)
|
||||
@@ -523,7 +527,7 @@ export class NativeRunCoordinatorStore {
|
||||
) {
|
||||
throw new Error("native_finalization_lease_conflict");
|
||||
}
|
||||
if (row.decisionId) await notifyNativePendingEffects(tx, this.#binding);
|
||||
if (row.decisionId) await notifyNativeFinalizationWork(tx, this.#binding);
|
||||
if (
|
||||
row.leaseOwner === input.leaseOwner &&
|
||||
row.leaseExpiresAt &&
|
||||
@@ -566,7 +570,7 @@ export class NativeRunCoordinatorStore {
|
||||
throw new Error("native_finalization_retry_invalid");
|
||||
}
|
||||
const [updated] = await this.#db.transaction(async tx => {
|
||||
await notifyNativePendingEffects(tx, this.#binding);
|
||||
await notifyNativeFinalizationWork(tx, this.#binding);
|
||||
if (input.failureCode === "native_workspace_sync_out_unsafe_archive") {
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, this.#binding);
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
|
||||
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
||||
@@ -39,6 +39,7 @@ import {
|
||||
nativeRunFinalizations,
|
||||
nativeRunResults,
|
||||
statusDecisions,
|
||||
statusDecisionEffects,
|
||||
workAssessments,
|
||||
workspaceOperations,
|
||||
} from "@paperclipai/db";
|
||||
@@ -215,7 +216,7 @@ async function claimCoordinator(input: {
|
||||
coordinator.leaseOwner !== leaseOwner
|
||||
)
|
||||
throw new Error("native_finalization_lease_busy");
|
||||
if (coordinator.decisionId) await notifyNativePendingEffects(tx, coordinator);
|
||||
if (coordinator.decisionId) await notifyNativeFinalizationWork(tx, coordinator);
|
||||
const [updated] = await tx
|
||||
.update(nativeRunFinalizations)
|
||||
.set({
|
||||
@@ -383,7 +384,7 @@ async function recordRetryableFailure(input: {
|
||||
? "native_workspace_sync_out_retry_exhausted"
|
||||
: "native_finalization_retry_exhausted"
|
||||
: input.failureCode;
|
||||
await notifyNativePendingEffects(tx, { companyId: input.run.companyId, runId: input.run.id });
|
||||
await notifyNativeFinalizationWork(tx, { companyId: input.run.companyId, runId: input.run.id });
|
||||
if (phase === "terminal_failure") await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.run.companyId);
|
||||
if (phase === "terminal_failure" && failureCode === "native_workspace_sync_out_unsafe_archive") {
|
||||
await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: input.run.companyId, issueId: input.coordinator.issueId });
|
||||
@@ -848,6 +849,21 @@ async function acceptedResponseDigestMatches(
|
||||
export async function repairCommittedNativeChatResponse(
|
||||
db: Db,
|
||||
input: { companyId: string; issueId: string; runId: string },
|
||||
): Promise<boolean> {
|
||||
try {
|
||||
return await recoverCommittedNativeChatResponse(db, input);
|
||||
} catch (error) {
|
||||
logger.warn(
|
||||
{ runId: input.runId, error },
|
||||
"Committed chat response is awaiting presentation retry",
|
||||
);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export async function recoverCommittedNativeChatResponse(
|
||||
db: Db,
|
||||
input: { companyId: string; issueId: string; runId: string; onlyWhenDue?: boolean },
|
||||
): Promise<boolean> {
|
||||
let agentId: string | null = null;
|
||||
let materialized = false;
|
||||
@@ -909,9 +925,11 @@ export async function repairCommittedNativeChatResponse(
|
||||
return false;
|
||||
// An existing selected answer, including a subsequently deleted one, must
|
||||
// never be replaced or resurrected by a background presentation repair.
|
||||
const recordedPresentation = record(record(run.resultJson).presentationDecision);
|
||||
if (
|
||||
typeof record(record(run.resultJson).presentationDecision).commentId ===
|
||||
"string"
|
||||
typeof recordedPresentation.commentId === "string" ||
|
||||
(recordedPresentation.commentAction !== undefined && recordedPresentation.commentAction !== "create") ||
|
||||
record(run.contextSnapshot).skipIssueComment === true
|
||||
)
|
||||
return false;
|
||||
const [coordinator] = await tx
|
||||
@@ -933,6 +951,16 @@ export async function repairCommittedNativeChatResponse(
|
||||
!coordinator.assessmentId
|
||||
)
|
||||
return false;
|
||||
if (input.onlyWhenDue) {
|
||||
if (coordinator.nextAttemptAt && coordinator.nextAttemptAt.getTime() > Date.now()) return false;
|
||||
const [pendingEffect] = await tx.select({ id: statusDecisionEffects.id }).from(statusDecisionEffects).where(and(
|
||||
eq(statusDecisionEffects.companyId, input.companyId),
|
||||
eq(statusDecisionEffects.issueId, input.issueId),
|
||||
eq(statusDecisionEffects.decisionId, coordinator.decisionId),
|
||||
inArray(statusDecisionEffects.deliveryState, ["pending", "failed"]),
|
||||
)).limit(1);
|
||||
if (pendingEffect) return false;
|
||||
}
|
||||
const [accepted] = await tx
|
||||
.select()
|
||||
.from(nativeRunResults)
|
||||
@@ -1123,13 +1151,7 @@ export async function repairCommittedNativeChatResponse(
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof CommittedChatResponseAuthorizationError) return false;
|
||||
// Keep transient failures visible and retryable without reclassifying the
|
||||
// committed run or acknowledging a publication that never committed.
|
||||
logger.warn(
|
||||
{ runId: input.runId, error },
|
||||
"Committed chat response is awaiting presentation retry",
|
||||
);
|
||||
return false;
|
||||
throw error;
|
||||
}
|
||||
if (materialized && agentId)
|
||||
publishChatPublicationCommitSignal({
|
||||
@@ -1167,6 +1189,7 @@ export async function finalizeNativeRun(input: {
|
||||
workspaceFinalizeStatus: "succeeded" | "failed";
|
||||
/** Reconciliation owns terminal run projection; the live heartbeat does it afterward. */
|
||||
projectRunStatus?: boolean;
|
||||
skipNativeResponsePresentation?: boolean;
|
||||
/** Workspace-only replay must not consume the provider recovery budget. */
|
||||
preserveProviderAttempt?: boolean;
|
||||
failpoint?: NativeStatusCommitFailpoint;
|
||||
@@ -1220,7 +1243,7 @@ export async function finalizeNativeRun(input: {
|
||||
await projectCommittedRun({ db: input.db, run: admittedRun, coordinator });
|
||||
if (input.projectRunStatus && !presentationAlreadyMaterialized)
|
||||
await materializeCommittedReviewResponse(input.db, input.runId);
|
||||
if (input.projectRunStatus)
|
||||
if (input.projectRunStatus && !input.skipNativeResponsePresentation)
|
||||
await repairCommittedNativeChatResponse(input.db, {
|
||||
companyId: admittedRun.companyId,
|
||||
issueId: coordinator.issueId,
|
||||
@@ -1721,7 +1744,7 @@ export async function finalizeNativeRun(input: {
|
||||
}
|
||||
if (input.projectRunStatus)
|
||||
await materializeCommittedReviewResponse(input.db, input.runId);
|
||||
if (input.projectRunStatus && finalizationPhase === "committed")
|
||||
if (input.projectRunStatus && finalizationPhase === "committed" && !input.skipNativeResponsePresentation)
|
||||
await repairCommittedNativeChatResponse(input.db, {
|
||||
companyId: run.companyId,
|
||||
issueId: coordinator.issueId,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
|
||||
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
||||
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
|
||||
@@ -3898,7 +3898,7 @@ export async function reconcileRetainedNativeSessionCleanup(
|
||||
renameSync(owned.root, archive);
|
||||
}
|
||||
renameSync(copy, owned.root);
|
||||
await notifyNativePendingEffects(tx, { companyId: owned.run.companyId, runId: owned.run.id });
|
||||
await notifyNativeFinalizationWork(tx, { companyId: owned.run.companyId, runId: owned.run.id });
|
||||
await tx
|
||||
.update(nativeRunFinalizations)
|
||||
.set({
|
||||
@@ -3956,7 +3956,7 @@ export async function reconcileRetainedNativeSessionCleanup(
|
||||
.limit(1)
|
||||
.then((rows) => rows[0]);
|
||||
if (!current || current.leaseOwner !== leaseOwner) return;
|
||||
await notifyNativePendingEffects(tx, { companyId: owned.run.companyId, runId: owned.run.id });
|
||||
await notifyNativeFinalizationWork(tx, { companyId: owned.run.companyId, runId: owned.run.id });
|
||||
await tx
|
||||
.update(nativeRunFinalizations)
|
||||
.set({
|
||||
@@ -9700,7 +9700,7 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
)
|
||||
.for("update");
|
||||
await notifyExecutionRunWork(tx);
|
||||
await notifyNativePendingEffects(tx, input.execution.binding);
|
||||
await notifyNativeFinalizationWork(tx, input.execution.binding);
|
||||
const updated = await tx
|
||||
.update(nativeRunFinalizations)
|
||||
.set({
|
||||
@@ -10036,7 +10036,7 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
}
|
||||
const releaseNow = new Date();
|
||||
const released = await timeController("coordinator_release", () => input.db.transaction(async tx => {
|
||||
await notifyNativePendingEffects(tx, input.execution.binding);
|
||||
await notifyNativeFinalizationWork(tx, input.execution.binding);
|
||||
return tx.update(nativeRunFinalizations)
|
||||
.set({
|
||||
leaseOwner: null,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
||||
import { alias } from "drizzle-orm/pg-core";
|
||||
import { notifyUnsafeWorkspaceExportRecovery, type UnsafeWorkspaceExportScope } from "../unsafe-workspace-export-notifications.js";
|
||||
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
||||
@@ -101,7 +101,7 @@ async function recoverCandidate(db: Db, candidate: { runId: string; companyId: s
|
||||
phase: "workspace_finalize", status: "succeeded", exitCode: 0, finishedAt: now,
|
||||
metadata: { owningService: "native_workspace_finalizer", workspaceSync: { omitted: true, reason: "restore_unsafe_archive", legacy: true } },
|
||||
});
|
||||
await notifyNativePendingEffects(tx, { companyId: run.companyId, runId: run.id });
|
||||
await notifyNativeFinalizationWork(tx, { companyId: run.companyId, runId: run.id });
|
||||
await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null,
|
||||
failureDetail: { workspaceFinalizeAttempt: 0, legacyUnsafeExportOmitted: { resultId: result.id, recoveredAt: now.toISOString() } }, updatedAt: now,
|
||||
}).where(eq(nativeRunFinalizations.runId, run.id));
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
||||
import { notifyEnvironmentCleanup } from "../environment-cleanup-notifications.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import type { Environment, EnvironmentLease } from "@paperclipai/shared";
|
||||
@@ -135,7 +135,7 @@ export async function retryNativeWorkspaceExport(input: {
|
||||
const now = new Date();
|
||||
const retry = { ...receipt, requestId: randomUUID(), actorId: input.actorId, repairNote: input.repairNote, requestedAt: now.toISOString(),
|
||||
stoppedProvider: initial.lease.metadata?.remoteExecutionTermination };
|
||||
await notifyNativePendingEffects(tx, { companyId: current.run.companyId, runId: current.run.id });
|
||||
await notifyNativeFinalizationWork(tx, { companyId: current.run.companyId, runId: current.run.id });
|
||||
await tx.update(nativeRunFinalizations).set({ phase: "result_accepted", failureCode: null, nextAttemptAt: null,
|
||||
failureDetail: { ...current.coordinator.failureDetail, workspaceFinalizeAttempt: 0, workspaceExportRetry: retry }, updatedAt: now }).where(eq(nativeRunFinalizations.runId, current.run.id));
|
||||
await tx.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null,
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { nativeResponsePresentationWork } from "./native-response-presentation-work.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { asc, eq, sql } from "drizzle-orm";
|
||||
import { generateKeyPairSync, randomUUID } from "node:crypto";
|
||||
@@ -65,6 +67,8 @@ import {
|
||||
subscribeCompanyLiveEvents,
|
||||
} from "../live-events.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_CONTROL_PLANE_TEST_DATABASE_URL");
|
||||
|
||||
describe("PaperclipControlPlanePort conformance", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startRecoveryPostgres>> | null = null;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
@@ -87,8 +91,11 @@ describe("PaperclipControlPlanePort conformance", () => {
|
||||
const governanceRunnerId = "00000000-0000-4000-8000-000000000021";
|
||||
|
||||
beforeAll(async () => {
|
||||
temporary = await startRecoveryPostgres("paperclip-native-port-");
|
||||
db = createDb(temporary.connectionString);
|
||||
if (external) db = createDb(external);
|
||||
else {
|
||||
temporary = await startRecoveryPostgres("paperclip-native-port-");
|
||||
db = createDb(temporary.connectionString);
|
||||
}
|
||||
const identity = CONTROL_PLANE_CONFORMANCE_OPEN.identity;
|
||||
await db.insert(companies).values({ id: identity.companyId, name: "Phase 6", issuePrefix: "P6C" });
|
||||
await db.insert(agents).values({
|
||||
@@ -645,7 +652,7 @@ describe("PaperclipControlPlanePort conformance", () => {
|
||||
completionContractSha256: localContractSha,
|
||||
contextSnapshot: { issueId },
|
||||
});
|
||||
const observerDb = createDb(temporary!.connectionString);
|
||||
const observerDb = createDb(external ?? temporary!.connectionString);
|
||||
const observed: Array<Record<string, unknown>> = [];
|
||||
const committedCallbacks: string[] = [];
|
||||
let visibilityCheck: Promise<Array<{ eventType: string }>> | null = null;
|
||||
@@ -1096,7 +1103,10 @@ describe("PaperclipControlPlanePort conformance", () => {
|
||||
.where(eq(heartbeatRuns.id, taskRunId));
|
||||
expect(await repair()).toBe(false);
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: {} }).where(eq(heartbeatRuns.id, taskRunId));
|
||||
expect(await repair()).toBe(true);
|
||||
const work = nativeResponsePresentationWork(db);
|
||||
const [entry] = await work.load({ companyId: identity.companyId, runId: taskRunId });
|
||||
expect(entry?.id).toBe(taskRunId);
|
||||
expect(await work.check(entry)).toBeNull();
|
||||
expect(await repair()).toBe(false);
|
||||
expect(await db.select({ body: issueComments.body }).from(issueComments)
|
||||
.where(eq(issueComments.createdByRunId, taskRunId)).orderBy(asc(issueComments.createdAt)))
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyNativePendingEffects } from "../native-pending-effect-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
|
||||
import { notifyAutomaticCompletionReviews } from "../automatic-completion-review-notifications.js";
|
||||
import { notifyObsoletePolicyReviews } from "../obsolete-policy-review-notifications.js";
|
||||
@@ -154,7 +154,7 @@ export async function dispatchPendingNativeStatusEffects(input: {
|
||||
targetId: effect.targetId,
|
||||
});
|
||||
}
|
||||
await notifyNativePendingEffects(tx, input);
|
||||
await notifyNativeFinalizationWork(tx, input);
|
||||
await notifyChatRunMilestones(tx, input);
|
||||
await tx
|
||||
.update(statusDecisionEffects)
|
||||
@@ -1241,7 +1241,7 @@ async function materializeDecisionEffect(input: {
|
||||
)
|
||||
.for("update");
|
||||
if (pending.length === 0) throw new Error("native_pending_effect_missing");
|
||||
await notifyNativePendingEffects(input.tx, { companyId: input.companyId });
|
||||
await notifyNativeFinalizationWork(input.tx, { companyId: input.companyId, issueId: input.issue.id });
|
||||
await input.tx
|
||||
.update(statusDecisionEffects)
|
||||
.set({
|
||||
@@ -1931,7 +1931,7 @@ export async function commitNativeStatusDecision(input: {
|
||||
effect => effect.kind === "record_finalization_error",
|
||||
);
|
||||
await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, input.companyId);
|
||||
await notifyNativePendingEffects(tx, input);
|
||||
await notifyNativeFinalizationWork(tx, input);
|
||||
await tx
|
||||
.update(nativeRunFinalizations)
|
||||
.set({
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { notifyNativePendingEffects } from "./native-pending-effect-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "./native-finalization-notifications.js";
|
||||
import { and, desc, eq, gte, ne, or, sql } from "drizzle-orm";
|
||||
import { agentWakeupRequests, heartbeatRuns, issueRecoveryActions, issues, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
|
||||
import { readSystemRecovery } from "./system-recovery-state.js";
|
||||
@@ -46,7 +46,7 @@ export async function resumePreservedResultFinalization(db: Db, companyId: strin
|
||||
const generationId = intent?.id ?? recovery.generationId;
|
||||
if (coordinator.recoveryHistory.some(entry => entry.kind === "preserved_result_finalization" &&
|
||||
entry.generationId === generationId)) return false;
|
||||
await notifyNativePendingEffects(tx, { companyId, runId });
|
||||
await notifyNativeFinalizationWork(tx, { companyId, runId });
|
||||
await tx.update(nativeRunFinalizations).set({
|
||||
phase: coordinator.assessmentId ? "arbitrating" : "workspace_finalizing",
|
||||
nextAttemptAt: new Date(), failureCode: null, failureDetail: null, updatedAt: new Date(),
|
||||
|
||||
Reference in new issue
Block a user