mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
feat(slack): show useful progress without extra messages (#20827)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Chat connectors show agent work in the source conversation. > - Slack progress uses a small set of general status messages. > - Native runtimes already mark some completed assistant notes as public progress. > - The server can select those notes and control their size and rate. > - This pull request keeps one progress message and sends the final answer as a new reply. > - Users can follow useful changes without reading each tool event. ## Linked Issues or Issue Description **What existing behavior does this improve?** Automatic progress updates in Slack task threads. **Current behavior** Users see general messages such as “Maya is making progress”. The first answer can replace that message. A source update that arrives during the progress delay can be lost if no later event arrives. **Proposed behavior** Show the latest short public note in one message. Wait 10 seconds before the first update and at least 30 seconds between updates. Recover pending work after a restart. Send the final answer as a fresh reply. **Reason and benefit** Users can see what the agent checks and what it found. The thread stays short. The model needs no new tool call or summary model. Related change: [resolved-card chat bindings](https://github.com/paperclipai/paperclip/pull/20825). This branch includes that master revision. The progress change uses the existing source authority checks. ## What Changed - Select completed `agentMessage` items from the normalized PRP `progress` channel. Keep unknown channels, reasoning, tool output, and raw provider fields out of progress text. - Sanitize each note. Keep at most two sentences and 400 characters. Add a task link when a public board URL is available. - Prefer public notes over general tool status. Remove repeated text and combine bursts into the latest pending note. - Use run events and the publication outbox as saved state. Use the shared deadline worker for the grace period and update delay. A delivery receipt wakes the worker when an earlier update is still in flight. Settled history does not poll. - Apply the run's registered-secret filter before text selection or storage. Reuse the latest usable note within a sweep. Invalidate the cache when the source event or secret registry changes. - Reuse the exact source-bound Slack fast-response receipt when it replaced the working placeholder. Check its current outbound message link before editing it. - Send Slack answers as new replies. After delivery, queue a separate cleanup edit for a generic working message. Keep its current outbound link as the edit authority. Leave useful public notes in place. Cancel pending public progress when an answer, question, or terminal run takes over. Keep the existing delivery policy for other connectors. - Add one short Slack prompt instruction. Update product documentation and regression tests. The Slack module owns public-note selection, cadence, lane eligibility, and cleanup. The shared progress service holds its existing issue and run locks and calls a narrow provider policy. The secret registry supplies redaction and a version token for the per-sweep note cache. The existing outbox owns delivery, retries, and authority checks. The Slack module owns receipt reuse. Shared delivery changes supply the receipt wake and select the Slack reply policy. No schema or configuration change is required. ## Verification - The initial content-selection regression failed before the implementation and passed after it. - The follow-up reproduced two review defects before the fixes. A registered synthetic secret appeared in saved progress. A generic working message stayed active after the answer. Both regressions now pass. - The final content, database-worker, query, protocol-boundary, and redaction selection passes 78 tests across four files. - The earlier affected transport selection passes all 65 cases. It covers public progress, native cadence, separate answers, Slack and Telegram retry isolation, FIFO restart, and cleanup ownership. Its first run passed 64 cases and found one old assertion that rejected the intended cleanup edit. The corrected assertion checks the exact successor message ID. - Full server typecheck passes: `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm --filter @paperclipai/server typecheck`. An earlier attempt found two transaction type errors. The helper contracts now accept their actual database operations. The final run passes all source and test layers. - The broad Slack selection stopped after 189 passing cases when the sandbox killed its worker with SIGKILL. That attempt is not a pass. The focused transport run above supplies affected local coverage. Required CI must supply broad coverage on the final head. - `pnpm verify --explain` returns exit 2 because the product documentation change requires broad CI coverage. No full local suite was started. - Preflight installed dependencies and built the SDK. Cargo is absent, so local Rust checks could not run. No Rust source changed. - [CI on `02552bb584`](https://github.com/paperclipai/paperclip/actions/runs/43493597242) passed build, typecheck, policy, all browser shards, Runner/Rust checks, and the other test lanes. One chat shard failed in three Slack cases. Each expected one edit after an answer was already sent. The intended cleanup adds a “Response sent.” edit before the later failure notice. All three failures reproduced locally. The test assertions now check both exact edit texts and the original run's message ID. They also check that another sweep adds no edits. - The latest local run passes all 85 selected transport cases. It includes all 18 Slack/Telegram answer and failure orders, plus progress, deferred FIFO, retry isolation, and cleanup coverage. Command: `pnpm exec vitest run server/src/__tests__/chat-channels.integration.test.ts -t 'preserves selected answer and failure lanes|public progress|native progress|keeps intermediate.*agent comments|keeps a.*retry.*queue|FIFO'`. - [Final CI](https://github.com/paperclipai/paperclip/actions/runs/43499797545) passed on `55ec4c053fe5263d9033d807777dbf2e679407e3`. All 52 executed jobs passed. This includes build, typecheck, general tests, serialized tests, all eight browser shards, Runner/Rust checks, and the required aggregates. The affected chat shard now passes. The test-file selection job was skipped because this run used full coverage. - [Greptile](https://github.com/paperclipai/paperclip/runs/132142301124) completed on the same head with 5/5 and no new findings. All four inline review threads are resolved. The review job and Socket checks passed. Of 61 check runs, 57 succeeded, three were skipped, and one was neutral. No checks remain pending or failed. The PR is mergeable with a clean state. - [Superagent Security Scan](https://github.com/paperclipai/paperclip/runs/132142345827) returned a neutral result with “PR scan workflow failed”. It supplied no security verdict. Do not count it as a successful scan. - Live Slack notification and unread behavior has not been tested. Service tests use real PostgreSQL and a fake provider runtime. ## Risks - Only runtimes that map commentary to the explicit public `progress` channel supply contextual prose. Other runtimes retain general status messages. - Slack final answers now create a fresh reply. This is an intended delivery change. - Public commentary must suit the current conversation audience. Text sanitization adds protection; it does not authorize sharing unrelated private material. - Source events and outbox rows supply recovery state. No new timer table or recurring idle scan is added. - The existing large integration suite retains its provider fixtures and lifecycle assertions. No source file crosses the 1,000-line threshold in this change. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code tools. The exact deployment ID was not supplied. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
06a4a07781
commit
0ead74b4f3
16 files changed
+632
-104
No files matched your search
@@ -417,6 +417,25 @@ in-app only.
|
||||
|
||||
Coordinator is the standard workspace for everyone. Settings → Experimental offers a personal **Classic workspace** opt-in, off by default. Historical Coordinator experiment settings do not enable Classic. Agent Chat is available to everyone on new and existing instances. The retired `enableAgentChat` key remains accepted for compatibility; stored and managed off values are ignored. Each person has one persistent task-backed conversation per agent and company, with ordinary company task visibility. The shared task composer, transcript, tools, files, and document panel remain the interaction surface. Agents clarify goals and hand substantial execution to linked, assigned tasks; a reply ends a turn without completing the conversation. `/new` starts fresh provider context in the same conversation while preserving visible history and artifacts. Healthy idle conversations wait for a message and do not count as unfinished execution work. See `doc/plans/2026-09-10-agent-chat.md` for the implementation contract.
|
||||
|
||||
### Slack task progress
|
||||
|
||||
Slack threads reuse one editable progress message for each active run. Native
|
||||
runtimes with an explicit public progress channel can supply a short completed
|
||||
commentary note, up to 400 characters, plus a task link when a public board URL
|
||||
is configured. The server coalesces bursts, suppresses repeated text, allows a
|
||||
10-second initial grace period, and waits at least 30 seconds between updates.
|
||||
A pending note is recovered from run events and the publication outbox after a
|
||||
restart. Delivery receipts wake the progress owner; idle history does not poll.
|
||||
|
||||
Reasoning, unknown assistant channels, raw tool output, and provider diagnostics
|
||||
cannot supply progress prose. Providers without a public channel retain factual
|
||||
lifecycle fallbacks. Questions, approvals, and final answers bypass the progress
|
||||
cooldown. Slack final answers use a fresh reply, so users can see completion
|
||||
without watching message edits. After the answer is sent, a generic working
|
||||
message changes to “Response sent.” A useful public note stays in place.
|
||||
Registered run secrets are redacted before progress text is saved or sent.
|
||||
Other connectors retain their delivery policy.
|
||||
|
||||
### Agent chat project handoff (2026-09-11)
|
||||
|
||||
Chat supports research and full plan drafting/revision in its existing plan document. On handoff, each ordinary assigned task receives the relevant plan in its own `plan` document, committed with task creation before execution is scheduled. The source plan remains in the conversation. Plan acceptance hands off execution; it never switches the conversation into implementation.
|
||||
|
||||
@@ -1324,6 +1324,13 @@ describe("renderPaperclipWakePrompt", () => {
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("asks Slack for brief public progress without adding a reporting tool", () => {
|
||||
const prompt = renderPaperclipWakePrompt({ ...ordinaryExternalChatWake, externalChatProvider: "slack" });
|
||||
expect(prompt).toContain("Give a brief public update when you change approach or learn something useful");
|
||||
expect(prompt).toContain("current conversation's audience");
|
||||
expect(renderPaperclipWakePrompt(ordinaryExternalChatWake)).not.toContain("Slack can show this public commentary");
|
||||
});
|
||||
|
||||
it("renders one authoritative direct-response contract for fresh and resumed external-chat turns", () => {
|
||||
for (const prompt of [
|
||||
renderPaperclipWakePrompt(ordinaryExternalChatWake),
|
||||
|
||||
@@ -2450,6 +2450,9 @@ function renderPaperclipWakePromptBody(
|
||||
]),
|
||||
"The harness owns task state and persists your final assistant response. If the runtime offers a semantic completion operation, emit exactly one semantic completion and do not duplicate that response in a Paperclip comment or status update.",
|
||||
"The semantic completion summary is the user-visible final answer. Include every requested answer, exact value, description, and any actionable file-access or delivery limitation there; a statement that you read, checked, or prepared something is not a substitute. Private progress commentary is not delivered as the final answer.",
|
||||
...(normalized.externalChatProvider === "slack"
|
||||
? ["Give a brief public update when you change approach or learn something useful. Say what changed and what you will check next. Slack can show this public commentary as progress. Keep it suitable for the current conversation's audience; do not include internal reasoning or raw tool output."]
|
||||
: []),
|
||||
...(normalized.externalChatProvider === "github"
|
||||
? [
|
||||
"GitHub publication uses the task-scoped bot tools when they are available. Paperclip creates one working comment for this request. You may and should periodically edit that same comment with update_comment during longer work, reporting brief, factual progress or blockers before the final result. Use a distinct stable idempotency key for each update and reuse it for retries. Publish the final answer with comment, or a requested review with begin_review and submit_review; these replace the same working comment. This is the provider reply, not a Paperclip progress or completion comment. Your semantic completion remains internal to Paperclip and does not post another GitHub reply. Never substitute personal credentials or choose another comment, connection, or repository.",
|
||||
|
||||
@@ -72,6 +72,7 @@ import { companyFastResponses, fastResponseRequests } from "@paperclipai/db";
|
||||
import { toolAccessService } from "../services/tool-access.js";
|
||||
import { toolActionRequests, toolInvocations, toolPolicies } from "@paperclipai/db";
|
||||
import { GitHubPublicationLeaseLost, withGitHubPublicationLease } from "../services/chat-github-publication-lease.js";
|
||||
import { createRunSecretRedactionRegistry } from "../services/run-secret-redaction.js";
|
||||
import { secretService } from "../services/secrets.js";
|
||||
import { githubChatWizardService } from "../services/chat-github-wizard.js";
|
||||
import { dispatchGitHubBotCloudEvent, registerGitHubBotCloudSetupPoller } from "../services/chat-github-cloud-ingress.js";
|
||||
@@ -11007,6 +11008,35 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
expect(receipt).toMatchObject({ authorAgentId: fixture.assignedAgentId, createdByRunId: null });
|
||||
expect((await readIssuesById(job.issueId!))[0].status).not.toBe("done");
|
||||
expect(provider).toHaveBeenCalledTimes(1);
|
||||
if (channelProvider === "slack") {
|
||||
await db.update(heartbeatRuns).set({ runtimeMode: "native", status: "running",
|
||||
contextSnapshot: await chatWakeContext({ endpointId: endpoint.id, issueId: job.issueId!, provider: "slack", providerMessageId: "9000.1" }),
|
||||
}).where(eq(heartbeatRuns.id, queuedRun.id));
|
||||
await db.insert(chatPublications).values({ companyId: fixture.companyId, endpointId: endpoint.id,
|
||||
conversationId: job.conversationId!, issueId: job.issueId!, idempotencyKey: `run:${queuedRun.id}:working:${endpoint.id}`,
|
||||
payload: { text: "Maya is working…", progressState: "working" }, createdAt: new Date(Date.now() - 60_000),
|
||||
});
|
||||
await service.processPendingPublications();
|
||||
const [working] = await db.select().from(chatPublications).where(eq(chatPublications.idempotencyKey, `run:${queuedRun.id}:working:${endpoint.id}`));
|
||||
expect(working).toMatchObject({ state: "cancelled", redactedError: "Acknowledged by contextual fast response" });
|
||||
await db.insert(heartbeatRunEvents).values({ companyId: fixture.companyId, agentId: fixture.assignedAgentId, runId: queuedRun.id,
|
||||
seq: 1, eventType: "item.completed", payload: { prpEvent: { schema: "paperclip.prp.event.v1", eventType: "item.completed", runId: queuedRun.id,
|
||||
payload: { kind: "agentMessage", channel: "progress", text: "Found the border rule. Checking the fix." } } },
|
||||
});
|
||||
await db.insert(heartbeatRunEvents).values({ companyId: fixture.companyId, agentId: fixture.assignedAgentId, runId: queuedRun.id,
|
||||
seq: 2, eventType: "tool.execution.completed", payload: { result: "PRIVATE tool output" },
|
||||
});
|
||||
expect(await enqueueSafeNativeChatProgress(db, { scope: { companyId: fixture.companyId, issueId: job.issueId! } })).toBe(1);
|
||||
await service.processPendingPublications();
|
||||
const slack = runtime.endpoints.get(endpoint.id)!;
|
||||
expect(slack.posts).toHaveLength(1);
|
||||
expect(slack.edits).toEqual([expect.objectContaining({ text: expect.stringContaining("Found the border rule. Checking the fix.") })]);
|
||||
await addSelectedChatFinal({ companyId: fixture.companyId, agentId: fixture.assignedAgentId, issueId: job.issueId!, runId: queuedRun.id, body: "Fixed the border." });
|
||||
await service.processPendingPublications();
|
||||
expect(slack.posts).toHaveLength(2);
|
||||
expect(slack.posts[1].text).toBe("Fixed the border.");
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, queuedRun.id));
|
||||
}
|
||||
});
|
||||
|
||||
it("mirrors Paperclip messages with user attribution and returns only the selected Slack reply", async () => {
|
||||
@@ -36006,14 +36036,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
const providerRuntime = runtime.endpoints.get(endpoint.id);
|
||||
expect(providerRuntime?.posts.map((post) => post.text)).toEqual(provider === "github" ? [] : [
|
||||
"Maya is working…",
|
||||
"answer-final",
|
||||
]);
|
||||
expect(providerRuntime?.edits).toEqual(provider === "github" ? [] : [
|
||||
{
|
||||
threadId: thread.thread.id,
|
||||
messageId: "outbound-1",
|
||||
text: "answer-final",
|
||||
},
|
||||
]);
|
||||
await service.processPendingPublications();
|
||||
expect(providerRuntime?.edits).toEqual(provider === "slack" ? [expect.objectContaining({
|
||||
messageId: "outbound-1", text: "Response sent.",
|
||||
})] : []);
|
||||
const commentPublications = await db
|
||||
.select()
|
||||
.from(chatPublications)
|
||||
@@ -36028,7 +36056,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
expect.objectContaining({
|
||||
commentId: comments[2].id,
|
||||
state: "published",
|
||||
providerMessageId: "outbound-1",
|
||||
providerMessageId: "outbound-2",
|
||||
}),
|
||||
]);
|
||||
expect(
|
||||
@@ -36052,7 +36080,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
eq(chatMessageLinks.direction, "outbound"),
|
||||
),
|
||||
),
|
||||
).toHaveLength(provider === "github" ? 0 : 1);
|
||||
).toHaveLength(provider === "github" ? 0 : 2);
|
||||
await service.shutdown();
|
||||
},
|
||||
);
|
||||
@@ -36355,7 +36383,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
result: { sessionStatus: "active" },
|
||||
});
|
||||
|
||||
// A final response produces one edit, then a fresh revision of the status
|
||||
// A final response produces one new reply, then a fresh revision of the status
|
||||
// lane. Neither a retry nor restart posts another copy of that response.
|
||||
await addSelectedChatFinal({
|
||||
agentId: fixture.assignedAgentId,
|
||||
@@ -36369,11 +36397,11 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
await resumed.service.processPendingSlackSessionSyncs(25, action.id);
|
||||
expect(statusCalls).toEqual(["processing", "active", "active"]);
|
||||
expect(firstProviderRuntime.posts.length).toBe(providerPosts);
|
||||
expect(runtime.endpoints.get(endpoint.id)?.posts.length ?? 0).toBe(0);
|
||||
expect(runtime.endpoints.get(endpoint.id)?.posts.length ?? 0).toBe(1);
|
||||
expect(
|
||||
runtime.endpoints
|
||||
.get(endpoint.id)
|
||||
?.edits.filter((edit) => edit.text === "SLACK-SESSION-DONE"),
|
||||
?.posts.filter((post) => post.text === "SLACK-SESSION-DONE"),
|
||||
).toHaveLength(1);
|
||||
|
||||
// A worker can select a due status row, then lose to another worker before
|
||||
@@ -36510,7 +36538,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("coalesces one Slack run's lifecycle and final response despite an interleaved task control", async () => {
|
||||
it("keeps one Slack progress message and a separate final response despite an interleaved task control", async () => {
|
||||
const fixture = await seedCompany();
|
||||
const { callbacks, endpoint, runtime, service } =
|
||||
await configuredSlackEndpoint(fixture);
|
||||
@@ -36607,6 +36635,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
const providerRuntime = runtime.endpoints.get(endpoint.id);
|
||||
expect(providerRuntime?.posts).toEqual([
|
||||
{ threadId: thread.thread.id, text: "Maya is queued." },
|
||||
{ threadId: thread.thread.id, text: finalText, chunks: [finalText] },
|
||||
]);
|
||||
expect(providerRuntime?.edits).toEqual([
|
||||
{
|
||||
@@ -36619,11 +36648,6 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
messageId: "outbound-1",
|
||||
text: expect.stringMatching(/ — /),
|
||||
},
|
||||
{
|
||||
threadId: thread.thread.id,
|
||||
messageId: "outbound-1",
|
||||
text: finalText,
|
||||
},
|
||||
]);
|
||||
const renderedSlackMessages = new Map(
|
||||
providerRuntime!.posts.map((post, index) => [
|
||||
@@ -36633,7 +36657,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
);
|
||||
for (const edit of providerRuntime!.edits)
|
||||
renderedSlackMessages.set(edit.messageId, edit.text);
|
||||
expect([...renderedSlackMessages.values()]).toEqual([finalText]);
|
||||
expect([...renderedSlackMessages.values()]).toEqual([expect.stringMatching(/ — /), finalText]);
|
||||
const publications = await db
|
||||
.select()
|
||||
.from(chatPublications)
|
||||
@@ -36659,10 +36683,14 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
eq(chatMessageLinks.conversationId, conversation.id),
|
||||
eq(chatMessageLinks.direction, "outbound"),
|
||||
),
|
||||
),
|
||||
).orderBy(asc(chatMessageLinks.providerMessageId)),
|
||||
).toEqual([
|
||||
expect.objectContaining({
|
||||
providerMessageId: "outbound-1",
|
||||
commentId: null,
|
||||
}),
|
||||
expect.objectContaining({
|
||||
providerMessageId: "outbound-2",
|
||||
commentId: finalComment.id,
|
||||
}),
|
||||
]);
|
||||
@@ -36673,6 +36701,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
await service.processPendingPublications();
|
||||
expect(providerRuntime?.posts).toEqual([
|
||||
{ threadId: thread.thread.id, text: "Maya is queued." },
|
||||
{ threadId: thread.thread.id, text: finalText, chunks: [finalText] },
|
||||
{ threadId: thread.thread.id, text: expect.stringMatching(/ — /) },
|
||||
]);
|
||||
});
|
||||
@@ -69910,7 +69939,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
});
|
||||
|
||||
it.each(["slack", "telegram"] as const)(
|
||||
"keeps a %s retry's queue, working, and selected final on its own provider identity",
|
||||
"keeps a %s retry's queue and working progress on its own provider identity and delivers its selected final",
|
||||
async (provider) => {
|
||||
const context = await failedChatRetryFixture(provider);
|
||||
try {
|
||||
@@ -69980,12 +70009,24 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
expect(finalPublication).toMatchObject({
|
||||
state: "published",
|
||||
attempts: 1,
|
||||
providerMessageId: queued.providerMessageId,
|
||||
});
|
||||
expect(context.providerRuntime.posts).toHaveLength(1);
|
||||
expect(context.providerRuntime.edits.at(-1)?.text).toContain(
|
||||
"EXACT-RETRY-ANSWER",
|
||||
);
|
||||
if (provider === "slack") {
|
||||
expect(finalPublication.providerMessageId).not.toBe(
|
||||
queued.providerMessageId,
|
||||
);
|
||||
expect(context.providerRuntime.posts).toHaveLength(2);
|
||||
expect(context.providerRuntime.posts.at(-1)?.text).toContain(
|
||||
"EXACT-RETRY-ANSWER",
|
||||
);
|
||||
} else {
|
||||
expect(finalPublication.providerMessageId).toBe(
|
||||
queued.providerMessageId,
|
||||
);
|
||||
expect(context.providerRuntime.posts).toHaveLength(1);
|
||||
expect(context.providerRuntime.edits.at(-1)?.text).toContain(
|
||||
"EXACT-RETRY-ANSWER",
|
||||
);
|
||||
}
|
||||
} finally {
|
||||
await context.service.shutdown();
|
||||
}
|
||||
@@ -71810,18 +71851,24 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
failurePublication?.providerMessageId,
|
||||
);
|
||||
expect(
|
||||
order === "failure_first"
|
||||
provider === "slack" || order === "failure_first"
|
||||
? failurePublication?.providerMessageId
|
||||
: answerPublication?.providerMessageId,
|
||||
).toBe(originalWorking!.providerMessageId);
|
||||
expect(context.providerRuntime.posts).toHaveLength(3);
|
||||
expect(context.providerRuntime.edits).toHaveLength(1);
|
||||
expect(context.providerRuntime.edits[0]?.messageId).toBe(
|
||||
originalWorking!.providerMessageId,
|
||||
);
|
||||
expect(context.providerRuntime.edits[0]?.messageId).not.toBe(
|
||||
successorWorking!.providerMessageId,
|
||||
);
|
||||
const expectedEdits = [
|
||||
...(provider === "slack" && order === "published_final"
|
||||
? ["Response sent."]
|
||||
: []),
|
||||
provider === "slack" || order === "failure_first"
|
||||
? failurePublication!.payload.text
|
||||
: answerPublication!.payload.text,
|
||||
].map((text) => ({
|
||||
threadId: context.thread.thread.id,
|
||||
messageId: originalWorking!.providerMessageId,
|
||||
text,
|
||||
}));
|
||||
expect(context.providerRuntime.edits).toEqual(expectedEdits);
|
||||
const links = await db
|
||||
.select({
|
||||
providerMessageId: chatMessageLinks.providerMessageId,
|
||||
@@ -71861,7 +71908,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
await enqueueChatRunMilestones(db);
|
||||
await context.service.processPendingPublications(100);
|
||||
expect(context.providerRuntime.posts).toHaveLength(3);
|
||||
expect(context.providerRuntime.edits).toHaveLength(1);
|
||||
expect(context.providerRuntime.edits).toEqual(expectedEdits);
|
||||
await expect(
|
||||
db
|
||||
.select({ status: heartbeatRuns.status })
|
||||
@@ -72068,6 +72115,47 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it("edits Slack public progress and posts the final as a fresh reply, cancelling an older queued edit", async () => {
|
||||
const context = await safeNativeProgressFixture("slack", "109");
|
||||
try {
|
||||
const run = await context.createRun("public progress");
|
||||
const addPublic = (seq: number, text: string, channel = "progress") => db.insert(heartbeatRunEvents).values({
|
||||
companyId: context.fixture.companyId, runId: run.runId, agentId: context.fixture.assignedAgentId,
|
||||
seq, eventType: "item.completed", message: "PRIVATE event metadata", payload: { prpEvent: {
|
||||
schema: "paperclip.prp.event.v1", runId: run.runId, eventType: "item.completed",
|
||||
payload: { kind: "agentMessage", channel, text },
|
||||
} },
|
||||
});
|
||||
const secret = "synthetic-unprefixed-progress-secret";
|
||||
await createRunSecretRedactionRegistry(db).register(context.fixture.companyId, run.runId, secret);
|
||||
await addPublic(1, `Found the retry bug with ${secret}. Checking recovery next.`);
|
||||
await context.addEvent(run, "tool.execution.completed", 2);
|
||||
expect(await enqueueChatRunMilestones(db, { since: new Date(0) })).toBe(1);
|
||||
await context.service.processPendingPublications(100);
|
||||
expect(context.providerRuntime.posts).toHaveLength(1);
|
||||
expect(context.providerRuntime.edits).toEqual([expect.objectContaining({
|
||||
messageId: "outbound-1", text: expect.stringContaining("Found the retry bug with ***REDACTED***. Checking recovery next."),
|
||||
})]);
|
||||
// The cooldown must retain the newest note, with no post or edit yet.
|
||||
for (let seq = 3; seq < 23; seq++) await addPublic(seq, `Checking recovery case ${seq}.`);
|
||||
expect(await enqueueChatRunMilestones(db, { since: new Date(0) })).toBe(0);
|
||||
await db.update(chatPublications).set({ createdAt: new Date(Date.now() - 31_000), publishedAt: new Date(Date.now() - 31_000) })
|
||||
.where(like(chatPublications.idempotencyKey, `run:${run.runId}:working:${context.endpoint.id}:native:%`));
|
||||
expect(await enqueueChatRunMilestones(db, { since: new Date(0) })).toBe(1);
|
||||
await addSelectedChatFinal({ agentId: context.fixture.assignedAgentId, body: "Fixed retry recovery.",
|
||||
companyId: context.fixture.companyId, issueId: context.conversation.issueId, runId: run.runId });
|
||||
// Final authority wins even while the run still says running.
|
||||
await context.service.processPendingPublications(100);
|
||||
expect(context.providerRuntime.posts).toHaveLength(2);
|
||||
expect(context.providerRuntime.posts[1].text).toBe("Fixed retry recovery.");
|
||||
expect(context.providerRuntime.edits).toHaveLength(1);
|
||||
expect(JSON.stringify(context.providerRuntime.edits)).not.toContain("PRIVATE");
|
||||
expect(JSON.stringify(context.providerRuntime)).not.toContain(secret);
|
||||
} finally {
|
||||
await retirePublicationFixture(context.service, context.endpoint.id);
|
||||
}
|
||||
});
|
||||
|
||||
it("enforces the native progress cadence boundary and one publication per phase", async () => {
|
||||
const context = await safeNativeProgressFixture("telegram", "8");
|
||||
const run = await context.createRun("cadence boundaries");
|
||||
@@ -77705,21 +77793,22 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
}
|
||||
|
||||
it.each(["slack", "telegram"] as const)(
|
||||
"updates only the linked same-run %s failure notice when recovering an accepted response",
|
||||
"delivers a recovered %s answer without editing another run's failure notice",
|
||||
async (provider) => {
|
||||
const context = await committedChatResponseRecoveryFixture(provider);
|
||||
try {
|
||||
const unrelatedRunId = randomUUID();
|
||||
const failures = [];
|
||||
for (const [runId, messageId] of [
|
||||
[context.runId, "same-run-message"],
|
||||
[unrelatedRunId, "later-turn-message"],
|
||||
]) {
|
||||
await linkedCommittedResponseFailure(context, runId, messageId);
|
||||
failures.push(await linkedCommittedResponseFailure(context, runId, messageId));
|
||||
}
|
||||
await expect(context.repair()).resolves.toBe(true);
|
||||
await context.service.processPendingPublications(100);
|
||||
expect(context.providerRuntime.posts).toEqual([]);
|
||||
expect(context.providerRuntime.edits).toEqual([
|
||||
expect(context.providerRuntime.posts).toEqual(provider === "slack" ? [{ threadId: context.thread.thread.id, text: context.result.summary }] : []);
|
||||
expect(context.providerRuntime.edits).toEqual(provider === "slack" ? [] : [
|
||||
{
|
||||
threadId: context.thread.thread.id,
|
||||
messageId: "same-run-message",
|
||||
@@ -77727,8 +77816,8 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
},
|
||||
]);
|
||||
await context.service.processPendingPublications(100);
|
||||
expect(context.providerRuntime.edits).toHaveLength(1);
|
||||
expect(context.providerRuntime.posts).toEqual([]);
|
||||
expect(context.providerRuntime.edits).toHaveLength(provider === "slack" ? 0 : 1);
|
||||
expect(context.providerRuntime.posts).toEqual(provider === "slack" ? [{ threadId: context.thread.thread.id, text: context.result.summary }] : []);
|
||||
const [answer] = await db
|
||||
.select()
|
||||
.from(chatPublications)
|
||||
@@ -77740,7 +77829,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
);
|
||||
expect(answer).toMatchObject({
|
||||
state: "published",
|
||||
providerMessageId: "same-run-message",
|
||||
providerMessageId: provider === "slack" ? "outbound-1" : "same-run-message",
|
||||
attempts: 1,
|
||||
});
|
||||
await expect(
|
||||
@@ -77753,7 +77842,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
eq(chatMessageLinks.providerMessageId, "same-run-message"),
|
||||
),
|
||||
),
|
||||
).resolves.toEqual([{ publicationId: answer.id }]);
|
||||
).resolves.toEqual([{ publicationId: provider === "slack" ? failures[0].id : answer.id }]);
|
||||
await expect(
|
||||
db
|
||||
.select({ id: heartbeatRuns.id })
|
||||
@@ -80209,9 +80298,12 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
runId,
|
||||
});
|
||||
await restarted.processPendingPublications();
|
||||
expect(runtime.edits.at(-1)).toMatchObject({
|
||||
expect(runtime.posts.at(-1)?.text).toBe("The exact successor answer");
|
||||
expect(runtime.edits).toHaveLength(2);
|
||||
expect(runtime.edits.at(-1)).toEqual({
|
||||
threadId: context.thread.thread.id,
|
||||
messageId: queued.providerMessageId,
|
||||
text: "The exact successor answer",
|
||||
text: "Response sent.",
|
||||
});
|
||||
await addSelectedChatFinal({
|
||||
agentId: fixture.assignedAgentId,
|
||||
@@ -80223,6 +80315,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
await restarted.processPendingPublications();
|
||||
expect(runtime.posts.at(-1)?.text).toBe("A separate second answer");
|
||||
expect(runtime.edits).toHaveLength(2);
|
||||
expect(runtime.posts).toHaveLength(3);
|
||||
await expect(restarted.processPendingPublications()).resolves.toBe(0);
|
||||
} finally {
|
||||
await restarted?.shutdown();
|
||||
@@ -80279,9 +80372,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
expect(failure?.state).toBe("published");
|
||||
expect(answer?.providerMessageId).not.toBe(failure?.providerMessageId);
|
||||
expect(
|
||||
order === "answer_first"
|
||||
? answer?.providerMessageId
|
||||
: failure?.providerMessageId,
|
||||
failure?.providerMessageId,
|
||||
).toBe(queued!.providerMessageId);
|
||||
const runtime = context.runtime.endpoints.get(context.endpoint.id)!;
|
||||
expect(runtime.posts).toHaveLength(2);
|
||||
@@ -80666,7 +80757,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("does not let removed-input cleanup overwrite a pre-run FIFO lane already replaced by its answer", async () => {
|
||||
it("keeps a separate FIFO answer visible while removed-input cleanup edits its queued notice", async () => {
|
||||
const context = await deferredChatQueueFixture();
|
||||
try {
|
||||
await context.service.processPendingPublications();
|
||||
@@ -80692,9 +80783,10 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
0,
|
||||
);
|
||||
const runtime = context.runtime.endpoints.get(context.endpoint.id)!;
|
||||
expect(runtime.posts).toHaveLength(1);
|
||||
expect(runtime.posts).toHaveLength(2);
|
||||
expect(runtime.posts.at(-1)?.text).toBe("Already answered");
|
||||
expect(runtime.edits).toHaveLength(1);
|
||||
expect(runtime.edits[0]?.text).toBe("Already answered");
|
||||
expect(runtime.edits[0]?.text).toBe("This queued message was removed.");
|
||||
await expect(
|
||||
db
|
||||
.select({ state: chatPublications.state })
|
||||
@@ -80705,7 +80797,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
|
||||
like(chatPublications.idempotencyKey, "wake:%:removed:%"),
|
||||
),
|
||||
),
|
||||
).resolves.toEqual([{ state: "cancelled" }]);
|
||||
).resolves.toEqual([{ state: "published" }]);
|
||||
} finally {
|
||||
await context.service.shutdown();
|
||||
}
|
||||
|
||||
@@ -3,7 +3,7 @@ import { persistReprojectedWorkspaceDiffs, projectCodexWorkspaceDiffsFromTrace }
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq, like, and } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { chatPublications, createDb, heartbeatRunEvents, heartbeatRuns, issues, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
import { chatPublications, chatMessageLinks, createDb, heartbeatRunEvents, heartbeatRuns, issues, issueComments, issueThreadInteractions, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { createChatMilestoneFixtures } from "./helpers/chat-milestone-fixtures.js";
|
||||
@@ -14,6 +14,11 @@ import { appendHeartbeatRunEvent } from "../services/heartbeat-run-events.js";
|
||||
import { NativeRunCoordinatorStore } from "../services/native-runtime/native-run-coordinator-store.js";
|
||||
import { PaperclipControlPlanePort } from "../services/native-runtime/paperclip-control-plane-port.js";
|
||||
import type { PrpEvent } from "../vendor/paperclip-runner/index.js";
|
||||
import { createRunSecretRedactionRegistry } from "../services/run-secret-redaction.js";
|
||||
import { publicProgressEvent } from "../services/chat-channels/slack/native-progress.js";
|
||||
import { publicNativeChatProgressText } from "../services/safe-native-chat-progress.js";
|
||||
import { enqueueSlackProgressCompletion, slackProgressCompletionTarget } from "../services/chat-channels/slack/public-progress.js";
|
||||
import { REDACTED_EVENT_VALUE } from "../redaction.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_PROGRESS_TEST_DATABASE_URL");
|
||||
@@ -79,6 +84,135 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
|
||||
};
|
||||
}
|
||||
|
||||
async function commentary(text: string) {
|
||||
return appendHeartbeatRunEvent(db, { ...event(), payload: { prpEvent: {
|
||||
schema: "paperclip.prp.event.v1", eventType: "item.completed", runId,
|
||||
payload: { kind: "agentMessage", channel: "progress", text },
|
||||
} }, message: "PRIVATE event metadata" });
|
||||
}
|
||||
|
||||
it("coalesces public commentary through grace, cooldown and restart, then becomes quiet", async () => {
|
||||
await db.update(chatPublications).set({ createdAt: new Date() }).where(eq(chatPublications.issueId, f.issueId));
|
||||
await commentary("Checking retries.");
|
||||
await start();
|
||||
expect(await publications()).toHaveLength(0);
|
||||
expect(coordinator!.nextWakeAt()).not.toBeNull();
|
||||
await vi.advanceTimersByTimeAsync(11_000);
|
||||
await vi.waitFor(async () => expect(await publications()).toHaveLength(1));
|
||||
const [first] = await publications();
|
||||
expect(first.payload.text).toContain("Checking retries.");
|
||||
await db.update(chatPublications).set({ state: "published", publishedAt: new Date() }).where(eq(chatPublications.id, first.id));
|
||||
for (let index = 0; index < 20; index++) await commentary(`Found retry case ${index}. Checking recovery next.`);
|
||||
// A later coarse operation must not displace the useful public note.
|
||||
await appendHeartbeatRunEvent(db, event(false, "tool.execution.completed"));
|
||||
await coordinator!.stop(); await start();
|
||||
expect(await publications()).toHaveLength(1);
|
||||
expect(coordinator!.nextWakeAt()).not.toBeNull();
|
||||
await vi.advanceTimersByTimeAsync(31_000);
|
||||
await vi.waitFor(async () => expect(await publications()).toHaveLength(2));
|
||||
expect((await publications()).map(row => row.payload.text).join(" ")).toContain("Found retry case 19.");
|
||||
expect(JSON.stringify(await publications())).not.toContain("PRIVATE");
|
||||
await settle(); await quiet();
|
||||
// Repeated whitespace and another completed item must not create work.
|
||||
await commentary("Found retry case 19. Checking recovery next.");
|
||||
await settle(); expect(await publications()).toHaveLength(2); await quiet();
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
it.each(["progress", "commentary", "final", "summary", "unknown", "wrong-run", "wrong-schema", "wrong-kind"])(
|
||||
"keeps SQL selection and the public text boundary aligned for %s", async kind => {
|
||||
const payload = { prpEvent: { schema: kind === "wrong-schema" ? "other" : "paperclip.prp.event.v1",
|
||||
runId: kind === "wrong-run" ? randomUUID() : runId, eventType: "item.completed",
|
||||
payload: { kind: kind === "wrong-kind" ? "reasoning" : "agentMessage", channel: kind, text: "Checking recovery." },
|
||||
} };
|
||||
if (kind.startsWith("wrong-")) payload.prpEvent.payload.channel = "progress";
|
||||
await appendHeartbeatRunEvent(db, { ...event(), payload });
|
||||
const selected = await db.select({ payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents)
|
||||
.where(and(eq(heartbeatRunEvents.runId, runId), publicProgressEvent(heartbeatRunEvents)));
|
||||
const projected = publicNativeChatProgressText("item.completed", payload, runId);
|
||||
expect(selected.length > 0).toBe(projected !== null);
|
||||
expect(selected.length > 0).toBe(kind === "progress");
|
||||
},
|
||||
);
|
||||
|
||||
it("retires a generic progress link once and refuses a link consumed by another output", async () => {
|
||||
await db.update(chatPublications).set({ providerMessageId: "working" }).where(eq(chatPublications.issueId, f.issueId));
|
||||
const [base] = await db.select().from(chatPublications).where(eq(chatPublications.issueId, f.issueId));
|
||||
await db.insert(chatMessageLinks).values({ companyId: f.companyId, endpointId: f.endpointId,
|
||||
conversationId: f.conversationId, publicationId: base.id, providerMessageId: "working", direction: "outbound" });
|
||||
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
authorType: "agent", authorAgentId: f.agentId, createdByRunId: runId, body: "Answer" }).returning();
|
||||
const [answer] = await db.insert(chatPublications).values({ companyId: f.companyId, endpointId: f.endpointId,
|
||||
conversationId: f.conversationId, issueId: f.issueId, commentId: comment.id,
|
||||
idempotencyKey: "answer", payload: { text: "Answer" }, state: "published", providerMessageId: "answer" }).returning();
|
||||
await db.transaction(async tx => {
|
||||
await enqueueSlackProgressCompletion(tx, answer);
|
||||
await enqueueSlackProgressCompletion(tx, answer);
|
||||
});
|
||||
const cleanup = await db.select().from(chatPublications).where(like(chatPublications.idempotencyKey, `%:slack-progress-close:${base.id}`));
|
||||
expect(cleanup).toHaveLength(1);
|
||||
expect(cleanup[0].payload).toEqual({ text: "Response sent.", progressState: "completed" });
|
||||
expect(await slackProgressCompletionTarget(db, cleanup[0])).toBe("working");
|
||||
await db.update(chatMessageLinks).set({ publicationId: answer.id }).where(eq(chatMessageLinks.publicationId, base.id));
|
||||
expect(await slackProgressCompletionTarget(db, cleanup[0])).toBeNull();
|
||||
});
|
||||
|
||||
it("redacts registered run secrets before saving public progress", async () => {
|
||||
const secret = "ordinary-synthetic-value-without-credential-prefix";
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "a".repeat(64));
|
||||
try {
|
||||
await createRunSecretRedactionRegistry(db).register(f.companyId, runId, secret);
|
||||
await commentary(`Checked ${secret}. Verifying recovery.`);
|
||||
await start(); await settle();
|
||||
const saved = await publications();
|
||||
expect(saved).toHaveLength(1);
|
||||
expect(saved[0].payload.text).toContain(REDACTED_EVENT_VALUE);
|
||||
expect(JSON.stringify(saved)).not.toContain(secret);
|
||||
expect(errors).toEqual([]);
|
||||
} finally { vi.unstubAllEnvs(); }
|
||||
});
|
||||
|
||||
it("pages past unusable public notes to keep the latest useful commentary", async () => {
|
||||
await commentary("Found the retry cause. Checking recovery.");
|
||||
for (let index = 0; index < 51; index++) await commentary("x".repeat(450));
|
||||
await commentary("<thinking>PRIVATE</thinking>");
|
||||
await start(); await settle();
|
||||
expect(await publications()).toEqual([expect.objectContaining({ payload: expect.objectContaining({
|
||||
text: expect.stringContaining("Found the retry cause. Checking recovery."),
|
||||
}) })]);
|
||||
await quiet(); expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
it("uses a real lifecycle fallback when public notes are unusable", async () => {
|
||||
await commentary("x".repeat(450));
|
||||
await appendHeartbeatRunEvent(db, event(false, "tool.execution.completed"));
|
||||
await start(); await settle();
|
||||
expect(await publications()).toEqual([expect.objectContaining({ payload: expect.objectContaining({ text: "Maya is using tools…" }) })]);
|
||||
await quiet(); expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
it.each(["terminal", "question", "answer"])("cancels the cooldown deadline when %s takes over", async reason => {
|
||||
await commentary("Checking retries.");
|
||||
await start(); await settle();
|
||||
const [first] = await publications();
|
||||
await db.update(chatPublications).set({ state: "published", publishedAt: new Date() }).where(eq(chatPublications.id, first.id));
|
||||
await commentary("Found the cause. Checking recovery.");
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await vi.waitFor(() => expect(coordinator!.nextWakeAt()).not.toBeNull());
|
||||
if (reason === "terminal") await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, runId));
|
||||
if (reason === "question") await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
kind: "ask_user_questions", continuationPolicy: "wake_assignee", status: "pending", createdByAgentId: f.agentId, sourceRunId: runId,
|
||||
payload: { version: 1, questions: [] } });
|
||||
if (reason === "answer") {
|
||||
const [comment] = await db.insert(issueComments).values({ companyId: f.companyId, issueId: f.issueId,
|
||||
authorType: "agent", authorAgentId: f.agentId, createdByRunId: runId, body: "Done", metadata: { version: 1, sections: [], authorizationReason: "paperclip_runner_protocol" } }).returning();
|
||||
await db.insert(chatPublications).values({ companyId: f.companyId, endpointId: f.endpointId, conversationId: f.conversationId,
|
||||
issueId: f.issueId, commentId: comment.id, idempotencyKey: `answer:${runId}`, payload: { text: "Done" } });
|
||||
}
|
||||
await vi.advanceTimersByTimeAsync(31_000); await settle();
|
||||
expect(await publications()).toHaveLength(1); await quiet(); expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
it.each([false, true])("wakes from a real committed event after idle (native source: %s)", async nativeSource => {
|
||||
await start(); await settle(); await quiet(); expect(await publications()).toEqual([]);
|
||||
const input = event(nativeSource);
|
||||
@@ -116,7 +250,7 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu
|
||||
// Model a base transaction that began before the event but committed
|
||||
// afterwards, without a wall-clock delay in the suite. PostgreSQL's
|
||||
// default now() is transaction-start time, not commit time.
|
||||
await tx.update(chatPublications).set({ createdAt: new Date(Date.now() - 300_000) })
|
||||
await tx.update(chatPublications).set({ createdAt: new Date(Date.now() - 300_000), state: "published" })
|
||||
.where(eq(chatPublications.idempotencyKey, `run:${runId}:working:${f.endpointId}`));
|
||||
await vi.advanceTimersByTimeAsync(1000); expect(await publications()).toEqual([]);
|
||||
});
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
export const CONTEXTUAL_FAST_RESPONSE_ACKNOWLEDGEMENT = "Acknowledged by contextual fast response";
|
||||
@@ -1,3 +1,4 @@
|
||||
import { slackPublicProgressReceiptToReplace, slackPublicProgressSupersessionReason, isSlackPublicProgress, slackProgressCompletionTarget } from "./slack/public-progress.js";
|
||||
import { imessageProgressSupersessionReason } from "./imessage-photon/publications.js";
|
||||
import { fastResponseRequests } from "@paperclipai/db";
|
||||
|
||||
@@ -74,6 +75,8 @@ export function createChatPublicationReplacement(db: Db, options: ChatPublicatio
|
||||
async function runOwnershipMilestoneSupersessionReason(
|
||||
publication: typeof chatPublications.$inferSelect,
|
||||
): Promise<string | null> {
|
||||
const completion = await slackProgressCompletionTarget(db, publication);
|
||||
if (completion === null) return "Slack progress message is no longer replaceable";
|
||||
const progress = publication.payload.progressState;
|
||||
const typingReason = imessageProgressSupersessionReason(progress);
|
||||
if (typingReason) {
|
||||
@@ -121,6 +124,10 @@ export function createChatPublicationReplacement(db: Db, options: ChatPublicatio
|
||||
.for("update")
|
||||
.then((rows) => rows[0] ?? null);
|
||||
});
|
||||
if ((await endpointRecord(publication.endpointId))?.endpoint.provider === "slack") {
|
||||
const reason = await slackPublicProgressSupersessionReason(db, publication, runId);
|
||||
if (reason) return reason;
|
||||
}
|
||||
const baseProgressKey = `run:${runId}:${progress}:${publication.endpointId}`;
|
||||
const nativeProgressPrefix = `run:${runId}:working:${publication.endpointId}:native:`;
|
||||
const isTerminalizableProgress =
|
||||
@@ -335,6 +342,8 @@ export function createChatPublicationReplacement(db: Db, options: ChatPublicatio
|
||||
.then((rows) => rows[0]?.runId ?? null)
|
||||
: null);
|
||||
if (!currentRunId) return null;
|
||||
const completion = await slackProgressCompletionTarget(db, publication);
|
||||
if (completion !== undefined) return completion;
|
||||
return db
|
||||
.select({
|
||||
id: chatPublications.id,
|
||||
@@ -377,7 +386,12 @@ export function createChatPublicationReplacement(db: Db, options: ChatPublicatio
|
||||
Boolean(row.providerMessageId) &&
|
||||
row.payload.progressState !== undefined,
|
||||
);
|
||||
if (!replacement?.providerMessageId) return null;
|
||||
if (!replacement?.providerMessageId) {
|
||||
if ((await endpointRecord(publication.endpointId))?.endpoint.provider === "slack") {
|
||||
return slackPublicProgressReceiptToReplace(db, publication, currentRunId);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
if (
|
||||
(await providerProgressLaneConsumed(
|
||||
publication,
|
||||
@@ -675,6 +689,7 @@ export function createChatPublicationReplacement(db: Db, options: ChatPublicatio
|
||||
|
||||
async function progressSupersededByFastResponse(publication: typeof chatPublications.$inferSelect): Promise<boolean> {
|
||||
if (!["queued", "working"].includes(publication.payload.progressState ?? "")) return false;
|
||||
if (isSlackPublicProgress(publication)) return false;
|
||||
const runId = runIdFromMilestonePublication(publication);
|
||||
const [run] = runId ? await db.select({ context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, publication.companyId), eq(heartbeatRuns.id, runId))) : [];
|
||||
const sourceCommentId = typeof run?.context?.wakeCommentId === "string" ? run.context.wakeCommentId : typeof run?.context?.commentId === "string" ? run.context.commentId : publication.commentId;
|
||||
|
||||
@@ -1,3 +1,6 @@
|
||||
import { CONTEXTUAL_FAST_RESPONSE_ACKNOWLEDGEMENT } from "../progress-reasons.js";
|
||||
import { slackAnswerNeedsFreshReply, enqueueSlackProgressCompletion } from "../slack/public-progress.js";
|
||||
import { notifyChatNativeProgress } from "../../chat-native-progress-notifications.js";
|
||||
import type { ChatPublicationPreparationWait } from "./preparation.js";
|
||||
import { notifyChatResponseAvailability } from "../../chat-response-availability.js";
|
||||
import { notifyVoicePublication } from "../../voice/voice-reply-notifications.js";
|
||||
@@ -640,7 +643,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
|
||||
// because it now occupies the next FIFO slot.
|
||||
const progressSupersessionReason =
|
||||
(await runOwnershipMilestoneSupersessionReason(publication)) ??
|
||||
((await progressSupersededByFastResponse(publication)) ? "Acknowledged by contextual fast response" : null) ??
|
||||
((await progressSupersededByFastResponse(publication)) ? CONTEXTUAL_FAST_RESPONSE_ACKNOWLEDGEMENT : null) ??
|
||||
((await runProgressSupersededByPublishedInteraction(publication))
|
||||
? "Run progress was superseded by its provider interaction"
|
||||
: null);
|
||||
@@ -648,6 +651,9 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
|
||||
await publicationLease.assertOwned();
|
||||
await db.transaction(async (tx) => {
|
||||
await notifyChatPublicationWork(tx, publication);
|
||||
if (progressSupersessionReason === CONTEXTUAL_FAST_RESPONSE_ACKNOWLEDGEMENT) {
|
||||
await notifyChatNativeProgress(tx, { companyId: publication.companyId, issueId: publication.issueId });
|
||||
}
|
||||
return tx
|
||||
.update(chatPublications)
|
||||
.set({
|
||||
@@ -691,7 +697,8 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
|
||||
publication,
|
||||
endpoint,
|
||||
);
|
||||
const replaceProviderMessageId = endpoint.provider !== "imessage-photon" && CAPABILITIES[endpoint.provider]
|
||||
const freshSlackAnswer = endpoint.provider === "slack" && slackAnswerNeedsFreshReply(publication, payload);
|
||||
const replaceProviderMessageId = !freshSlackAnswer && endpoint.provider !== "imessage-photon" && CAPABILITIES[endpoint.provider]
|
||||
.messageEdits
|
||||
? (closedProgress?.providerMessageId ??
|
||||
(await interactionResolutionPublicationToReplace(
|
||||
@@ -881,6 +888,9 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
|
||||
await credentialLease.assertOwned(tx);
|
||||
const committedAt = new Date();
|
||||
await notifyChatPublicationWork(tx, publication);
|
||||
if (publication.payload.progressState === "working") {
|
||||
await notifyChatNativeProgress(tx, { companyId: publication.companyId, issueId: publication.issueId });
|
||||
}
|
||||
await notifyVoicePublication(tx, authorizationClaim.endpoint, publication.conversationId);
|
||||
const [completedPublication] = await tx
|
||||
.update(chatPublications)
|
||||
@@ -905,6 +915,7 @@ export function createChatPublicationWorker(db: Db, options: PublicationWorkerOp
|
||||
"Chat publication ownership changed before commit",
|
||||
);
|
||||
}
|
||||
if (freshSlackAnswer) await enqueueSlackProgressCompletion(tx, publication);
|
||||
await stageSlackForm?.(tx);
|
||||
if (telegramDraft) {
|
||||
const completedDraft = await tx
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
import { and, desc, eq, lt, sql, type SQLWrapper } from "drizzle-orm";
|
||||
import { heartbeatRunEvents, type Db } from "@paperclipai/db";
|
||||
import { publicNativeChatProgressText } from "../../safe-native-chat-progress.js";
|
||||
import { createRunSecretRedactionRegistry } from "../../run-secret-redaction.js";
|
||||
import type { NativeProgressPolicy } from "../../chat-native-progress-policy.js";
|
||||
import { CONTEXTUAL_FAST_RESPONSE_ACKNOWLEDGEMENT } from "../progress-reasons.js";
|
||||
|
||||
export function publicProgressEvent(event: { eventType: SQLWrapper; payload: SQLWrapper; runId: SQLWrapper }) {
|
||||
return sql<boolean>`coalesce((${event.eventType} = 'item.completed'
|
||||
and ${event.payload}->'prpEvent'->>'schema' = 'paperclip.prp.event.v1'
|
||||
and ${event.payload}->'prpEvent'->>'eventType' = 'item.completed'
|
||||
and ${event.payload}->'prpEvent'->>'runId' = ${event.runId}::text
|
||||
and ${event.payload}->'prpEvent'->'payload'->>'kind' = 'agentMessage'
|
||||
and ${event.payload}->'prpEvent'->'payload'->>'channel' = 'progress'
|
||||
and jsonb_typeof(${event.payload}->'prpEvent'->'payload'->'text') = 'string'), false)`;
|
||||
}
|
||||
|
||||
async function latestPublicProgress(db: Pick<Db, "select">, scope: { companyId: string; runId: string }, signal?: AbortSignal) {
|
||||
const redact = await createRunSecretRedactionRegistry(db as Db).redactorForRun(scope.companyId, scope.runId);
|
||||
let beforeSeq: number | undefined;
|
||||
for (;;) {
|
||||
signal?.throwIfAborted();
|
||||
const events = await db.select({ seq: heartbeatRunEvents.seq, payload: heartbeatRunEvents.payload })
|
||||
.from(heartbeatRunEvents)
|
||||
.where(and(eq(heartbeatRunEvents.companyId, scope.companyId), eq(heartbeatRunEvents.runId, scope.runId),
|
||||
publicProgressEvent(heartbeatRunEvents), beforeSeq === undefined ? undefined : lt(heartbeatRunEvents.seq, beforeSeq)))
|
||||
.orderBy(desc(heartbeatRunEvents.seq)).limit(50);
|
||||
for (const event of events) {
|
||||
const text = publicNativeChatProgressText("item.completed", redact(event.payload), scope.runId);
|
||||
if (text) return { seq: event.seq, text };
|
||||
}
|
||||
if (events.length < 50) return null;
|
||||
beforeSeq = events[events.length - 1].seq;
|
||||
}
|
||||
}
|
||||
|
||||
export const slackNativeProgressPolicy = {
|
||||
latestPublicNote: latestPublicProgress,
|
||||
additionalBaseEligibility: (base: { state: SQLWrapper; redactedError: SQLWrapper }) =>
|
||||
and(eq(base.state, "cancelled"), eq(base.redactedError, CONTEXTUAL_FAST_RESPONSE_ACKNOWLEDGEMENT)),
|
||||
acceptsBase: (base, hasPublicNote) => ["pending", "retry", "streaming", "published"].includes(base.state) ||
|
||||
(hasPublicNote && base.state === "cancelled" && base.redactedError === CONTEXTUAL_FAST_RESPONSE_ACKNOWLEDGEMENT),
|
||||
nextDelivery: ({ base, previous, now }) => {
|
||||
if (previous.some(row => ["pending", "retry", "streaming", "delivery_unknown"].includes(row.state)) ||
|
||||
!["published", "cancelled"].includes(base.state)) return { ready: false };
|
||||
const lastProgressAt = Math.max(base.createdAt.getTime(), ...previous.map(row =>
|
||||
Math.max(row.createdAt.getTime(), row.publishedAt?.getTime() ?? 0)));
|
||||
const dueAt = previous.length ? lastProgressAt + 30_000 : base.createdAt.getTime() + 10_000;
|
||||
return now < dueAt ? { ready: false, dueAt } : { ready: true };
|
||||
},
|
||||
} satisfies NativeProgressPolicy & { additionalBaseEligibility: (base: { state: SQLWrapper; redactedError: SQLWrapper }) => SQLWrapper | undefined };
|
||||
@@ -0,0 +1,107 @@
|
||||
import type { DbOrTransaction } from "../runtime-types.js";
|
||||
import { hasChatRunOwnedProviderInteraction } from "../../chat-interaction-arbitration.js";
|
||||
import { notifyChatPublicationWork } from "../../chat-work-notifications.js";
|
||||
import type { SafeChatPublicationPayload } from "@paperclipai/shared";
|
||||
import { and, eq, inArray, isNull, sql } from "drizzle-orm";
|
||||
import { chatMessageLinks, chatPublications, fastResponseRequests, heartbeatRuns, issueComments, type Db } from "@paperclipai/db";
|
||||
|
||||
export async function slackPublicProgressReceiptToReplace(
|
||||
db: Db,
|
||||
publication: typeof chatPublications.$inferSelect,
|
||||
runId: string,
|
||||
): Promise<string | null> {
|
||||
if (!publication.idempotencyKey.includes(":native:public:")) return null;
|
||||
const [receipt] = await db.select({ providerMessageId: chatPublications.providerMessageId })
|
||||
.from(fastResponseRequests)
|
||||
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.companyId, publication.companyId),
|
||||
sql`${fastResponseRequests.sourceCommentId}::text = coalesce(${heartbeatRuns.contextSnapshot}->>'wakeCommentId', ${heartbeatRuns.contextSnapshot}->>'commentId')`))
|
||||
.innerJoin(chatPublications, and(eq(chatPublications.companyId, publication.companyId),
|
||||
eq(chatPublications.commentId, fastResponseRequests.commentId), eq(chatPublications.endpointId, publication.endpointId),
|
||||
eq(chatPublications.conversationId, publication.conversationId), eq(chatPublications.issueId, publication.issueId), eq(chatPublications.state, "published")))
|
||||
.innerJoin(chatMessageLinks, and(eq(chatMessageLinks.publicationId, chatPublications.id),
|
||||
eq(chatMessageLinks.companyId, publication.companyId), eq(chatMessageLinks.endpointId, publication.endpointId),
|
||||
eq(chatMessageLinks.conversationId, publication.conversationId), eq(chatMessageLinks.direction, "outbound"),
|
||||
eq(chatMessageLinks.providerMessageId, chatPublications.providerMessageId)))
|
||||
.where(and(eq(fastResponseRequests.companyId, publication.companyId), eq(fastResponseRequests.endpointId, publication.endpointId),
|
||||
eq(fastResponseRequests.conversationId, publication.conversationId)))
|
||||
.limit(1);
|
||||
return receipt?.providerMessageId ?? null;
|
||||
}
|
||||
|
||||
export function slackAnswerNeedsFreshReply(
|
||||
publication: Pick<typeof chatPublications.$inferSelect, "commentId">,
|
||||
payload: SafeChatPublicationPayload,
|
||||
): boolean {
|
||||
return Boolean(publication.commentId) && payload.progressState === undefined;
|
||||
}
|
||||
|
||||
export function isSlackPublicProgress(publication: Pick<typeof chatPublications.$inferSelect, "idempotencyKey">) {
|
||||
return publication.idempotencyKey.includes(":native:public:");
|
||||
}
|
||||
|
||||
export async function slackPublicProgressSupersessionReason(
|
||||
db: Db, publication: typeof chatPublications.$inferSelect, runId: string,
|
||||
): Promise<string | null> {
|
||||
if (!isSlackPublicProgress(publication)) return null;
|
||||
if (await hasChatRunOwnedProviderInteraction(db, { companyId: publication.companyId, issueId: publication.issueId, runId })) {
|
||||
return "Run progress was superseded by its provider interaction";
|
||||
}
|
||||
const [answer] = await db.select({ id: chatPublications.id }).from(chatPublications)
|
||||
.innerJoin(issueComments, and(eq(issueComments.id, chatPublications.commentId), eq(issueComments.companyId, publication.companyId)))
|
||||
.where(and(eq(chatPublications.companyId, publication.companyId), eq(chatPublications.endpointId, publication.endpointId),
|
||||
eq(chatPublications.conversationId, publication.conversationId), eq(chatPublications.issueId, publication.issueId),
|
||||
eq(issueComments.createdByRunId, runId), eq(issueComments.authorType, "agent"),
|
||||
sql`${chatPublications.payload}->>'progressState' is null`,
|
||||
inArray(chatPublications.state, ["pending", "retry", "streaming", "published", "delivery_unknown"]))).limit(1);
|
||||
return answer ? "Run progress was superseded by its authored answer" : null;
|
||||
}
|
||||
|
||||
const completionMarker = ":slack-progress-close:";
|
||||
|
||||
export async function slackProgressCompletionTarget(
|
||||
db: Db, publication: typeof chatPublications.$inferSelect,
|
||||
): Promise<string | null | undefined> {
|
||||
if (!publication.idempotencyKey.includes(completionMarker)) return undefined;
|
||||
const targetId = publication.idempotencyKey.split(completionMarker)[1];
|
||||
const [target] = await db.select({ providerMessageId: chatPublications.providerMessageId })
|
||||
.from(chatPublications)
|
||||
.innerJoin(chatMessageLinks, and(eq(chatMessageLinks.publicationId, chatPublications.id),
|
||||
eq(chatMessageLinks.companyId, publication.companyId), eq(chatMessageLinks.endpointId, publication.endpointId),
|
||||
eq(chatMessageLinks.conversationId, publication.conversationId), eq(chatMessageLinks.direction, "outbound"),
|
||||
eq(chatMessageLinks.providerMessageId, chatPublications.providerMessageId)))
|
||||
.where(and(sql`${chatPublications.id}::text = ${targetId}`, eq(chatPublications.companyId, publication.companyId),
|
||||
eq(chatPublications.endpointId, publication.endpointId), eq(chatPublications.conversationId, publication.conversationId),
|
||||
eq(chatPublications.issueId, publication.issueId), eq(chatPublications.state, "published"), isNull(chatPublications.commentId),
|
||||
sql`starts_with(${chatPublications.idempotencyKey}, ${`run:${publication.idempotencyKey.split(":")[1]}:working:${publication.endpointId}`})`,
|
||||
sql`${chatPublications.payload}->>'progressState' in ('queued', 'working')`)).limit(1);
|
||||
return target?.providerMessageId ?? null;
|
||||
}
|
||||
|
||||
export async function enqueueSlackProgressCompletion(db: DbOrTransaction, answer: typeof chatPublications.$inferSelect) {
|
||||
if (!answer.commentId) return;
|
||||
const [comment] = await db.select({ runId: issueComments.createdByRunId }).from(issueComments)
|
||||
.where(and(eq(issueComments.id, answer.commentId), eq(issueComments.companyId, answer.companyId),
|
||||
eq(issueComments.issueId, answer.issueId), eq(issueComments.authorType, "agent")));
|
||||
if (!comment?.runId) return;
|
||||
const baseKey = `run:${comment.runId}:working:${answer.endpointId}`;
|
||||
const targets = await db.select({ id: chatPublications.id, key: chatPublications.idempotencyKey })
|
||||
.from(chatPublications)
|
||||
.innerJoin(chatMessageLinks, and(eq(chatMessageLinks.publicationId, chatPublications.id),
|
||||
eq(chatMessageLinks.companyId, answer.companyId), eq(chatMessageLinks.endpointId, answer.endpointId),
|
||||
eq(chatMessageLinks.conversationId, answer.conversationId), eq(chatMessageLinks.direction, "outbound"),
|
||||
eq(chatMessageLinks.providerMessageId, chatPublications.providerMessageId)))
|
||||
.where(and(eq(chatPublications.companyId, answer.companyId), eq(chatPublications.endpointId, answer.endpointId),
|
||||
eq(chatPublications.conversationId, answer.conversationId), eq(chatPublications.issueId, answer.issueId),
|
||||
eq(chatPublications.state, "published"), isNull(chatPublications.commentId),
|
||||
sql`${chatPublications.payload}->>'progressState' in ('queued', 'working')`,
|
||||
sql`(${chatPublications.idempotencyKey} = ${baseKey} or starts_with(${chatPublications.idempotencyKey}, ${`${baseKey}:native:`}))`));
|
||||
for (const target of targets) {
|
||||
if (isSlackPublicProgress({ idempotencyKey: target.key })) continue;
|
||||
await notifyChatPublicationWork(db, answer);
|
||||
await db.insert(chatPublications).values({ companyId: answer.companyId, endpointId: answer.endpointId,
|
||||
conversationId: answer.conversationId, issueId: answer.issueId,
|
||||
idempotencyKey: `run:${comment.runId}:completed:${answer.endpointId}${completionMarker}${target.id}`,
|
||||
payload: { text: "Response sent.", progressState: "completed" }, state: "pending",
|
||||
}).onConflictDoNothing();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
import { and, eq, inArray, ne, or, sql, type SQLWrapper } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { slackNativeProgressPolicy } from "./chat-channels/slack/native-progress.js";
|
||||
|
||||
export type NativeProgressBase = { createdAt: Date; state: string; redactedError: string | null };
|
||||
export type NativeProgressHistory = { createdAt: Date; publishedAt: Date | null; state: string };
|
||||
export type NativeProgressPolicy = {
|
||||
latestPublicNote?: (db: Pick<Db, "select">, scope: { companyId: string; runId: string }, signal?: AbortSignal) => Promise<{ seq: number; text: string } | null>;
|
||||
acceptsBase: (base: NativeProgressBase, hasPublicNote: boolean) => boolean;
|
||||
nextDelivery: (input: { base: NativeProgressBase; previous: NativeProgressHistory[]; eventCreatedAt: Date; now: number }) => { ready: boolean; dueAt?: number };
|
||||
};
|
||||
|
||||
const defaultPolicy: NativeProgressPolicy = {
|
||||
acceptsBase: base => ["pending", "retry", "streaming", "published"].includes(base.state),
|
||||
nextDelivery: ({ base, previous, eventCreatedAt }) => ({
|
||||
ready: eventCreatedAt.getTime() - Math.max(base.createdAt.getTime(), ...previous.map(row => row.createdAt.getTime())) >= 20_000,
|
||||
}),
|
||||
};
|
||||
|
||||
export function nativeChatProgressPolicy(provider: string): NativeProgressPolicy {
|
||||
return provider === "slack" ? slackNativeProgressPolicy : defaultPolicy;
|
||||
}
|
||||
|
||||
export function nativeProgressBaseEligible(provider: SQLWrapper, base: { state: SQLWrapper; redactedError: SQLWrapper }) {
|
||||
return or(inArray(base.state, ["pending", "retry", "streaming", "published"]),
|
||||
and(eq(provider, "slack"), slackNativeProgressPolicy.additionalBaseEligibility(base)));
|
||||
}
|
||||
|
||||
export function nativeProgressEventEligible(provider: SQLWrapper, eventCreatedAt: SQLWrapper, baseCreatedAt: SQLWrapper) {
|
||||
return or(eq(provider, "slack"), and(ne(provider, "slack"), sql`${eventCreatedAt} >= ${baseCreatedAt} + interval '20 seconds'`));
|
||||
}
|
||||
@@ -19,6 +19,7 @@ export function nativeChatProgressWork(db: Db) {
|
||||
scope: { companyId: entry.companyId, issueId: entry.issueId }, limit: batchSize, signal,
|
||||
});
|
||||
if (result.inserted === batchSize) return { ...entry, dueAt: Date.now() };
|
||||
if (result.nextDueAt !== null) return { ...entry, dueAt: result.contended ? Math.min(result.nextDueAt, Date.now() + 1000) : result.nextDueAt };
|
||||
return result.contended ? { ...entry, dueAt: Date.now() + 1000 } : null;
|
||||
}
|
||||
function register(coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { and, asc, desc, eq, gt, gte, inArray, like, notExists, sql, type Param, type Placeholder } from "drizzle-orm";
|
||||
import { and, asc, desc, eq, gt, gte, inArray, like, notExists, or, sql, type Param, type Placeholder } from "drizzle-orm";
|
||||
import { alias } from "drizzle-orm/pg-core";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { agents, chatConversations, chatEndpoints, chatPublications, heartbeatRunEvents, heartbeatRuns, issueComments, issues } from "@paperclipai/db";
|
||||
@@ -11,6 +11,10 @@ import { allowsAutomaticRunPublication, chatRunProjectionWindow, type ChatRunPro
|
||||
import { SAFE_NATIVE_CHAT_PROGRESS_EVENT_TYPES, safeNativeChatProgressForEvent } from "./safe-native-chat-progress.js";
|
||||
import { columnPlaceholder, preparedStatements } from "./prepared-statements.js";
|
||||
|
||||
import { runSecretRedactionVersionSql } from "./run-secret-redaction.js";
|
||||
import { nativeChatProgressPolicy, nativeProgressBaseEligible, nativeProgressEventEligible } from "./chat-native-progress-policy.js";
|
||||
import { publicChatTaskUrl } from "./chat-task-url.js";
|
||||
|
||||
type SafeNativeChatProgressCandidate = {
|
||||
eventId: number;
|
||||
eventSeq: number;
|
||||
@@ -23,10 +27,9 @@ type SafeNativeChatProgressCandidate = {
|
||||
endpointId: string;
|
||||
conversationId: string;
|
||||
agentName: string;
|
||||
provider: string;
|
||||
};
|
||||
|
||||
const SAFE_NATIVE_CHAT_PROGRESS_CADENCE_MS = 20_000;
|
||||
|
||||
type NativeProgressCursor = Pick<SafeNativeChatProgressCandidate, "eventCreatedAt" | "runId" | "conversationId">;
|
||||
type NativeProgressInput = { since?: Date; limit?: number; scope?: ChatRunProjectionScope; signal?: AbortSignal };
|
||||
|
||||
@@ -64,6 +67,7 @@ export function nativeProgressPageQuery(db: Db, bind: NativeProgressPageBinding)
|
||||
endpointId: chatConversations.endpointId,
|
||||
conversationId: chatConversations.id,
|
||||
agentName: agents.name,
|
||||
provider: chatEndpoints.provider,
|
||||
})
|
||||
.from(heartbeatRunEvents)
|
||||
.innerJoin(
|
||||
@@ -103,12 +107,7 @@ export function nativeProgressPageQuery(db: Db, bind: NativeProgressPageBinding)
|
||||
eq(basePublication.conversationId, chatConversations.id),
|
||||
eq(basePublication.issueId, chatConversations.issueId),
|
||||
sql`${basePublication.idempotencyKey} = 'run:' || ${heartbeatRuns.id}::text || ':working:' || ${chatConversations.endpointId}::text`,
|
||||
inArray(basePublication.state, [
|
||||
"pending",
|
||||
"retry",
|
||||
"streaming",
|
||||
"published",
|
||||
]),
|
||||
nativeProgressBaseEligible(chatEndpoints.provider, basePublication),
|
||||
),
|
||||
)
|
||||
.where(
|
||||
@@ -119,7 +118,7 @@ export function nativeProgressPageQuery(db: Db, bind: NativeProgressPageBinding)
|
||||
gte(heartbeatRunEvents.createdAt, bind.since),
|
||||
bind.companyId !== undefined ? eq(heartbeatRuns.companyId, bind.companyId) : undefined,
|
||||
bind.issueId !== undefined ? eq(chatConversations.issueId, bind.issueId) : undefined,
|
||||
sql`${heartbeatRunEvents.createdAt} >= ${basePublication.createdAt} + interval '20 seconds'`,
|
||||
nativeProgressEventEligible(chatEndpoints.provider, heartbeatRunEvents.createdAt, basePublication.createdAt),
|
||||
notExists(
|
||||
db
|
||||
.select({ id: laterEvent.id })
|
||||
@@ -195,11 +194,13 @@ export async function loadNativeChatProgressScopes(db: Db, scope?: ChatRunProjec
|
||||
}
|
||||
}
|
||||
|
||||
/** A suppressed event is settled; lock contention still owes another attempt. */
|
||||
export async function reconcileNativeChatProgress(db: Db, options: NativeProgressInput = {}) {
|
||||
const input = chatRunProjectionWindow(options);
|
||||
const pages = nativeProgressPages(db, input.since, options.scope);
|
||||
const publicNotes = new Map<string, { seq: number; text: string } | null>();
|
||||
let inserted = 0, contended = false;
|
||||
let nextDueAt: number | null = null;
|
||||
const defer = (dueAt: number) => { nextDueAt = nextDueAt === null ? dueAt : Math.min(nextDueAt, dueAt); };
|
||||
let cursor: NativeProgressCursor | null = null;
|
||||
while (inserted < input.limit) {
|
||||
options.signal?.throwIfAborted();
|
||||
@@ -243,7 +244,9 @@ export async function reconcileNativeChatProgress(db: Db, options: NativeProgres
|
||||
// short lock makes terminal status, a later event, or a final comment
|
||||
// win before this non-terminal projection can be inserted.
|
||||
const currentRun = await tx
|
||||
.select({ id: heartbeatRuns.id })
|
||||
.select({ id: heartbeatRuns.id,
|
||||
redactionVersion: runSecretRedactionVersionSql(),
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.where(
|
||||
and(
|
||||
@@ -319,7 +322,14 @@ export async function reconcileNativeChatProgress(db: Db, options: NativeProgres
|
||||
) {
|
||||
return 0;
|
||||
}
|
||||
const progress = safeNativeChatProgressForEvent(
|
||||
const policy = nativeChatProgressPolicy(row.provider);
|
||||
const noteKey = `${row.companyId}:${row.runId}:${currentEvent.seq}:${currentRun.redactionVersion}`;
|
||||
if (policy.latestPublicNote && !publicNotes.has(noteKey)) {
|
||||
publicNotes.set(noteKey, await policy.latestPublicNote(tx, row, options.signal));
|
||||
}
|
||||
const publicNote = policy.latestPublicNote ? publicNotes.get(noteKey) ?? null : null;
|
||||
const publicText = publicNote?.text ?? null;
|
||||
const progress = publicText ? { phase: "public", text: publicText } : safeNativeChatProgressForEvent(
|
||||
currentEvent.eventType,
|
||||
row.agentName,
|
||||
);
|
||||
@@ -327,7 +337,7 @@ export async function reconcileNativeChatProgress(db: Db, options: NativeProgres
|
||||
|
||||
const baseKey = `run:${row.runId}:working:${row.endpointId}`;
|
||||
const [currentBase] = await tx
|
||||
.select({ createdAt: chatPublications.createdAt })
|
||||
.select({ createdAt: chatPublications.createdAt, state: chatPublications.state, redactedError: chatPublications.redactedError })
|
||||
.from(chatPublications)
|
||||
.where(
|
||||
and(
|
||||
@@ -336,22 +346,19 @@ export async function reconcileNativeChatProgress(db: Db, options: NativeProgres
|
||||
eq(chatPublications.conversationId, row.conversationId),
|
||||
eq(chatPublications.issueId, row.issueId),
|
||||
eq(chatPublications.idempotencyKey, baseKey),
|
||||
inArray(chatPublications.state, [
|
||||
"pending",
|
||||
"retry",
|
||||
"streaming",
|
||||
"published",
|
||||
]),
|
||||
),
|
||||
)
|
||||
.limit(1);
|
||||
if (!currentBase) return 0;
|
||||
if (!currentBase || !policy.acceptsBase(currentBase, publicText !== null)) return 0;
|
||||
|
||||
const progressKeyPrefix = `${baseKey}:native:`;
|
||||
const previousProgress = await tx
|
||||
.select({
|
||||
createdAt: chatPublications.createdAt,
|
||||
idempotencyKey: chatPublications.idempotencyKey,
|
||||
payload: chatPublications.payload,
|
||||
state: chatPublications.state,
|
||||
publishedAt: chatPublications.publishedAt,
|
||||
})
|
||||
.from(chatPublications)
|
||||
.where(
|
||||
@@ -363,28 +370,9 @@ export async function reconcileNativeChatProgress(db: Db, options: NativeProgres
|
||||
like(chatPublications.idempotencyKey, `${progressKeyPrefix}%`),
|
||||
),
|
||||
);
|
||||
if (
|
||||
previousProgress.some((publication) =>
|
||||
publication.idempotencyKey.startsWith(
|
||||
`${progressKeyPrefix}${progress.phase}:`,
|
||||
),
|
||||
)
|
||||
) {
|
||||
return 0;
|
||||
}
|
||||
const lastProgressAt = Math.max(
|
||||
currentBase.createdAt.getTime(),
|
||||
...previousProgress.map((publication) =>
|
||||
publication.createdAt.getTime(),
|
||||
),
|
||||
);
|
||||
if (
|
||||
currentEvent.createdAt.getTime() - lastProgressAt <
|
||||
SAFE_NATIVE_CHAT_PROGRESS_CADENCE_MS
|
||||
) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (publicText) {
|
||||
if (previousProgress.some(publication => publication.payload.text.split("\n\n")[0] === publicText)) return 0;
|
||||
} else if (previousProgress.some(publication => publication.idempotencyKey.startsWith(`${progressKeyPrefix}${progress.phase}:`))) return 0;
|
||||
const bindings = await resolveChatOriginPublicationBindings(
|
||||
tx,
|
||||
row.companyId,
|
||||
@@ -436,6 +424,12 @@ export async function reconcileNativeChatProgress(db: Db, options: NativeProgres
|
||||
.limit(1);
|
||||
if (explicitlyAuthoredFinal.length > 0) return 0;
|
||||
|
||||
const delivery = policy.nextDelivery({ base: currentBase, previous: previousProgress,
|
||||
eventCreatedAt: currentEvent.createdAt, now: Date.now() });
|
||||
if (delivery.dueAt !== undefined) defer(delivery.dueAt);
|
||||
if (!delivery.ready) return 0;
|
||||
|
||||
const taskUrl = publicText ? publicChatTaskUrl(row.issueId) : null;
|
||||
await notifyChatPublicationWork(tx);
|
||||
const insertedRows = await tx
|
||||
.insert(chatPublications)
|
||||
@@ -444,14 +438,16 @@ export async function reconcileNativeChatProgress(db: Db, options: NativeProgres
|
||||
endpointId: row.endpointId,
|
||||
conversationId: row.conversationId,
|
||||
issueId: row.issueId,
|
||||
idempotencyKey: `${progressKeyPrefix}${progress.phase}:${currentEvent.seq}`,
|
||||
idempotencyKey: `${progressKeyPrefix}${progress.phase}:${publicNote?.seq ?? currentEvent.seq}`,
|
||||
payload: projectSafeChatPublication({
|
||||
classification: "external",
|
||||
source: "safe_milestone",
|
||||
text: progress.text,
|
||||
text: publicText && taskUrl
|
||||
? `${publicText}\n\n[View task](${taskUrl})` : progress.text,
|
||||
progressState: "working",
|
||||
}),
|
||||
state: "pending",
|
||||
createdAt: new Date(),
|
||||
})
|
||||
.onConflictDoNothing()
|
||||
.returning({ id: chatPublications.id });
|
||||
@@ -461,10 +457,9 @@ export async function reconcileNativeChatProgress(db: Db, options: NativeProgres
|
||||
}
|
||||
if (rows.length < pageSize) break;
|
||||
}
|
||||
return { inserted, contended };
|
||||
return { inserted, contended, nextDueAt };
|
||||
}
|
||||
|
||||
/** Explicit/manual callers retain the original count-only contract. */
|
||||
export async function enqueueSafeNativeChatProgress(db: Db, options: NativeProgressInput = {}): Promise<number> {
|
||||
return (await reconcileNativeChatProgress(db, options)).inserted;
|
||||
}
|
||||
@@ -25,6 +25,10 @@ const REGISTRY_KEY = "paperclipSecretRedactions";
|
||||
// Project only the registry: run contexts can contain megabytes of prompt data.
|
||||
const registrySnapshot = sql`jsonb_build_object('paperclipSecretRedactions', ${heartbeatRuns.contextSnapshot} -> 'paperclipSecretRedactions')`;
|
||||
|
||||
export function runSecretRedactionVersionSql() {
|
||||
return sql<string>`md5(coalesce((${heartbeatRuns.contextSnapshot}->${REGISTRY_KEY})::text, ''))`;
|
||||
}
|
||||
|
||||
type RegistryEntry = {
|
||||
fingerprintSha256: string;
|
||||
material: StoredSecretVersionMaterial;
|
||||
|
||||
@@ -3,6 +3,7 @@ import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
SAFE_NATIVE_CHAT_PROGRESS_EVENT_TYPES,
|
||||
safeNativeChatProgressForEvent,
|
||||
publicNativeChatProgressText,
|
||||
} from "./safe-native-chat-progress.js";
|
||||
|
||||
describe("safe native chat progress", () => {
|
||||
@@ -65,3 +66,35 @@ describe("safe native chat progress", () => {
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
const publicEvent = (channel = "progress", kind = "agentMessage", text = "Found the retry bug. Checking recovery next.") => ({
|
||||
prpEvent: { schema: "paperclip.prp.event.v1", eventType: "item.completed", runId: "run-1",
|
||||
payload: { kind, channel, text } },
|
||||
});
|
||||
describe("public native progress", () => {
|
||||
it("selects only completed public commentary from the bound run", () => {
|
||||
expect(publicNativeChatProgressText("item.completed", publicEvent(), "run-1")).toBe("Found the retry bug. Checking recovery next.");
|
||||
expect(publicNativeChatProgressText("item.delta", publicEvent(), "run-1")).toBeNull();
|
||||
expect(publicNativeChatProgressText("item.completed", publicEvent(), "other-run")).toBeNull();
|
||||
});
|
||||
it.each(["summary", "detail", "unknown", "final", "commentary"])("rejects the unmapped %s channel", channel => {
|
||||
expect(publicNativeChatProgressText("item.completed", publicEvent(channel), "run-1")).toBeNull();
|
||||
});
|
||||
it("rejects tool and reasoning items even with a public channel", () => {
|
||||
for (const kind of ["reasoning", "commandExecution", "toolCall"])
|
||||
expect(publicNativeChatProgressText("item.completed", publicEvent("progress", kind), "run-1")).toBeNull();
|
||||
});
|
||||
it("keeps two sentences and redacts credentials and broadcast mentions", () => {
|
||||
const input = publicEvent("progress", "agentMessage", 'Checking @channel with token="secret123". Testing recovery. More details follow.');
|
||||
const text = publicNativeChatProgressText("item.completed", input, "run-1");
|
||||
expect(text).toContain("Testing recovery.");
|
||||
expect(text).not.toContain("More details");
|
||||
expect(text).not.toContain("secret123");
|
||||
expect(text).not.toContain("@channel");
|
||||
});
|
||||
it("sanitizes before bounding and keeps complete sentences", () => {
|
||||
const text = "<thinking>PRIVATE</thinking>Checking retries. " + "A".repeat(450);
|
||||
expect(publicNativeChatProgressText("item.completed", publicEvent("progress", "agentMessage", text), "run-1")).toBe("Checking retries.");
|
||||
expect(publicNativeChatProgressText("item.completed", publicEvent("progress", "agentMessage", "x".repeat(450)), "run-1")).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -1,3 +1,5 @@
|
||||
import { projectSafeChatPublicationText } from "./chat-publication-projection.js";
|
||||
|
||||
export const SAFE_NATIVE_CHAT_PROGRESS_EVENT_TYPES = [
|
||||
"workspace.ready",
|
||||
"research.started",
|
||||
@@ -81,3 +83,25 @@ export function safeNativeChatProgressForEvent(
|
||||
const phase = SAFE_NATIVE_CHAT_PROGRESS_PHASES[eventType];
|
||||
return { phase, text: SAFE_NATIVE_CHAT_PROGRESS_TEXT[phase](agentName) };
|
||||
}
|
||||
|
||||
export function publicNativeChatProgressText(
|
||||
eventType: string,
|
||||
payload: Record<string, unknown> | null,
|
||||
runId: string,
|
||||
): string | null {
|
||||
if (eventType !== "item.completed") return null;
|
||||
const event = payload?.prpEvent as Record<string, unknown> | undefined;
|
||||
if (!event || event.schema !== "paperclip.prp.event.v1" || event.eventType !== eventType || event.runId !== runId) return null;
|
||||
const item = event.payload as Record<string, unknown> | undefined;
|
||||
if (!item || item.kind !== "agentMessage" || item.channel !== "progress" || typeof item.text !== "string" || item.text.length > 16_000) return null;
|
||||
const text = projectSafeChatPublicationText(item.text).replace(/\s+/g, " ").trim();
|
||||
if (!text || text === "Update available in Paperclip.") return null;
|
||||
const sentences = text.match(/.+?(?:[.!?](?=\s|$)|$)/g) ?? [];
|
||||
let note = "";
|
||||
for (const sentence of sentences.slice(0, 2)) {
|
||||
const next = [note, sentence.trim()].filter(Boolean).join(" ");
|
||||
if (next.length > 400) break;
|
||||
note = next;
|
||||
}
|
||||
return note || null;
|
||||
}
|
||||
Reference in new issue
Block a user