mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 20:29:18 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run commits its status decision and its final reply comment
in separate steps
> - Background presentation recovery publishes an accepted reply when
the live path does not record one
> - That worker dropped the run after one refused attempt and only threw
on an error, with no record on the run
> - A task could then show done while the user never saw the answer,
until an unrelated wake reloaded the queue
> - This pull request makes the publication of an accepted ordinary task
reply durable, bounded, and recorded
> - The benefit is that a task cannot close silently without its answer
## Linked Issues or Issue Description
Refs #19130 (verify requested publication at finalization, for
documents), #19183 (deliver messages saved while a native run
finalizes), #19186 (stale owner done). This change does not duplicate
them. It covers the final reply comment.
**What happened?**
A native run accepted a result with disposition `done`. The status
decision set the task to done. The accepted final reply was not
published as a task comment. The run had no publication record (no
`run.presentation.resolved` event and no `presentationDecision`). The
reply was published by background recovery 58 minutes later, after an
unrelated wake reloaded the presentation queue.
**Expected behavior**
The accepted reply is published promptly. A refused or failed attempt is
retried with backoff and never posts twice. If publication fails
permanently, the run and the task show a named failure with an owner.
**Steps to reproduce**
1. Commit an accepted native task result with disposition `done`, with
the final reply not yet visible to presentation recovery.
2. Run the native response presentation worker check once. The attempt
is refused.
3. On `master`, the worker returns `null` and drops the run. Nothing
records the attempt. Nothing retries it until another notification for
that scope.
**Paperclip version or commit**
`master` at 3b17b5ce15.
**Deployment mode**
Cloud staging, native `paperclip_runner` agent.
Root cause notes: the staging evidence shows that the live completion
path did not reach its presentation step for the affected run (no
`lifecycle run succeeded` event). Staging server logs for that window
are not available, so the exact first refusal is not proven. The worker
defect is proven: one refusal makes `check()` return `null`, which
deletes the scheduled entry. One thrown error escapes with only
in-memory backoff and no record. The fix closes that gap for every
refusal or error cause.
## What Changed
- `native-response-presentation-work.ts`: when an accepted ordinary task
reply (`done` or `needs_review`, not an Agent Chat conversation) is
still unpublished after a due attempt, the worker records the attempt
and keeps the run scheduled. A thrown error for this class is recorded
instead of escaping. When the reply is published, an open retry receipt
is set to `resolved`.
- New `native-response-publication-receipt.ts`: stores the receipt in
`heartbeat_runs.result_json.responsePublication` (attempts, last reason,
last error, next attempt). Backoff is 30 s doubling to at most 5 min,
for 6 attempts. Each retry appends a `run.response_publication.retrying`
run event. After the last attempt it records
`native_response_publication_failed` with the run's agent as owner, a
`run.response_publication.failed` run event, and an
`issue.response_publication_failed` task activity.
- `native-response-presentation-query.ts`: the candidate query exposes
the saved retry deadline, so a restarted worker reconstructs the
schedule. It excludes a run whose publication is recorded as failed, so
retries stay bounded.
- An open retry receipt is settled to `resolved` in the same write that
records the reply's `commentId` (background recovery and the live
`recordRunPresentationDecision`). A candidate that disappears for
another reason leaves the receipt unchanged, so the bound cannot reset.
- When the reply comment already exists but its decision was not
recorded, recovery now records it as `reuse` with that `commentId`. It
does not post again and does not count a refusal.
- Under the run lock, an attempt is not counted again when another
worker already saved a future deadline. The worker also honors that
saved deadline after recovery.
- Idempotency is unchanged: the comment is written only while the
presentation decision has no `commentId`, under the existing coordinator
lock. The receipt write never overwrites a run that already has a
`commentId`.
- `doc/execution-semantics.md`: documents the contract.
## Verification
- `cd server && npx vitest run
src/services/native-runtime/native-response-publication-retry.test.ts`
- Red on `master` (3 failed): after one refusal `check()` returned
`null` (dropped); one injected transaction error escaped with no record;
the refusal loop never stopped (more than 10 attempts, no failure
recorded).
- Green with this change (5 passed, including an already posted reply
and two concurrent workers): one refusal keeps the run scheduled and
recorded, a reloaded worker keeps the same deadline, the reply posts
exactly once after it becomes visible, and repeated checks do not
double-post; one injected error is recorded and retried; permanent
refusal stops after bounded attempts with the named, owned failure on
the run event and task activity.
- Related suites pass: `agent-chat-response-recovery.test.ts`,
`presentation-ownership-read.test.ts`,
`idle-native-finalization.test.ts`, `chat-monitor-continuation.test.ts`,
`native-preserved-monitor-wait.test.ts`,
`paperclip-control-plane-port.test.ts`,
`chat-channels.integration.test.ts`,
`photon/channel.integration.test.ts` (2047 of 2049 passed in one
combined run). One local failure needs a bundled Slack skill file that
is missing in this worktree. The other ("materializes direct
external-chat finals once") failed once under load and then passed three
times in isolation and in the combined set with this change.
- Not run: a live staging run. The change has not been deployed.
## Risks
- Behavior change: an ordinary task reply that recovery keeps refusing
now gets a recorded failure and a task activity after about 12 minutes.
Before, it was dropped silently. A persistent integrity decline (for
example a digest mismatch) now becomes visible this way. This is
intended.
- The status decision still commits before the reply. This PR does not
hold `done` on the comment. It makes a missing reply visible with an
owner.
- Agent Chat conversations and yielded chat responses keep their
existing behavior and their completion outbox owner.
- No schema migration. The receipt lives in the existing `result_json`
column.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
137 lines
7.1 KiB
TypeScript
137 lines
7.1 KiB
TypeScript
import { and, eq, sql } from "drizzle-orm";
|
|
import { heartbeatRuns, type Db } from "@paperclipai/db";
|
|
import { logActivity } from "../activity-log.js";
|
|
import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
|
|
|
|
/**
|
|
* Durable, bounded publication of an accepted native task reply.
|
|
*
|
|
* The status decision for an accepted result commits independently of the
|
|
* final reply comment. When publication is refused or fails, the attempt is
|
|
* recorded on the run and retried with backoff. Exhausted retries leave a
|
|
* named failure with an owner on the run and the task instead of a silent
|
|
* done. The comment write itself stays idempotent through the presentation
|
|
* decision's `commentId`, so retries never double-post.
|
|
*/
|
|
export const NATIVE_RESPONSE_PUBLICATION_SCHEMA = "paperclip.native_response_publication.v1";
|
|
export const NATIVE_RESPONSE_PUBLICATION_FAILED = "native_response_publication_failed";
|
|
export const NATIVE_RESPONSE_PUBLICATION_MAX_ATTEMPTS = 6;
|
|
const BASE_DELAY_MS = 30_000;
|
|
const MAX_DELAY_MS = 300_000;
|
|
|
|
export type NativeResponsePublicationReason = "publication_refused" | "publication_error";
|
|
|
|
type Receipt = {
|
|
schema: typeof NATIVE_RESPONSE_PUBLICATION_SCHEMA;
|
|
state: "retrying" | "failed" | "resolved";
|
|
attempts: number;
|
|
firstAttemptAt: string;
|
|
lastAttemptAt: string;
|
|
lastReason: NativeResponsePublicationReason;
|
|
lastError?: string | null;
|
|
nextAttemptAt?: string | null;
|
|
reasonCode?: string;
|
|
owner?: { kind: "agent"; agentId: string };
|
|
failedAt?: string;
|
|
resolvedAt?: string;
|
|
};
|
|
|
|
export function nativeResponsePublicationDelayMs(attempts: number) {
|
|
return Math.min(MAX_DELAY_MS, BASE_DELAY_MS * 2 ** Math.max(0, attempts - 1));
|
|
}
|
|
|
|
function readReceipt(resultJson: unknown): Receipt | null {
|
|
if (!resultJson || typeof resultJson !== "object") return null;
|
|
const receipt = (resultJson as Record<string, unknown>).responsePublication;
|
|
return receipt && typeof receipt === "object" ? receipt as Receipt : null;
|
|
}
|
|
|
|
function errorText(error: unknown) {
|
|
if (!error) return null;
|
|
const text = error instanceof Error ? error.message : String(error);
|
|
return text.slice(0, 500);
|
|
}
|
|
|
|
function patchReceipt(tx: Db, runId: string, receipt: Receipt) {
|
|
// Never overwrite a materialized reply's run record with retry bookkeeping.
|
|
return tx.update(heartbeatRuns).set({
|
|
resultJson: sql`coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb)
|
|
|| jsonb_build_object('responsePublication', ${JSON.stringify(receipt)}::jsonb)`,
|
|
updatedAt: new Date(),
|
|
}).where(and(eq(heartbeatRuns.id, runId),
|
|
sql`jsonb_typeof(${heartbeatRuns.resultJson} #> '{presentationDecision,commentId}') is distinct from 'string'`))
|
|
.returning({ id: heartbeatRuns.id });
|
|
}
|
|
|
|
/** Record one unsuccessful attempt. Returns the next due time, or null once
|
|
* the bounded retry is exhausted and the failure has been recorded. */
|
|
export async function recordNativeResponsePublicationAttempt(db: Db, input: {
|
|
companyId: string; issueId: string; runId: string;
|
|
reason: NativeResponsePublicationReason; error?: unknown;
|
|
}): Promise<{ state: "retrying"; nextAttemptAt: number } | { state: "failed" } | null> {
|
|
return db.transaction(async transaction => {
|
|
const tx = transaction as unknown as Db;
|
|
const [run] = await tx.select({ id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, resultJson: heartbeatRuns.resultJson })
|
|
.from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)))
|
|
.for("update");
|
|
if (!run) return null;
|
|
const previous = readReceipt(run.resultJson);
|
|
if (previous?.state === "failed") return { state: "failed" as const };
|
|
const now = new Date();
|
|
// Another worker already counted this due attempt and saved the next
|
|
// deadline. Keep its schedule instead of counting the attempt twice.
|
|
const scheduled = previous?.state === "retrying" && previous.nextAttemptAt ? Date.parse(previous.nextAttemptAt) : NaN;
|
|
if (Number.isFinite(scheduled) && scheduled > now.getTime()) return { state: "retrying" as const, nextAttemptAt: scheduled };
|
|
const attempts = (previous?.state === "retrying" ? previous.attempts : 0) + 1;
|
|
const base = {
|
|
schema: NATIVE_RESPONSE_PUBLICATION_SCHEMA as typeof NATIVE_RESPONSE_PUBLICATION_SCHEMA,
|
|
attempts,
|
|
firstAttemptAt: previous?.state === "retrying" ? previous.firstAttemptAt : now.toISOString(),
|
|
lastAttemptAt: now.toISOString(),
|
|
lastReason: input.reason,
|
|
lastError: errorText(input.error),
|
|
};
|
|
if (attempts < NATIVE_RESPONSE_PUBLICATION_MAX_ATTEMPTS) {
|
|
const nextAttemptAt = new Date(now.getTime() + nativeResponsePublicationDelayMs(attempts));
|
|
const [updated] = await patchReceipt(tx, run.id, { ...base, state: "retrying", nextAttemptAt: nextAttemptAt.toISOString() });
|
|
if (!updated) return null;
|
|
await appendHeartbeatRunEvent(tx, {
|
|
companyId: input.companyId, runId: run.id, agentId: run.agentId,
|
|
eventType: "run.response_publication.retrying", stream: "system", level: "warn",
|
|
message: `Final reply publication ${input.reason === "publication_error" ? "failed" : "was refused"}; retry ${attempts + 1} of ${NATIVE_RESPONSE_PUBLICATION_MAX_ATTEMPTS} is scheduled`,
|
|
payload: { attempts, reason: input.reason, nextAttemptAt: nextAttemptAt.toISOString() },
|
|
});
|
|
return { state: "retrying" as const, nextAttemptAt: nextAttemptAt.getTime() };
|
|
}
|
|
const owner = { kind: "agent" as const, agentId: run.agentId };
|
|
const [updated] = await patchReceipt(tx, run.id, { ...base, state: "failed", nextAttemptAt: null,
|
|
reasonCode: NATIVE_RESPONSE_PUBLICATION_FAILED, owner, failedAt: now.toISOString() });
|
|
if (!updated) return null;
|
|
await appendHeartbeatRunEvent(tx, {
|
|
companyId: input.companyId, runId: run.id, agentId: run.agentId,
|
|
eventType: "run.response_publication.failed", stream: "system", level: "error",
|
|
message: `The accepted final reply could not be published to the task after ${attempts} attempts`,
|
|
payload: { reasonCode: NATIVE_RESPONSE_PUBLICATION_FAILED, attempts, lastReason: input.reason, owner },
|
|
});
|
|
await logActivity(tx, {
|
|
companyId: input.companyId, actorType: "system", actorId: "native-response-publication",
|
|
action: "issue.response_publication_failed", entityType: "issue", entityId: input.issueId,
|
|
agentId: run.agentId, runId: run.id,
|
|
details: { runId: run.id, reasonCode: NATIVE_RESPONSE_PUBLICATION_FAILED, attempts, lastReason: input.reason,
|
|
lastError: base.lastError, owner },
|
|
});
|
|
return { state: "failed" as const };
|
|
});
|
|
}
|
|
|
|
/** JSON object to merge into `result_json` in the same write that records a
|
|
* published reply's `commentId`. It settles an open retry receipt atomically
|
|
* with publication, and is empty when no retry is open. */
|
|
export function settledResponsePublicationSql() {
|
|
return sql`(case when ${heartbeatRuns.resultJson} #>> '{responsePublication,state}' = 'retrying'
|
|
then jsonb_build_object('responsePublication', ((${heartbeatRuns.resultJson} -> 'responsePublication') - 'nextAttemptAt')
|
|
|| jsonb_build_object('state', 'resolved', 'resolvedAt', ${new Date().toISOString()}::text))
|
|
else '{}'::jsonb end)`;
|
|
}
|