mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(recovery): settle uncertain runner API receipts instead of fencing them forever (#21522)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs change Paperclip through the runner `call_api`
tool. Each mutation reserves a receipt before dispatch, so a crash
cannot replay a mutation that may have committed.
> - When the response of a mutation is lost, the receipt records an
unknown outcome. `priorRecoveryApiReceipt` then fences every later
mutation with the same operation id on that task, in every later run.
> - Nothing can settle that fence. The agent gets "Reconcile the
existing operation", but it has no tool to do that. The task stays stuck
for agents.
> - This pull request records the server's own final status for each run
mutation, and settles an unknown receipt from that record. A later
change by a board user on the task also lifts the fence.
> - The benefit is that a lost response no longer blocks a task's
mutations forever, and the fence still holds when the outcome is really
unknown.
## Linked Issues or Issue Description
**What happened?**
A routine run sent `PATCH /api/issues/{id}` to hand the task to a board
user. The server committed the change, but the server was slow and the
response took longer than the 30 second client timeout. The receipt
recorded `api_transport_failure` with `outcome: "unknown"`. Hours later,
the board user commented and reassigned the task to the agent. The new
run did the work, but both of its hand-off PATCH calls returned
`api_outcome_unknown` within 2 seconds and never reached the route.
Every later run on that task would get the same result.
**Expected behavior**
When the server has a record of the earlier call's result, the uncertain
receipt settles from that record. When a person has changed the task
after the uncertain run, the agent can act on the current state.
**Steps to reproduce**
1. In a native run, send a `call_api` mutation that commits on the
server, but let the client time out before the response.
2. Start a later run in the same retry or continuation ancestry.
3. Send any mutation with the same operation id. It returns
`api_outcome_unknown` and is not dispatched.
**Relevant logs or output**
```
call_api PATCH /api/issues/{id} -> { ok: false, status: null, error: "api_outcome_unknown", outcome: "unknown",
guidance: "Reconcile the existing operation before repeating it. ..." }
```
No related public PR exists. I searched open and closed PRs for this
fence.
## What Changed
- `runner-api-client.ts`: `executeRunnerApi` sends the receipt key in
`X-Paperclip-Runner-Call` with mutations. Reads do not send it.
- `paperclip-runner-tool-authority.ts`: `#callApi` passes the receipt
key of the reserved mutation to the client.
- `middleware/runner-call-outcome.ts` (new): for an agent JWT request
with a run id and a valid call key, it records the final HTTP status as
a `runner.api_completed` activity row of the run. It records when the
handler ends the response, so it also records after the client has gone
away. It records one time per request.
- `recovery-effect-receipts.ts`: `priorRecoveryApiReceipt` settles an
unknown receipt:
- A recorded 2xx is an applied effect. An exact retry from the ancestry
gets an applied result and is not dispatched again. Other calls
continue.
- A recorded definite 4xx rejection (not 408) no longer fences, the same
as a completed 4xx receipt today.
- A recorded 5xx, or no record, keeps the fence.
- A board user's `issue.updated` or `issue.comment_added` on the task
after the uncertain run finished lifts the fence of that run. Agent,
system, read and inbox activity do not.
- The fenced result now tells the agent to read the current state and
not to repeat a change that is already there.
## Verification
- `pnpm --filter @paperclipai/server run typecheck` passes.
- `npx vitest run src/middleware/runner-call-outcome.test.ts
src/services/task-retry.test.ts
src/services/native-runtime/runner-api.test.ts
src/services/native-runtime/runner-api.integration.test.ts
src/services/native-runtime/runner-api-artifacts.test.ts
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/__tests__/teams-catalog-service.test.ts
src/__tests__/primary-agent-guidance.test.ts
src/__tests__/skill-import-discovery.test.ts` (in `server/`): 2418
passed, 2 skipped.
- `npx vitest run src/__tests__/runner-api-recovery.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts
src/services/native-runtime/confirmation-comment-api.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/runner-api-log-redaction.test.ts
src/services/native-runtime/runner-api-artifacts.integration.test.ts
src/services/native-runtime/native-case-tools.integration.test.ts
src/__tests__/connection-management.integration.test.ts`: 404 passed, 1
failed. The failure is `runner-api-artifacts.integration.test.ts >
preserves exact HTTP bytes in the immutable asset and downloaded helper
input` ("Artifact installation is unavailable to this run or provider").
It fails the same way with the unchanged `master` files in my
environment, so it is not from this change. CI covers it.
- Red before the fix: with the `master` version of
`recovery-effect-receipts.ts`, the new settlement and supersession cases
in `task-retry.test.ts` fail.
- New tests:
- `task-retry.test.ts`: settlement by recorded status 200, 422, 500, and
a record of a different call only; supersession by a later user, agent
or system change, including a change before the run finished and a read
marker.
- `runner-call-outcome.test.ts`: records the final status one time;
ignores reads, a missing or malformed key, agent API keys, board actors
and non-UUID run ids.
- `runner-api.test.ts`: the call key header is sent with mutations only.
## Risks
- Each run mutation that carries a call key adds one activity row
(`runner.api_completed`, entity `heartbeat_run`). The row has no request
body and no response body. Only the status and the receipt key hash.
- The settlement trusts the server's own final status. Routes commit
before they respond, so a 2xx means the effect is applied. A route that
answers 2xx before it commits its effect is still treated as applied.
- A run can write its own outcome record with its own token. This can
only settle that run's own receipts, so it does not give an agent new
access.
- Receipts made before this change have no record. They keep the fence
until a board user changes the task.
- No migration.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
e82c4f0a31
commit
089748fbf9
8 files changed
+252
-13
No files matched your search
@@ -98,6 +98,7 @@ import { readinessRoutes } from "./routes/ready.js";
|
||||
import { livenessRoutes } from "./routes/liveness.js";
|
||||
import { cloudRuntimeIdentityMiddleware } from "./middleware/cloud-runtime-identity.js";
|
||||
import { cloudControlMiddleware } from "./middleware/cloud-control.js";
|
||||
import { runnerCallOutcomeMiddleware } from "./middleware/runner-call-outcome.js";
|
||||
import { cloudRoutes } from "./routes/cloud.js";
|
||||
import { companyRoutes } from "./routes/companies.js";
|
||||
import { companySkillRoutes } from "./routes/company-skills.js";
|
||||
@@ -623,6 +624,7 @@ export async function createApp(
|
||||
// REPLACES whatever actor the request otherwise resolved to, and only on
|
||||
// the one endpoint it authorizes (see the middleware for the contract).
|
||||
app.use(cloudControlMiddleware());
|
||||
app.use(runnerCallOutcomeMiddleware(db));
|
||||
app.use(connectionEventWakeRoutes(db));
|
||||
app.use(skillAccessRequestContext);
|
||||
app.use("/api/auth", authRoutes(db));
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
import type { Request, Response } from "express";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
|
||||
const persistActivity = vi.hoisted(() => vi.fn(async () => ({})));
|
||||
vi.mock("../services/activity-log.js", () => ({ persistActivity }));
|
||||
|
||||
import { runnerCallOutcomeMiddleware } from "./runner-call-outcome.js";
|
||||
|
||||
const runId = "11111111-2222-4333-8444-555555555555";
|
||||
const callKey = "d".repeat(64);
|
||||
const agent = { type: "agent", source: "agent_jwt", agentId: "agent-a", companyId: "company-a", runId } as const;
|
||||
|
||||
function invoke(actor: unknown, options: { method?: string; key?: string | null } = {}) {
|
||||
const key = options.key === undefined ? callKey : options.key;
|
||||
const end = vi.fn();
|
||||
const req = { method: options.method ?? "PATCH", actor,
|
||||
header: (name: string) => (name.toLowerCase() === "x-paperclip-runner-call" ? key ?? undefined : undefined) } as unknown as Request;
|
||||
const res = { statusCode: 200, end } as unknown as Response;
|
||||
const next = vi.fn();
|
||||
runnerCallOutcomeMiddleware({} as Db)(req, res, next);
|
||||
return { res, end, next };
|
||||
}
|
||||
|
||||
describe("runner call outcome", () => {
|
||||
beforeEach(() => persistActivity.mockClear());
|
||||
|
||||
it.each([201, 422, 500])("records the final status %s of an agent run's keyed mutation once", (status) => {
|
||||
const { res, end, next } = invoke(agent);
|
||||
expect(next).toHaveBeenCalledOnce();
|
||||
res.statusCode = status;
|
||||
res.end();
|
||||
res.end();
|
||||
expect(end).toHaveBeenCalledTimes(2);
|
||||
expect(persistActivity).toHaveBeenCalledExactlyOnceWith({}, { companyId: "company-a", actorType: "agent", actorId: "agent-a",
|
||||
agentId: "agent-a", runId, action: "runner.api_completed", entityType: "heartbeat_run", entityId: runId,
|
||||
details: { callKey, status } });
|
||||
});
|
||||
|
||||
it.each([
|
||||
["a read", agent, { method: "GET" }],
|
||||
["a missing call key", agent, { key: null }],
|
||||
["a malformed call key", agent, { key: "not-a-key" }],
|
||||
["an agent API key", { ...agent, source: "agent_key" }, {}],
|
||||
["a board actor", { type: "board", source: "session", runId }, {}],
|
||||
["a non-UUID run", { ...agent, runId: "run-a" }, {}],
|
||||
] as const)("ignores %s", (_case, actor, options) => {
|
||||
const { res, end, next } = invoke(actor, options);
|
||||
expect(next).toHaveBeenCalledOnce();
|
||||
res.end();
|
||||
expect(end).toHaveBeenCalledOnce();
|
||||
expect(persistActivity).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,41 @@
|
||||
import type { RequestHandler } from "express";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { RUNNER_API_CALL_HEADER } from "../services/native-runtime/runner-api-client.js";
|
||||
import { persistActivity } from "../services/activity-log.js";
|
||||
import { RUNNER_API_COMPLETED_ACTION } from "../services/recovery-effect-receipts.js";
|
||||
import { logger } from "./logger.js";
|
||||
|
||||
const CALL_KEY = /^[0-9a-f]{64}$/;
|
||||
const RUN_ID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
|
||||
const READ_METHODS = new Set(["GET", "HEAD", "OPTIONS"]);
|
||||
|
||||
export async function recordRunnerApiOutcome(db: Db, input: {
|
||||
companyId: string; agentId: string; runId: string; callKey: string; status: number;
|
||||
}) {
|
||||
await persistActivity(db, { companyId: input.companyId, actorType: "agent", actorId: input.agentId,
|
||||
agentId: input.agentId, runId: input.runId, action: RUNNER_API_COMPLETED_ACTION,
|
||||
entityType: "heartbeat_run", entityId: input.runId, details: { callKey: input.callKey, status: input.status } });
|
||||
}
|
||||
|
||||
export function runnerCallOutcomeMiddleware(db: Db): RequestHandler {
|
||||
return (req, res, next) => {
|
||||
const callKey = req.header(RUNNER_API_CALL_HEADER);
|
||||
const { type, source, agentId, companyId, runId } = req.actor ?? {};
|
||||
if (!callKey || !CALL_KEY.test(callKey) || READ_METHODS.has(req.method) || type !== "agent"
|
||||
|| source !== "agent_jwt" || !agentId || !companyId || !runId || !RUN_ID.test(runId)) {
|
||||
next();
|
||||
return;
|
||||
}
|
||||
const end = res.end;
|
||||
let recorded = false;
|
||||
res.end = function (this: typeof res, ...args: unknown[]) {
|
||||
if (!recorded) {
|
||||
recorded = true;
|
||||
recordRunnerApiOutcome(db, { companyId, agentId, runId, callKey, status: res.statusCode })
|
||||
.catch((err) => logger.warn({ err, runId }, "Failed to record a runner API call outcome"));
|
||||
}
|
||||
return (end as (...endArgs: unknown[]) => typeof res).apply(this, args);
|
||||
} as typeof res.end;
|
||||
next();
|
||||
};
|
||||
}
|
||||
@@ -1205,11 +1205,11 @@ export class PaperclipRunnerToolAuthority {
|
||||
if (!apiUrl) throw new Error("Paperclip API origin is unavailable");
|
||||
const token = this.#apiToken(bound);
|
||||
if (!token) throw new Error("Paperclip run authentication is unavailable");
|
||||
const execute = async () => {
|
||||
const execute = async (callKey?: string) => {
|
||||
const current = await this.#boundContext();
|
||||
assertAvailable(current.issue.workMode);
|
||||
const result = await executeRunnerApi(input, { ...context, workMode: current.issue.workMode }, {
|
||||
apiUrl, token,
|
||||
apiUrl, token, callKey,
|
||||
beforeDispatch: async () => {
|
||||
const fresh = await this.#boundContext();
|
||||
assertAvailable(fresh.issue.workMode);
|
||||
@@ -1253,7 +1253,7 @@ export class PaperclipRunnerToolAuthority {
|
||||
return null;
|
||||
});
|
||||
if (prior) return prior.result;
|
||||
const result = await execute();
|
||||
const result = await execute(key);
|
||||
const storedResult = provider
|
||||
? { encryptedResult: (await provider.createSecret({ value: JSON.stringify(result) })).material }
|
||||
: { result };
|
||||
|
||||
@@ -36,9 +36,12 @@ export interface RunnerApiContext {
|
||||
runId: string;
|
||||
workMode: string;
|
||||
}
|
||||
export const RUNNER_API_CALL_HEADER = "X-Paperclip-Runner-Call";
|
||||
|
||||
export interface RunnerApiIo {
|
||||
apiUrl: string;
|
||||
token: string;
|
||||
callKey?: string;
|
||||
fetch?: typeof fetch;
|
||||
/** Server-owned page size for the dedicated saved-result reader. */
|
||||
responseTextPageBytes?: number;
|
||||
@@ -209,6 +212,7 @@ export async function executeRunnerApi(input: RunnerApiCall, context: RunnerApiC
|
||||
const url = runnerApiUrl(operation, input, context, io.apiUrl);
|
||||
if (!io.token) throw new Error("Paperclip run authentication is unavailable");
|
||||
const headers = new Headers({ Authorization: `Bearer ${io.token}`, "X-Paperclip-Run-Id": context.runId });
|
||||
if (io.callKey && !["GET", "HEAD", "OPTIONS"].includes(operation.method)) headers.set(RUNNER_API_CALL_HEADER, io.callKey);
|
||||
const textOffset = input.responseText?.offsetBytes ?? 0;
|
||||
const textLimit = input.responseText?.limitBytes
|
||||
?? (input.responseText ? io.responseTextPageBytes : undefined) ?? RUNNER_API_INLINE_BYTES;
|
||||
|
||||
@@ -7,6 +7,7 @@ import {
|
||||
import {
|
||||
executeRunnerApi,
|
||||
readBoundedResponse,
|
||||
RUNNER_API_CALL_HEADER,
|
||||
runnerApiUrl,
|
||||
validateRunnerApiCall,
|
||||
type RunnerApiCall,
|
||||
@@ -79,6 +80,15 @@ describe("capped list pagination", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("runner call identity", () => {
|
||||
it.each([[createProject, "call-key-a"], [projects, null]] as const)("sends the call key only with mutations: %s", async (operationId, expected) => {
|
||||
const fetcher = vi.fn(async (_url: string | URL | Request, _init?: RequestInit) => new Response("{}", { headers: { "content-type": "application/json" } }));
|
||||
await executeRunnerApi({ operationId, ...(operationId === createProject ? { body: { name: "Keyed" } } : {}) }, context,
|
||||
{ ...io(fetcher as unknown as typeof fetch), callKey: "call-key-a" });
|
||||
expect(new Headers(fetcher.mock.calls[0]![1]!.headers).get(RUNNER_API_CALL_HEADER)).toBe(expected);
|
||||
});
|
||||
});
|
||||
|
||||
describe("bounded response capture receipts", () => {
|
||||
it.each([projects, createProject])("reports oversize evidence without retrying %s", async operationId => {
|
||||
const fetcher = vi.fn(async () => new Response("", { headers: { "content-length": String(RUNNER_API_RESPONSE_MAX_BYTES + 1) } }));
|
||||
|
||||
@@ -1,11 +1,36 @@
|
||||
import { and, eq, sql, ne, or, inArray } from "drizzle-orm";
|
||||
import { heartbeatRuns, toolInvocations, type Db } from "@paperclipai/db";
|
||||
import { and, desc, eq, gt, sql, ne, or, inArray } from "drizzle-orm";
|
||||
import { activityLog, heartbeatRuns, toolInvocations, type Db } from "@paperclipai/db";
|
||||
import { getSecretProvider } from "../secrets/provider-registry.js";
|
||||
import { recoveryRunAncestry } from "./recovery-run-ancestry.js";
|
||||
|
||||
const record = (value: unknown): Record<string, unknown> =>
|
||||
value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
||||
|
||||
export const RUNNER_API_COMPLETED_ACTION = "runner.api_completed";
|
||||
|
||||
const UNKNOWN_API_OUTCOME = { ok: false, status: null, error: "api_outcome_unknown", outcome: "unknown",
|
||||
guidance: "An earlier attempt of this operation may already have applied. Read the current state, and do not repeat the change if it is already there. Independent work and reads remain available." };
|
||||
|
||||
async function settleUnknownApiReceipt(db: Db, input: { companyId: string; issueId: string },
|
||||
source: { id: string; finishedAt: Date | null }, callKey: string,
|
||||
): Promise<"fenced" | "superseded" | "rejected" | { status: number }> {
|
||||
const [outcome] = await db.select({ details: activityLog.details }).from(activityLog).where(and(
|
||||
eq(activityLog.companyId, input.companyId), eq(activityLog.runId, source.id),
|
||||
eq(activityLog.action, RUNNER_API_COMPLETED_ACTION), sql`${activityLog.details}->>'callKey' = ${callKey}`,
|
||||
)).orderBy(desc(activityLog.createdAt)).limit(1);
|
||||
const status = record(outcome?.details).status;
|
||||
if (typeof status === "number" && status >= 200 && status < 300) return { status };
|
||||
if (typeof status === "number" && status >= 400 && status < 500 && status !== 408) return "rejected";
|
||||
if (!source.finishedAt) return "fenced";
|
||||
const [human] = await db.select({ id: activityLog.id }).from(activityLog).where(and(
|
||||
eq(activityLog.companyId, input.companyId), eq(activityLog.entityType, "issue"),
|
||||
eq(activityLog.entityId, input.issueId), eq(activityLog.actorType, "user"),
|
||||
inArray(activityLog.action, ["issue.updated", "issue.comment_added"]),
|
||||
gt(activityLog.createdAt, source.finishedAt),
|
||||
)).limit(1);
|
||||
return human ? "superseded" : "fenced";
|
||||
}
|
||||
|
||||
/** Generic upstream tools have no common authoritative result-query protocol.
|
||||
* Preserve their uncertain mutation fence across recovery. Exact publication
|
||||
* protocols can reconcile through their existing owner instead. No transcript
|
||||
@@ -44,7 +69,8 @@ export async function priorRecoveryApiReceipt(db: Db, input: {
|
||||
issueId: input.issueId, sourceRunIds: [input.runId] }))
|
||||
.map(source => source.id).filter(id => id !== input.runId));
|
||||
if (!ancestry.size) return null;
|
||||
const sources = await db.select({ id: heartbeatRuns.id, result: heartbeatRuns.resultJson }).from(heartbeatRuns).where(and(
|
||||
const sources = await db.select({ id: heartbeatRuns.id, result: heartbeatRuns.resultJson,
|
||||
finishedAt: heartbeatRuns.finishedAt }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.companyId, input.companyId),
|
||||
sql`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text,
|
||||
${heartbeatRuns.contextSnapshot}->>'issueId') = ${input.issueId}`,
|
||||
@@ -53,23 +79,28 @@ export async function priorRecoveryApiReceipt(db: Db, input: {
|
||||
sql`${heartbeatRuns.createdAt} < (select created_at from heartbeat_runs where id = ${input.runId})`)));
|
||||
let completed: Record<string, unknown> | null = null;
|
||||
for (const source of sources) {
|
||||
for (const receipt of Object.values(record(source.result?.apiToolReceipts)).map(record)) {
|
||||
for (const [callKey, value] of Object.entries(record(source.result?.apiToolReceipts))) {
|
||||
const receipt = record(value);
|
||||
if (receipt.operationId !== input.operationId) continue;
|
||||
const result = receipt.encryptedResult
|
||||
? record(JSON.parse(await getSecretProvider("local_encrypted").resolveVersion({ material: record(receipt.encryptedResult), externalRef: null })))
|
||||
: record(receipt.result);
|
||||
if (receipt.state !== "completed" || result.outcome === "unknown" || result.error === "api_outcome_unknown") {
|
||||
return { state: "pending", result: { ok: false, status: null, error: "api_outcome_unknown",
|
||||
outcome: "unknown", guidance: "Reconcile the existing operation before repeating it. Independent work and reads remain available." } };
|
||||
const settlement = await settleUnknownApiReceipt(db, input, source, callKey);
|
||||
if (settlement === "fenced") return { state: "pending", result: UNKNOWN_API_OUTCOME };
|
||||
if (typeof settlement === "object" && ancestry.has(source.id) && receipt.digest === input.digest) {
|
||||
completed = { digest: receipt.digest, operationId: receipt.operationId, state: "completed",
|
||||
result: { ok: true, status: settlement.status, operationId: receipt.operationId, outcome: "applied",
|
||||
guidance: "An earlier attempt of this exact call was applied. Read the current state for its result." } };
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// A completed HTTP request is not necessarily a completed effect. A
|
||||
// definite client rejection may be retried after the user fixes its cause.
|
||||
// Preserve the transport owner's uncertainty classification above.
|
||||
if (result.ok === false && typeof result.status === "number" &&
|
||||
result.status >= 400 && result.status < 500 && result.status !== 408) continue;
|
||||
if (result.ok !== true) return { state: "pending", result: { ok: false, status: null,
|
||||
error: "api_outcome_unknown", outcome: "unknown",
|
||||
guidance: "Reconcile the existing operation before repeating it. Independent work and reads remain available." } };
|
||||
if (result.ok !== true) return { state: "pending", result: UNKNOWN_API_OUTCOME };
|
||||
// Successful effects belong to their original intent, not every later
|
||||
// request on this task. Uncertain effects above retain their wider fence.
|
||||
if (ancestry.has(source.id) && receipt.digest === input.digest) completed = receipt;
|
||||
|
||||
@@ -1,9 +1,13 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import type { AddressInfo } from "node:net";
|
||||
import express from "express";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { activityLog, agents, agentTaskSessions, agentWakeupRequests, chatActions, chatConversations, chatEndpoints, companies, budgetPolicies, costEvents, createDb, type Db, environmentLeases, environments, executionWorkspaces, executionWorkspaceSnapshots, workspaceOperations, heartbeatRunEvents, heartbeatRuns, issueComments, issueRelations, issues, approvals, issueApprovals, issueRecoveryActions, nativeRunFinalizations, toolApplications, toolConnections, toolInvocations, applyPendingMigrations } from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
|
||||
import { priorRecoveryApiReceipt, unresolvedRecoveryToolInvocation } from "./recovery-effect-receipts.js";
|
||||
import { recordRunnerApiOutcome, runnerCallOutcomeMiddleware } from "../middleware/runner-call-outcome.js";
|
||||
import { RUNNER_API_CALL_HEADER } from "./native-runtime/runner-api-client.js";
|
||||
import { requestTaskRetry } from "./task-retry.js";
|
||||
import { HttpError } from "../errors.js";
|
||||
import { NativeChatReviewPresentationContentionError } from "./native-runtime/native-chat-review-presentation.js";
|
||||
@@ -1165,6 +1169,99 @@ describe("durable task Retry admission", () => {
|
||||
expect(await priorRecoveryApiReceipt(db, call)).toBeNull();
|
||||
});
|
||||
|
||||
it.each([[200, "applied"], [422, "rejected"], [500, "fenced"], [null, "fenced"]] as const)(
|
||||
"settles an uncertain API receipt from the recorded server outcome %s of the same call", async (status, expected) => {
|
||||
const { input, retry, automatic } = await automaticSuccessorOfTaskRetry();
|
||||
const callKey = "a".repeat(64);
|
||||
const receipt = { operationId: "PATCH /api/issues/{id}", digest: "same-update", state: "completed",
|
||||
result: { ok: false, status: null, error: "api_transport_failure", outcome: "unknown" } };
|
||||
await db.update(heartbeatRuns).set({ resultJson: { apiToolReceipts: { [callKey]: receipt } } }).where(eq(heartbeatRuns.id, retry.id));
|
||||
const outcome = { companyId: input.companyId, agentId: retry.agentId, runId: retry.id };
|
||||
if (status === null) await recordRunnerApiOutcome(db, { ...outcome, callKey: "b".repeat(64), status: 200 });
|
||||
else await recordRunnerApiOutcome(db, { ...outcome, callKey, status });
|
||||
const call = { companyId: input.companyId, issueId: input.issueId, runId: automatic.id,
|
||||
operationId: receipt.operationId, digest: receipt.digest };
|
||||
const another = { ...call, digest: "another-update" };
|
||||
if (expected === "applied") {
|
||||
expect(await priorRecoveryApiReceipt(db, call)).toMatchObject({ state: "completed",
|
||||
result: { ok: true, status: 200, outcome: "applied" } });
|
||||
expect(await priorRecoveryApiReceipt(db, another)).toBeNull();
|
||||
} else if (expected === "rejected") {
|
||||
expect(await priorRecoveryApiReceipt(db, call)).toBeNull();
|
||||
} else {
|
||||
expect(await priorRecoveryApiReceipt(db, another)).toMatchObject({ state: "pending", result: { outcome: "unknown" } });
|
||||
}
|
||||
});
|
||||
|
||||
it("records the final status of a mutation whose client disconnected and settles its receipt without repeating it", async () => {
|
||||
const { input, retry, automatic } = await automaticSuccessorOfTaskRetry();
|
||||
const callKey = "e".repeat(64);
|
||||
const receipt = { operationId: "PATCH /api/issues/{id}", digest: "late-handoff", state: "completed",
|
||||
result: { ok: false, status: null, error: "api_transport_failure", outcome: "unknown" } };
|
||||
await db.update(heartbeatRuns).set({ resultJson: { apiToolReceipts: { [callKey]: receipt } } }).where(eq(heartbeatRuns.id, retry.id));
|
||||
let writes = 0;
|
||||
let entered!: () => void;
|
||||
const handlerEntered = new Promise<void>(resolve => { entered = resolve; });
|
||||
let releaseHandler!: () => void;
|
||||
const handlerReleased = new Promise<void>(resolve => { releaseHandler = resolve; });
|
||||
const app = express();
|
||||
app.use((req, _res, next) => {
|
||||
req.actor = { type: "agent", source: "agent_jwt", agentId: retry.agentId, companyId: input.companyId, runId: retry.id };
|
||||
next();
|
||||
});
|
||||
app.use(runnerCallOutcomeMiddleware(db));
|
||||
app.patch("/slow", async (req, res) => {
|
||||
const clientGone = new Promise<void>(resolve => req.once("close", () => resolve()));
|
||||
entered();
|
||||
await Promise.all([clientGone, handlerReleased]);
|
||||
writes += 1;
|
||||
res.status(200).json({ ok: true });
|
||||
});
|
||||
const server = app.listen(0, "127.0.0.1");
|
||||
await new Promise<void>(resolve => server.once("listening", () => resolve()));
|
||||
try {
|
||||
const { port } = server.address() as AddressInfo;
|
||||
const client = new AbortController();
|
||||
const response = fetch(`http://127.0.0.1:${port}/slow`, { method: "PATCH", signal: client.signal,
|
||||
headers: { [RUNNER_API_CALL_HEADER]: callKey, "content-type": "application/json" }, body: "{}" });
|
||||
await handlerEntered;
|
||||
client.abort();
|
||||
await expect(response).rejects.toThrow();
|
||||
releaseHandler();
|
||||
const outcomes = () => db.select({ details: activityLog.details }).from(activityLog).where(and(
|
||||
eq(activityLog.runId, retry.id), eq(activityLog.action, "runner.api_completed")));
|
||||
await vi.waitFor(async () => expect(await outcomes()).toHaveLength(1), { timeout: 5_000 });
|
||||
expect(await outcomes()).toEqual([{ details: { callKey, status: 200 } }]);
|
||||
expect(writes).toBe(1);
|
||||
expect(await priorRecoveryApiReceipt(db, { companyId: input.companyId, issueId: input.issueId, runId: automatic.id,
|
||||
operationId: receipt.operationId, digest: receipt.digest })).toMatchObject({ state: "completed",
|
||||
result: { ok: true, status: 200, outcome: "applied" } });
|
||||
} finally {
|
||||
await new Promise<void>(resolve => server.close(() => resolve()));
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["user", "agent", "system"] as const)("lifts an uncertain API receipt fence only after a later %s change", async actorType => {
|
||||
const { input, retry, automatic } = await automaticSuccessorOfTaskRetry();
|
||||
const receipt = { operationId: "PATCH /api/issues/{id}", digest: "handoff", state: "completed",
|
||||
result: { ok: false, status: null, error: "api_transport_failure", outcome: "unknown" } };
|
||||
const call = { companyId: input.companyId, issueId: input.issueId, runId: automatic.id,
|
||||
operationId: receipt.operationId, digest: "reassign" };
|
||||
const change = { companyId: input.companyId, actorType, actorId: actorType === "agent" ? retry.agentId : "board",
|
||||
entityType: "issue", entityId: input.issueId };
|
||||
await db.update(heartbeatRuns).set({ resultJson: { apiToolReceipts: { ["c".repeat(64)]: receipt } } })
|
||||
.where(eq(heartbeatRuns.id, retry.id));
|
||||
await db.insert(activityLog).values({ ...change, action: "issue.comment_added", createdAt: new Date(Date.now() + 5_000) });
|
||||
expect(await priorRecoveryApiReceipt(db, call)).toMatchObject({ state: "pending", result: { outcome: "unknown" } });
|
||||
const finishedAt = new Date(Date.now() + 10_000);
|
||||
await db.update(heartbeatRuns).set({ finishedAt }).where(eq(heartbeatRuns.id, retry.id));
|
||||
await db.insert(activityLog).values({ ...change, action: "issue.read_marked", createdAt: new Date(finishedAt.getTime() + 1_000) });
|
||||
expect(await priorRecoveryApiReceipt(db, call)).toMatchObject({ state: "pending", result: { outcome: "unknown" } });
|
||||
await db.insert(activityLog).values({ ...change, action: "issue.updated", createdAt: new Date(finishedAt.getTime() + 2_000) });
|
||||
if (actorType === "user") expect(await priorRecoveryApiReceipt(db, call)).toBeNull();
|
||||
else expect(await priorRecoveryApiReceipt(db, call)).toMatchObject({ state: "pending", result: { outcome: "unknown" } });
|
||||
});
|
||||
|
||||
it("automatic successors retain uncertain upstream mutation fences after task Retry", async () => {
|
||||
const { input, retry, automatic } = await automaticSuccessorOfTaskRetry();
|
||||
const [application] = await db.insert(toolApplications).values({ companyId: input.companyId,
|
||||
|
||||
Reference in new issue
Block a user