mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-22 16:51:28 +02:00
## Thinking Path > - Paperclip is the open source control plane people use to manage AI agents and their work > - Heartbeat execution relies on adapters distinguishing agent failures from failures in the harness running beneath the agent > - Codex MCP transport crashes can kill the CLI after the JSONL protocol has started but before it emits a protocol-terminal event > - Those interrupted streams were left unclassified, so the control plane terminalized the heartbeat as `heartbeat_failed` / `agent_failure` with no continuation > - Agent-level failure is already expressible through the JSONL protocol via an `error` event, `turn.failed`, or `turn.completed`, so an interrupted nonzero exit can be classified structurally without inspecting unstable error strings > - This pull request reports that shape as `codex_harness_crash` in the `transient_upstream` family and routes it through Paperclip's existing bounded retry and recovery-continuation paths > - The benefit is that transient Codex harness failures recover safely without misclassifying quoted agent output or depending on transport-specific wording ## Linked Issues or Issue Description - **What happened:** Codex MCP transport failures, including rmcp worker death, could terminate the CLI mid-turn after protocol output began but before any terminal JSONL event. The run then became an unclassified terminal heartbeat failure with `continuationCount: 0`; this occurred in 3 of 44 L3 Codex-lane trials during the associated benchmark investigation. - **Expected behavior:** a nonzero Codex exit after the protocol starts but before an `error`, `turn.failed`, or `turn.completed` event should be treated as a harness/infrastructure crash and enter the existing bounded retry policy. - **Why structural classification:** transport error strings vary, and stdout may quote agent output that merely discusses network failures. The protocol boundary identifies whether the agent itself produced a terminal result without regex matching. - **Recovery behavior:** `codex_harness_crash` maps to `errorFamily: transient_upstream`, using the existing `same_session` → `safer_invocation` → `fresh_session` ladder plus the recovery-continuation transient-infrastructure path. - Supersedes the regex-based approach in #10150, which is closed. ## What Changed - Added protocol-state tracking that identifies a nonzero exit after protocol start and before any protocol-terminal event as `codex_harness_crash`. - Propagated the structural classification as `transient_upstream` through the Codex adapter. - Added parse unit coverage, including a faithful crash-shaped stream, without matching stderr transport strings. - Added adapter execution coverage using a fake Codex process that emits a protocol prefix and then dies with the observed rmcp stderr line. - Added heartbeat bounded-retry coverage, including the `errorCode`-only fallback, and recovery-continuation classification coverage. ## Verification - `parse.test.ts` — 16 passed. - `codex-local-execute.test.ts` — 16 passed. - `heartbeat-retry-scheduling.test.ts` — 30 passed. - `service.pause-durability.test.ts` — 6 passed. - Server and Codex adapter TypeScript checks passed. - The branch commit is unchanged from the tested and pushed `88f5464d40` handoff. ## Risks - Low risk: the classification requires a nonzero exit after protocol start and before any protocol-terminal event, so normal agent-declared failures and completed turns keep their existing behavior. - The change intentionally broadens recovery for structurally interrupted Codex runs; bounded retry limits still prevent indefinite continuation loops. - No schema, migration, public API, UI, lockfile, or workflow changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex coding agent. The exact runtime model ID and context-window size were not exposed by the execution environment; capabilities used for the implementation included repository analysis, reasoning, code editing, and terminal-based test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details — the pre-existing, already-pushed branch name was explicitly prescribed for this replacement PR - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
329 lines
11 KiB
TypeScript
329 lines
11 KiB
TypeScript
import {
|
||
asString,
|
||
asNumber,
|
||
parseObject,
|
||
parseJson,
|
||
} from "@paperclipai/adapter-utils/server-utils";
|
||
|
||
const CODEX_TRANSIENT_UPSTREAM_RE =
|
||
/(?:we(?:'|’)re\s+currently\s+experiencing\s+high\s+demand|temporary\s+errors|rate[-\s]?limit(?:ed)?|too\s+many\s+requests|\b429\b|server\s+overloaded|service\s+unavailable|try\s+again\s+later)/i;
|
||
const CODEX_REMOTE_COMPACTION_RE = /remote\s+compact\s+task/i;
|
||
const CODEX_USAGE_LIMIT_RE =
|
||
/you(?:'|’)ve hit your usage limit for .+\.\s+switch to another model now,\s+or try again at\s+([^.!\n]+)(?:[.!]|\n|$)/i;
|
||
const CODEX_PROVIDER_QUOTA_RE =
|
||
/(?:you(?:'|’)ve hit your usage limit|usage limit|model (?:is )?at capacity|at capacity for this model|capacity limit)/i;
|
||
const CODEX_REFRESH_TOKEN_REUSED_RE =
|
||
/(?:refresh[_\s-]?token[_\s-]?reused|refresh token (?:has )?already been used|token reuse detected)/i;
|
||
const CODEX_REFRESH_TOKEN_EXPIRED_RE =
|
||
/(?:refresh[_\s-]?token[_\s-]?expired|refresh token (?:has )?expired|expired refresh token)/i;
|
||
const CODEX_REFRESH_TOKEN_INVALIDATED_RE =
|
||
/(?:refresh[_\s-]?token[_\s-]?(?:invalidated|revoked|invalid)|refresh token (?:has been )?(?:invalidated|revoked|invalid)|invalid refresh token|missing bearer)/i;
|
||
const CODEX_OAUTH_INVALID_GRANT_RE = /\binvalid_grant\b/i;
|
||
const CODEX_CONTEXTUAL_REFRESH_AUTH_INVALIDATED_RE =
|
||
/(?:(?:oauth|refresh|access[_\s-]?token|bearer|credential).{0,80}(?:\b401\b|unauthori[sz]ed|\binvalid[\s-]grant\b)|(?:\b401\b|unauthori[sz]ed|\binvalid[\s-]grant\b).{0,80}(?:oauth|refresh|access[_\s-]?token|bearer|credential))/i;
|
||
|
||
export type CodexAuthRefreshFailureClass =
|
||
| "refresh_token_reused"
|
||
| "refresh_token_expired"
|
||
| "refresh_token_invalidated";
|
||
|
||
export function parseCodexJsonl(stdout: string) {
|
||
let sessionId: string | null = null;
|
||
let finalMessage: string | null = null;
|
||
let errorMessage: string | null = null;
|
||
let sawProtocolEvent = false;
|
||
let sawProtocolTerminalEvent = false;
|
||
const usage = {
|
||
inputTokens: 0,
|
||
cachedInputTokens: 0,
|
||
outputTokens: 0,
|
||
};
|
||
|
||
for (const rawLine of stdout.split(/\r?\n/)) {
|
||
const line = rawLine.trim();
|
||
if (!line) continue;
|
||
|
||
const event = parseJson(line);
|
||
if (!event) continue;
|
||
|
||
const type = asString(event.type, "");
|
||
if (type) sawProtocolEvent = true;
|
||
if (type === "error" || type === "turn.completed" || type === "turn.failed") {
|
||
sawProtocolTerminalEvent = true;
|
||
}
|
||
if (type === "thread.started") {
|
||
sessionId = asString(event.thread_id, sessionId ?? "") || sessionId;
|
||
continue;
|
||
}
|
||
|
||
if (type === "error") {
|
||
const msg = asString(event.message, "").trim();
|
||
if (msg) errorMessage = msg;
|
||
continue;
|
||
}
|
||
|
||
if (type === "item.completed") {
|
||
const item = parseObject(event.item);
|
||
if (asString(item.type, "") === "agent_message") {
|
||
const text = asString(item.text, "");
|
||
if (text) finalMessage = text;
|
||
}
|
||
continue;
|
||
}
|
||
|
||
if (type === "turn.completed") {
|
||
const usageObj = parseObject(event.usage);
|
||
usage.inputTokens = asNumber(usageObj.input_tokens, usage.inputTokens);
|
||
usage.cachedInputTokens = asNumber(usageObj.cached_input_tokens, usage.cachedInputTokens);
|
||
usage.outputTokens = asNumber(usageObj.output_tokens, usage.outputTokens);
|
||
continue;
|
||
}
|
||
|
||
if (type === "turn.failed") {
|
||
const err = parseObject(event.error);
|
||
const msg = asString(err.message, "").trim();
|
||
if (msg) errorMessage = msg;
|
||
}
|
||
}
|
||
|
||
return {
|
||
sessionId,
|
||
summary: finalMessage?.trim() ?? "",
|
||
usage,
|
||
usageBasis: "per_run" as const,
|
||
errorMessage,
|
||
sawProtocolEvent,
|
||
sawProtocolTerminalEvent,
|
||
};
|
||
}
|
||
|
||
/**
|
||
* Structural crash detection: the codex CLI can only report an agent-level
|
||
* failure through the JSONL protocol (an `error` event, `turn.failed`, or a
|
||
* finished `turn.completed` followed by a nonzero exit). A nonzero exit after
|
||
* the protocol stream started but before any terminal event means the process
|
||
* died out from under the agent (MCP transport crash, worker panic, killed
|
||
* tool server) — retriable infrastructure, not agent behavior. This
|
||
* deliberately does not match error text: transport failure strings vary, and
|
||
* stdout/stderr can quote agent output that merely discusses network errors.
|
||
*/
|
||
export function isCodexHarnessCrash(input: {
|
||
exitCode: number | null;
|
||
sawProtocolEvent: boolean;
|
||
sawProtocolTerminalEvent: boolean;
|
||
}): boolean {
|
||
if ((input.exitCode ?? 0) === 0) return false;
|
||
return input.sawProtocolEvent && !input.sawProtocolTerminalEvent;
|
||
}
|
||
|
||
export function isCodexUnknownSessionError(stdout: string, stderr: string): boolean {
|
||
const haystack = `${stdout}\n${stderr}`
|
||
.split(/\r?\n/)
|
||
.map((line) => line.trim())
|
||
.filter(Boolean)
|
||
.join("\n");
|
||
return /unknown (session|thread)|session .* not found|thread .* not found|conversation .* not found|missing rollout path for thread|state db missing rollout path|state db returned stale rollout path|no rollout found for thread id/i.test(
|
||
haystack,
|
||
);
|
||
}
|
||
|
||
function buildCodexErrorHaystack(input: {
|
||
stdout?: string | null;
|
||
stderr?: string | null;
|
||
errorMessage?: string | null;
|
||
}): string {
|
||
return [
|
||
input.errorMessage ?? "",
|
||
input.stdout ?? "",
|
||
input.stderr ?? "",
|
||
]
|
||
.join("\n")
|
||
.split(/\r?\n/)
|
||
.map((line) => line.trim())
|
||
.filter(Boolean)
|
||
.join("\n");
|
||
}
|
||
|
||
export function classifyCodexAuthRefreshFailure(input: {
|
||
stdout?: string | null;
|
||
stderr?: string | null;
|
||
errorMessage?: string | null;
|
||
}): CodexAuthRefreshFailureClass | null {
|
||
const haystack = buildCodexErrorHaystack(input);
|
||
|
||
if (CODEX_REFRESH_TOKEN_REUSED_RE.test(haystack)) return "refresh_token_reused";
|
||
if (CODEX_REFRESH_TOKEN_EXPIRED_RE.test(haystack)) return "refresh_token_expired";
|
||
if (CODEX_REFRESH_TOKEN_INVALIDATED_RE.test(haystack)) return "refresh_token_invalidated";
|
||
if (CODEX_OAUTH_INVALID_GRANT_RE.test(haystack)) return "refresh_token_invalidated";
|
||
if (CODEX_CONTEXTUAL_REFRESH_AUTH_INVALIDATED_RE.test(haystack)) return "refresh_token_invalidated";
|
||
return null;
|
||
}
|
||
|
||
function readTimeZoneParts(date: Date, timeZone: string) {
|
||
const values = new Map(
|
||
new Intl.DateTimeFormat("en-US", {
|
||
timeZone,
|
||
hourCycle: "h23",
|
||
year: "numeric",
|
||
month: "2-digit",
|
||
day: "2-digit",
|
||
hour: "2-digit",
|
||
minute: "2-digit",
|
||
}).formatToParts(date).map((part) => [part.type, part.value]),
|
||
);
|
||
return {
|
||
year: Number.parseInt(values.get("year") ?? "", 10),
|
||
month: Number.parseInt(values.get("month") ?? "", 10),
|
||
day: Number.parseInt(values.get("day") ?? "", 10),
|
||
hour: Number.parseInt(values.get("hour") ?? "", 10),
|
||
minute: Number.parseInt(values.get("minute") ?? "", 10),
|
||
};
|
||
}
|
||
|
||
function normalizeResetTimeZone(timeZoneHint: string | null | undefined): string | null {
|
||
const normalized = timeZoneHint?.trim();
|
||
if (!normalized) return null;
|
||
if (/^(?:utc|gmt)$/i.test(normalized)) return "UTC";
|
||
|
||
try {
|
||
new Intl.DateTimeFormat("en-US", { timeZone: normalized }).format(new Date(0));
|
||
return normalized;
|
||
} catch {
|
||
return null;
|
||
}
|
||
}
|
||
|
||
function dateFromTimeZoneWallClock(input: {
|
||
year: number;
|
||
month: number;
|
||
day: number;
|
||
hour: number;
|
||
minute: number;
|
||
timeZone: string;
|
||
}): Date | null {
|
||
let candidate = new Date(Date.UTC(input.year, input.month - 1, input.day, input.hour, input.minute, 0, 0));
|
||
const targetUtc = Date.UTC(input.year, input.month - 1, input.day, input.hour, input.minute, 0, 0);
|
||
|
||
for (let attempt = 0; attempt < 4; attempt += 1) {
|
||
const actual = readTimeZoneParts(candidate, input.timeZone);
|
||
const actualUtc = Date.UTC(actual.year, actual.month - 1, actual.day, actual.hour, actual.minute, 0, 0);
|
||
const offsetMs = targetUtc - actualUtc;
|
||
if (offsetMs === 0) break;
|
||
candidate = new Date(candidate.getTime() + offsetMs);
|
||
}
|
||
|
||
const verified = readTimeZoneParts(candidate, input.timeZone);
|
||
if (
|
||
verified.year !== input.year ||
|
||
verified.month !== input.month ||
|
||
verified.day !== input.day ||
|
||
verified.hour !== input.hour ||
|
||
verified.minute !== input.minute
|
||
) {
|
||
return null;
|
||
}
|
||
|
||
return candidate;
|
||
}
|
||
|
||
function nextClockTimeInTimeZone(input: {
|
||
now: Date;
|
||
hour: number;
|
||
minute: number;
|
||
timeZoneHint: string;
|
||
}): Date | null {
|
||
const timeZone = normalizeResetTimeZone(input.timeZoneHint);
|
||
if (!timeZone) return null;
|
||
|
||
const nowParts = readTimeZoneParts(input.now, timeZone);
|
||
let retryAt = dateFromTimeZoneWallClock({
|
||
year: nowParts.year,
|
||
month: nowParts.month,
|
||
day: nowParts.day,
|
||
hour: input.hour,
|
||
minute: input.minute,
|
||
timeZone,
|
||
});
|
||
if (!retryAt) return null;
|
||
|
||
if (retryAt.getTime() <= input.now.getTime()) {
|
||
const nextDay = new Date(Date.UTC(nowParts.year, nowParts.month - 1, nowParts.day + 1, 0, 0, 0, 0));
|
||
retryAt = dateFromTimeZoneWallClock({
|
||
year: nextDay.getUTCFullYear(),
|
||
month: nextDay.getUTCMonth() + 1,
|
||
day: nextDay.getUTCDate(),
|
||
hour: input.hour,
|
||
minute: input.minute,
|
||
timeZone,
|
||
});
|
||
}
|
||
|
||
return retryAt;
|
||
}
|
||
|
||
function parseLocalClockTime(clockText: string, now: Date): Date | null {
|
||
const normalized = clockText.trim();
|
||
const match = normalized.match(/^(\d{1,2})(?::(\d{2}))?\s*([ap])\.?\s*m\.?(?:\s*\(([^)]+)\)|\s+([A-Z]{2,5}))?$/i);
|
||
if (!match) return null;
|
||
|
||
const hour12 = Number.parseInt(match[1] ?? "", 10);
|
||
const minute = Number.parseInt(match[2] ?? "0", 10);
|
||
if (!Number.isInteger(hour12) || hour12 < 1 || hour12 > 12) return null;
|
||
if (!Number.isInteger(minute) || minute < 0 || minute > 59) return null;
|
||
|
||
let hour24 = hour12 % 12;
|
||
if ((match[3] ?? "").toLowerCase() === "p") hour24 += 12;
|
||
|
||
const timeZoneHint = match[4] ?? match[5];
|
||
if (timeZoneHint) {
|
||
const explicitRetryAt = nextClockTimeInTimeZone({
|
||
now,
|
||
hour: hour24,
|
||
minute,
|
||
timeZoneHint,
|
||
});
|
||
if (explicitRetryAt) return explicitRetryAt;
|
||
}
|
||
|
||
const retryAt = new Date(now);
|
||
retryAt.setHours(hour24, minute, 0, 0);
|
||
if (retryAt.getTime() <= now.getTime()) {
|
||
retryAt.setDate(retryAt.getDate() + 1);
|
||
}
|
||
return retryAt;
|
||
}
|
||
|
||
export function extractCodexRetryNotBefore(input: {
|
||
stdout?: string | null;
|
||
stderr?: string | null;
|
||
errorMessage?: string | null;
|
||
}, now = new Date()): Date | null {
|
||
const haystack = buildCodexErrorHaystack(input);
|
||
const usageLimitMatch = haystack.match(CODEX_USAGE_LIMIT_RE);
|
||
if (!usageLimitMatch) return null;
|
||
return parseLocalClockTime(usageLimitMatch[1] ?? "", now);
|
||
}
|
||
|
||
export function isCodexTransientUpstreamError(input: {
|
||
stdout?: string | null;
|
||
stderr?: string | null;
|
||
errorMessage?: string | null;
|
||
}): boolean {
|
||
const haystack = buildCodexErrorHaystack(input);
|
||
|
||
if (isCodexProviderQuotaError(input)) return false;
|
||
if (!CODEX_TRANSIENT_UPSTREAM_RE.test(haystack)) return false;
|
||
// Keep automatic retries scoped to the observed remote-compaction/high-demand
|
||
// failure shape.
|
||
return CODEX_REMOTE_COMPACTION_RE.test(haystack) || /high\s+demand|temporary\s+errors/i.test(haystack);
|
||
}
|
||
|
||
export function isCodexProviderQuotaError(input: {
|
||
stdout?: string | null;
|
||
stderr?: string | null;
|
||
errorMessage?: string | null;
|
||
}): boolean {
|
||
const haystack = buildCodexErrorHaystack(input);
|
||
return CODEX_PROVIDER_QUOTA_RE.test(haystack) || extractCodexRetryNotBefore(input) != null;
|
||
}
|