Files
PaperClipAI/packages/adapters/codex-local/src/server/parse.ts
T
DottaandPaperclip 665408c6d0 fix(codex): classify mid-turn harness crashes structurally as retriable infra (#10210)
## Thinking Path

> - Paperclip is the open source control plane people use to manage AI
agents and their work
> - Heartbeat execution relies on adapters distinguishing agent failures
from failures in the harness running beneath the agent
> - Codex MCP transport crashes can kill the CLI after the JSONL
protocol has started but before it emits a protocol-terminal event
> - Those interrupted streams were left unclassified, so the control
plane terminalized the heartbeat as `heartbeat_failed` / `agent_failure`
with no continuation
> - Agent-level failure is already expressible through the JSONL
protocol via an `error` event, `turn.failed`, or `turn.completed`, so an
interrupted nonzero exit can be classified structurally without
inspecting unstable error strings
> - This pull request reports that shape as `codex_harness_crash` in the
`transient_upstream` family and routes it through Paperclip's existing
bounded retry and recovery-continuation paths
> - The benefit is that transient Codex harness failures recover safely
without misclassifying quoted agent output or depending on
transport-specific wording

## Linked Issues or Issue Description

- **What happened:** Codex MCP transport failures, including rmcp worker
death, could terminate the CLI mid-turn after protocol output began but
before any terminal JSONL event. The run then became an unclassified
terminal heartbeat failure with `continuationCount: 0`; this occurred in
3 of 44 L3 Codex-lane trials during the associated benchmark
investigation.
- **Expected behavior:** a nonzero Codex exit after the protocol starts
but before an `error`, `turn.failed`, or `turn.completed` event should
be treated as a harness/infrastructure crash and enter the existing
bounded retry policy.
- **Why structural classification:** transport error strings vary, and
stdout may quote agent output that merely discusses network failures.
The protocol boundary identifies whether the agent itself produced a
terminal result without regex matching.
- **Recovery behavior:** `codex_harness_crash` maps to `errorFamily:
transient_upstream`, using the existing `same_session` →
`safer_invocation` → `fresh_session` ladder plus the
recovery-continuation transient-infrastructure path.
- Supersedes the regex-based approach in #10150, which is closed.

## What Changed

- Added protocol-state tracking that identifies a nonzero exit after
protocol start and before any protocol-terminal event as
`codex_harness_crash`.
- Propagated the structural classification as `transient_upstream`
through the Codex adapter.
- Added parse unit coverage, including a faithful crash-shaped stream,
without matching stderr transport strings.
- Added adapter execution coverage using a fake Codex process that emits
a protocol prefix and then dies with the observed rmcp stderr line.
- Added heartbeat bounded-retry coverage, including the `errorCode`-only
fallback, and recovery-continuation classification coverage.

## Verification

- `parse.test.ts` — 16 passed.
- `codex-local-execute.test.ts` — 16 passed.
- `heartbeat-retry-scheduling.test.ts` — 30 passed.
- `service.pause-durability.test.ts` — 6 passed.
- Server and Codex adapter TypeScript checks passed.
- The branch commit is unchanged from the tested and pushed `88f5464d40`
handoff.

## Risks

- Low risk: the classification requires a nonzero exit after protocol
start and before any protocol-terminal event, so normal agent-declared
failures and completed turns keep their existing behavior.
- The change intentionally broadens recovery for structurally
interrupted Codex runs; bounded retry limits still prevent indefinite
continuation loops.
- No schema, migration, public API, UI, lockfile, or workflow changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex coding agent. The exact runtime model ID and
context-window size were not exposed by the execution environment;
capabilities used for the implementation included repository analysis,
reasoning, code editing, and terminal-based test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
— the pre-existing, already-pushed branch name was explicitly prescribed
for this replacement PR
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-07-24 16:40:47 -05:00

329 lines
11 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import {
asString,
asNumber,
parseObject,
parseJson,
} from "@paperclipai/adapter-utils/server-utils";
const CODEX_TRANSIENT_UPSTREAM_RE =
/(?:we(?:'|’)re\s+currently\s+experiencing\s+high\s+demand|temporary\s+errors|rate[-\s]?limit(?:ed)?|too\s+many\s+requests|\b429\b|server\s+overloaded|service\s+unavailable|try\s+again\s+later)/i;
const CODEX_REMOTE_COMPACTION_RE = /remote\s+compact\s+task/i;
const CODEX_USAGE_LIMIT_RE =
/you(?:'|’)ve hit your usage limit for .+\.\s+switch to another model now,\s+or try again at\s+([^.!\n]+)(?:[.!]|\n|$)/i;
const CODEX_PROVIDER_QUOTA_RE =
/(?:you(?:'|’)ve hit your usage limit|usage limit|model (?:is )?at capacity|at capacity for this model|capacity limit)/i;
const CODEX_REFRESH_TOKEN_REUSED_RE =
/(?:refresh[_\s-]?token[_\s-]?reused|refresh token (?:has )?already been used|token reuse detected)/i;
const CODEX_REFRESH_TOKEN_EXPIRED_RE =
/(?:refresh[_\s-]?token[_\s-]?expired|refresh token (?:has )?expired|expired refresh token)/i;
const CODEX_REFRESH_TOKEN_INVALIDATED_RE =
/(?:refresh[_\s-]?token[_\s-]?(?:invalidated|revoked|invalid)|refresh token (?:has been )?(?:invalidated|revoked|invalid)|invalid refresh token|missing bearer)/i;
const CODEX_OAUTH_INVALID_GRANT_RE = /\binvalid_grant\b/i;
const CODEX_CONTEXTUAL_REFRESH_AUTH_INVALIDATED_RE =
/(?:(?:oauth|refresh|access[_\s-]?token|bearer|credential).{0,80}(?:\b401\b|unauthori[sz]ed|\binvalid[\s-]grant\b)|(?:\b401\b|unauthori[sz]ed|\binvalid[\s-]grant\b).{0,80}(?:oauth|refresh|access[_\s-]?token|bearer|credential))/i;
export type CodexAuthRefreshFailureClass =
| "refresh_token_reused"
| "refresh_token_expired"
| "refresh_token_invalidated";
export function parseCodexJsonl(stdout: string) {
let sessionId: string | null = null;
let finalMessage: string | null = null;
let errorMessage: string | null = null;
let sawProtocolEvent = false;
let sawProtocolTerminalEvent = false;
const usage = {
inputTokens: 0,
cachedInputTokens: 0,
outputTokens: 0,
};
for (const rawLine of stdout.split(/\r?\n/)) {
const line = rawLine.trim();
if (!line) continue;
const event = parseJson(line);
if (!event) continue;
const type = asString(event.type, "");
if (type) sawProtocolEvent = true;
if (type === "error" || type === "turn.completed" || type === "turn.failed") {
sawProtocolTerminalEvent = true;
}
if (type === "thread.started") {
sessionId = asString(event.thread_id, sessionId ?? "") || sessionId;
continue;
}
if (type === "error") {
const msg = asString(event.message, "").trim();
if (msg) errorMessage = msg;
continue;
}
if (type === "item.completed") {
const item = parseObject(event.item);
if (asString(item.type, "") === "agent_message") {
const text = asString(item.text, "");
if (text) finalMessage = text;
}
continue;
}
if (type === "turn.completed") {
const usageObj = parseObject(event.usage);
usage.inputTokens = asNumber(usageObj.input_tokens, usage.inputTokens);
usage.cachedInputTokens = asNumber(usageObj.cached_input_tokens, usage.cachedInputTokens);
usage.outputTokens = asNumber(usageObj.output_tokens, usage.outputTokens);
continue;
}
if (type === "turn.failed") {
const err = parseObject(event.error);
const msg = asString(err.message, "").trim();
if (msg) errorMessage = msg;
}
}
return {
sessionId,
summary: finalMessage?.trim() ?? "",
usage,
usageBasis: "per_run" as const,
errorMessage,
sawProtocolEvent,
sawProtocolTerminalEvent,
};
}
/**
* Structural crash detection: the codex CLI can only report an agent-level
* failure through the JSONL protocol (an `error` event, `turn.failed`, or a
* finished `turn.completed` followed by a nonzero exit). A nonzero exit after
* the protocol stream started but before any terminal event means the process
* died out from under the agent (MCP transport crash, worker panic, killed
* tool server) — retriable infrastructure, not agent behavior. This
* deliberately does not match error text: transport failure strings vary, and
* stdout/stderr can quote agent output that merely discusses network errors.
*/
export function isCodexHarnessCrash(input: {
exitCode: number | null;
sawProtocolEvent: boolean;
sawProtocolTerminalEvent: boolean;
}): boolean {
if ((input.exitCode ?? 0) === 0) return false;
return input.sawProtocolEvent && !input.sawProtocolTerminalEvent;
}
export function isCodexUnknownSessionError(stdout: string, stderr: string): boolean {
const haystack = `${stdout}\n${stderr}`
.split(/\r?\n/)
.map((line) => line.trim())
.filter(Boolean)
.join("\n");
return /unknown (session|thread)|session .* not found|thread .* not found|conversation .* not found|missing rollout path for thread|state db missing rollout path|state db returned stale rollout path|no rollout found for thread id/i.test(
haystack,
);
}
function buildCodexErrorHaystack(input: {
stdout?: string | null;
stderr?: string | null;
errorMessage?: string | null;
}): string {
return [
input.errorMessage ?? "",
input.stdout ?? "",
input.stderr ?? "",
]
.join("\n")
.split(/\r?\n/)
.map((line) => line.trim())
.filter(Boolean)
.join("\n");
}
export function classifyCodexAuthRefreshFailure(input: {
stdout?: string | null;
stderr?: string | null;
errorMessage?: string | null;
}): CodexAuthRefreshFailureClass | null {
const haystack = buildCodexErrorHaystack(input);
if (CODEX_REFRESH_TOKEN_REUSED_RE.test(haystack)) return "refresh_token_reused";
if (CODEX_REFRESH_TOKEN_EXPIRED_RE.test(haystack)) return "refresh_token_expired";
if (CODEX_REFRESH_TOKEN_INVALIDATED_RE.test(haystack)) return "refresh_token_invalidated";
if (CODEX_OAUTH_INVALID_GRANT_RE.test(haystack)) return "refresh_token_invalidated";
if (CODEX_CONTEXTUAL_REFRESH_AUTH_INVALIDATED_RE.test(haystack)) return "refresh_token_invalidated";
return null;
}
function readTimeZoneParts(date: Date, timeZone: string) {
const values = new Map(
new Intl.DateTimeFormat("en-US", {
timeZone,
hourCycle: "h23",
year: "numeric",
month: "2-digit",
day: "2-digit",
hour: "2-digit",
minute: "2-digit",
}).formatToParts(date).map((part) => [part.type, part.value]),
);
return {
year: Number.parseInt(values.get("year") ?? "", 10),
month: Number.parseInt(values.get("month") ?? "", 10),
day: Number.parseInt(values.get("day") ?? "", 10),
hour: Number.parseInt(values.get("hour") ?? "", 10),
minute: Number.parseInt(values.get("minute") ?? "", 10),
};
}
function normalizeResetTimeZone(timeZoneHint: string | null | undefined): string | null {
const normalized = timeZoneHint?.trim();
if (!normalized) return null;
if (/^(?:utc|gmt)$/i.test(normalized)) return "UTC";
try {
new Intl.DateTimeFormat("en-US", { timeZone: normalized }).format(new Date(0));
return normalized;
} catch {
return null;
}
}
function dateFromTimeZoneWallClock(input: {
year: number;
month: number;
day: number;
hour: number;
minute: number;
timeZone: string;
}): Date | null {
let candidate = new Date(Date.UTC(input.year, input.month - 1, input.day, input.hour, input.minute, 0, 0));
const targetUtc = Date.UTC(input.year, input.month - 1, input.day, input.hour, input.minute, 0, 0);
for (let attempt = 0; attempt < 4; attempt += 1) {
const actual = readTimeZoneParts(candidate, input.timeZone);
const actualUtc = Date.UTC(actual.year, actual.month - 1, actual.day, actual.hour, actual.minute, 0, 0);
const offsetMs = targetUtc - actualUtc;
if (offsetMs === 0) break;
candidate = new Date(candidate.getTime() + offsetMs);
}
const verified = readTimeZoneParts(candidate, input.timeZone);
if (
verified.year !== input.year ||
verified.month !== input.month ||
verified.day !== input.day ||
verified.hour !== input.hour ||
verified.minute !== input.minute
) {
return null;
}
return candidate;
}
function nextClockTimeInTimeZone(input: {
now: Date;
hour: number;
minute: number;
timeZoneHint: string;
}): Date | null {
const timeZone = normalizeResetTimeZone(input.timeZoneHint);
if (!timeZone) return null;
const nowParts = readTimeZoneParts(input.now, timeZone);
let retryAt = dateFromTimeZoneWallClock({
year: nowParts.year,
month: nowParts.month,
day: nowParts.day,
hour: input.hour,
minute: input.minute,
timeZone,
});
if (!retryAt) return null;
if (retryAt.getTime() <= input.now.getTime()) {
const nextDay = new Date(Date.UTC(nowParts.year, nowParts.month - 1, nowParts.day + 1, 0, 0, 0, 0));
retryAt = dateFromTimeZoneWallClock({
year: nextDay.getUTCFullYear(),
month: nextDay.getUTCMonth() + 1,
day: nextDay.getUTCDate(),
hour: input.hour,
minute: input.minute,
timeZone,
});
}
return retryAt;
}
function parseLocalClockTime(clockText: string, now: Date): Date | null {
const normalized = clockText.trim();
const match = normalized.match(/^(\d{1,2})(?::(\d{2}))?\s*([ap])\.?\s*m\.?(?:\s*\(([^)]+)\)|\s+([A-Z]{2,5}))?$/i);
if (!match) return null;
const hour12 = Number.parseInt(match[1] ?? "", 10);
const minute = Number.parseInt(match[2] ?? "0", 10);
if (!Number.isInteger(hour12) || hour12 < 1 || hour12 > 12) return null;
if (!Number.isInteger(minute) || minute < 0 || minute > 59) return null;
let hour24 = hour12 % 12;
if ((match[3] ?? "").toLowerCase() === "p") hour24 += 12;
const timeZoneHint = match[4] ?? match[5];
if (timeZoneHint) {
const explicitRetryAt = nextClockTimeInTimeZone({
now,
hour: hour24,
minute,
timeZoneHint,
});
if (explicitRetryAt) return explicitRetryAt;
}
const retryAt = new Date(now);
retryAt.setHours(hour24, minute, 0, 0);
if (retryAt.getTime() <= now.getTime()) {
retryAt.setDate(retryAt.getDate() + 1);
}
return retryAt;
}
export function extractCodexRetryNotBefore(input: {
stdout?: string | null;
stderr?: string | null;
errorMessage?: string | null;
}, now = new Date()): Date | null {
const haystack = buildCodexErrorHaystack(input);
const usageLimitMatch = haystack.match(CODEX_USAGE_LIMIT_RE);
if (!usageLimitMatch) return null;
return parseLocalClockTime(usageLimitMatch[1] ?? "", now);
}
export function isCodexTransientUpstreamError(input: {
stdout?: string | null;
stderr?: string | null;
errorMessage?: string | null;
}): boolean {
const haystack = buildCodexErrorHaystack(input);
if (isCodexProviderQuotaError(input)) return false;
if (!CODEX_TRANSIENT_UPSTREAM_RE.test(haystack)) return false;
// Keep automatic retries scoped to the observed remote-compaction/high-demand
// failure shape.
return CODEX_REMOTE_COMPACTION_RE.test(haystack) || /high\s+demand|temporary\s+errors/i.test(haystack);
}
export function isCodexProviderQuotaError(input: {
stdout?: string | null;
stderr?: string | null;
errorMessage?: string | null;
}): boolean {
const haystack = buildCodexErrorHaystack(input);
return CODEX_PROVIDER_QUOTA_RE.test(haystack) || extractCodexRetryNotBefore(input) != null;
}