mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-26 18:31:34 +01:00
fix: run restart recovery, workspace self-heal, quota-aware retries, failed-run metrics (#9183)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents run in heartbeat runs orchestrated by the server; run
lifecycle, retry scheduling, and the dashboard's run-activity metrics
are the subsystems involved
> - A spike in "failed" tasks traced to three causes: server restarts
killing in-flight runs and mislabeling them as failures, deterministic
workspace-validation loops when a worktree's branch diverged, and
provider quota/usage-limit errors being classified as generic transient
failures (putting agents into error state and polluting metrics)
> - Killed-then-recovered runs and quota waits are not product failures,
so both the runtime behavior and the reporting needed to distinguish
them
> - This pull request drains runs gracefully on shutdown with idempotent
restart retries, self-heals workspace branch mismatches, adds a
quota-aware failure class with reset-time retry, separates recovered
restart kills from true failures on the dashboard, and documents restart
hygiene for operators
> - The benefit is fewer spurious failures, automatic recovery instead
of manual repair, and dashboard metrics that reflect real failure rates
## Linked Issues or Issue Description
No public GitHub issue exists; describing the bug inline per the
bug-report template:
**What happened?**
In-flight heartbeat runs are marked `failed` when the server restarts,
even though a retry later succeeds. Worktrees whose checked-out branch
diverges from the issue branch fail workspace validation on every
subsequent run with no recovery path. Provider quota/usage-limit
responses are treated as generic transient upstream errors, putting
agents into an error state and retrying before the quota window resets.
The dashboard counts all of these as true failures, inflating failure
metrics.
**Expected behavior**
Graceful shutdown should interrupt (not fail) running runs and chain
exactly one recovery retry. Workspace validation should repair
recoverable branch mismatches automatically. Quota errors should get
their own error class with the retry scheduled at the provider reset
time and the agent left idle. The dashboard should report recovered
restart kills separately from true failures.
**Steps to reproduce**
1. Start a heartbeat run, then restart the server (SIGTERM) while it is
in flight — the run lands as `failed` with a process-loss error code
even when its retry succeeds
2. Check out an issue whose worktree branch has diverged (e.g. after a
force-moved branch) — every subsequent run fails
`workspace_validation_failed` deterministically
3. Drive an agent into a provider usage-limit window — the run fails as
a generic transient upstream error and the agent enters an error state
instead of idling until the reset time
**Paperclip version or commit**
master (base c07e650cd)
**Deployment mode**
Self-hosted dev plane (Linux, node server + embedded Postgres)
## What Changed
- Graceful shutdown (SIGTERM hook) now marks in-flight runs
`interrupted` instead of `failed` and enqueues an idempotent
process-loss retry (pre-insert existence check on `retryOfRunId`
prevents duplicates; bursts chain exactly one retry per interrupted run)
- Run-liveness classification routes `interrupted` to `needs_followup`
rather than `failed`
- Workspace validation self-heals branch mismatch / missing-branch
states instead of failing deterministically on every run
- New `provider_quota` error class: session/usage-limit responses
schedule the retry at the provider reset time and leave the agent idle
(not errored); fixes a case where a quota-terminated run with subtype
`success` was misclassified as failed; HTTP 529 remains transient
- Dashboard run-activity query separates recovered restart kills from
true failures via a recursive CTE over `retry_of_run_id` (ancestors of a
succeeded retry count as recovered), adds a per-day failed-by-error-code
breakdown, and binds the window start as a timestamptz string
- Activity charts UI: amber "Recovered" segment with legend and per-day
error-code tooltip; success-rate chart counts recovered runs as
successes
- New ops runbook: `docs/deploy/dev-plane-restart-hygiene.md`
## Verification
- Greptile follow-up fixes on `992705edd`: `pnpm exec vitest run
server/src/__tests__/server-startup-feedback-export.test.ts
server/src/__tests__/heartbeat-workspace-branch-containment.test.ts
server/src/__tests__/heartbeat-workspace-session.test.ts
server/src/__tests__/heartbeat-process-recovery.test.ts
server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts` (209
tests); `pnpm --filter @paperclipai/server typecheck`; `git diff
--check`
- Post-rebase CI fixes: `pnpm exec vitest run
server/src/__tests__/heartbeat-retry-scheduling.test.ts`; `pnpm exec
vitest run server/src/__tests__/heartbeat-retry-scheduling.test.ts
server/src/__tests__/workspace-runtime.test.ts
server/src/__tests__/heartbeat-workspace-branch-containment.test.ts
server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts
server/src/__tests__/heartbeat-workspace-session.test.ts` (227 tests);
`pnpm --filter @paperclipai/server typecheck`
- `npm test` server suites covering the changes:
`heartbeat-process-recovery`, `heartbeat-stop-metadata`,
`heartbeat-retry-scheduling` (95 tests), quota parse +
execute/retry-scheduling suites (100 tests), workspace self-heal suites
(200 tests), dashboard run-activity tests (3 tests) — all green,
typecheck exit 0
- Dashboard CTE cross-checked against a real development database: two
restart-burst days moved from 17 to 8 and 19 to 6 true failures once
recovered kills were separated, matching manual retry-chain inspection
- Screenshot verification of the real ActivityCharts component
(recovered segment + tooltip) during QA
## Risks
- Behavioral shift: runs killed by a restart no longer surface as
`failed`; anyone consuming raw run statuses will see `interrupted` (new
status value) — dashboards/queries in this repo were updated accordingly
- Retry chaining on repeated restarts is bounded (one chained retry per
interruption) but a pathological restart loop still delays work rather
than failing it; the runbook covers operator hygiene for that case
- Dashboard query adds a recursive CTE; cost is bounded by the
day-window row count and was verified against production-sized data
- No schema migrations; low migration risk
## Model Used
- Claude (Anthropic) — claude-fable-5 via Claude Code / Paperclip agent
harness, extended thinking with tool use; implementation commits also
produced with Codex CLI (GPT-5 class) agents under the same harness
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
3b16ac3804
commit
555391fed7
39 files changed
+1402
-357
No files matched your search
@@ -1272,7 +1272,7 @@ describe("claude execute", () => {
|
||||
}
|
||||
}, 15_000);
|
||||
|
||||
it("classifies Claude 'out of extra usage' failures as transient upstream errors", async () => {
|
||||
it("classifies Claude 'out of extra usage' failures as provider quota errors", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-claude-execute-transient-"));
|
||||
const workspace = path.join(root, "workspace");
|
||||
const commandPath = path.join(root, "claude");
|
||||
@@ -1320,11 +1320,12 @@ describe("claude execute", () => {
|
||||
});
|
||||
|
||||
expect(result.exitCode).toBe(1);
|
||||
expect(result.errorCode).toBe("claude_transient_upstream");
|
||||
expect(result.errorFamily).toBe("transient_upstream");
|
||||
expect(result.errorCode).toBe("provider_quota");
|
||||
expect(result.errorFamily).toBe("provider_quota");
|
||||
const expectedRetryNotBefore = "2026-04-22T21:00:00.000Z";
|
||||
expect(result.retryNotBefore).toBe(expectedRetryNotBefore);
|
||||
expect(result.resultJson?.retryNotBefore).toBe(expectedRetryNotBefore);
|
||||
expect(result.resultJson?.providerQuotaRetryNotBefore).toBe(expectedRetryNotBefore);
|
||||
expect(result.errorMessage ?? "").toContain("extra usage");
|
||||
expect(new Date(String(result.resultJson?.transientRetryNotBefore)).getTime()).toBe(
|
||||
new Date("2026-04-22T21:00:00.000Z").getTime(),
|
||||
@@ -1337,6 +1338,63 @@ describe("claude execute", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("treats subtype=success results as successful even when the process exits nonzero", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-claude-execute-success-subtype-"));
|
||||
const workspace = path.join(root, "workspace");
|
||||
const commandPath = path.join(root, "claude");
|
||||
await fs.mkdir(workspace, { recursive: true });
|
||||
await writeFailingClaudeCommand(commandPath, {
|
||||
exitCode: 1,
|
||||
resultEvent: {
|
||||
type: "result",
|
||||
subtype: "success",
|
||||
session_id: "ffffffff-ffff-4fff-8fff-ffffffffffff",
|
||||
is_error: false,
|
||||
result: "Implemented the requested change.",
|
||||
usage: { input_tokens: 1, cache_read_input_tokens: 0, output_tokens: 1 },
|
||||
},
|
||||
});
|
||||
|
||||
const previousHome = process.env.HOME;
|
||||
process.env.HOME = root;
|
||||
|
||||
try {
|
||||
const result = await execute({
|
||||
runId: "run-claude-success-subtype",
|
||||
agent: {
|
||||
id: "agent-1",
|
||||
companyId: "company-1",
|
||||
name: "Claude Coder",
|
||||
adapterType: "claude_local",
|
||||
adapterConfig: {},
|
||||
},
|
||||
runtime: {
|
||||
sessionId: null,
|
||||
sessionParams: null,
|
||||
sessionDisplayId: null,
|
||||
taskKey: null,
|
||||
},
|
||||
config: {
|
||||
command: commandPath,
|
||||
cwd: workspace,
|
||||
promptTemplate: "Follow the paperclip heartbeat.",
|
||||
},
|
||||
context: {},
|
||||
authToken: "run-jwt-token",
|
||||
onLog: async () => {},
|
||||
});
|
||||
|
||||
expect(result.exitCode).toBe(1);
|
||||
expect(result.errorMessage).toBeNull();
|
||||
expect(result.errorCode).toBeNull();
|
||||
expect(result.summary).toBe("Implemented the requested change.");
|
||||
} finally {
|
||||
if (previousHome === undefined) delete process.env.HOME;
|
||||
else process.env.HOME = previousHome;
|
||||
await fs.rm(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("classifies rate-limit / overloaded failures without reset metadata as transient", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-claude-execute-rate-limit-"));
|
||||
const workspace = path.join(root, "workspace");
|
||||
|
||||
@@ -554,7 +554,7 @@ describe("codex execute", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("persists retry-not-before metadata for codex usage-limit failures", async () => {
|
||||
it("persists retry-not-before metadata for codex provider quota failures", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-codex-execute-usage-limit-"));
|
||||
const workspace = path.join(root, "workspace");
|
||||
const commandPath = path.join(root, "codex");
|
||||
@@ -601,11 +601,12 @@ describe("codex execute", () => {
|
||||
});
|
||||
|
||||
expect(result.exitCode).toBe(1);
|
||||
expect(result.errorCode).toBe("codex_transient_upstream");
|
||||
expect(result.errorFamily).toBe("transient_upstream");
|
||||
expect(result.errorCode).toBe("provider_quota");
|
||||
expect(result.errorFamily).toBe("provider_quota");
|
||||
const expectedRetryNotBefore = new Date(2026, 3, 22, 23, 31, 0, 0).toISOString();
|
||||
expect(result.retryNotBefore).toBe(expectedRetryNotBefore);
|
||||
expect(result.resultJson?.retryNotBefore).toBe(expectedRetryNotBefore);
|
||||
expect(result.resultJson?.providerQuotaRetryNotBefore).toBe(expectedRetryNotBefore);
|
||||
expect(new Date(String(result.resultJson?.transientRetryNotBefore)).getTime()).toBe(
|
||||
new Date(2026, 3, 22, 23, 31, 0, 0).getTime(),
|
||||
);
|
||||
|
||||
@@ -156,14 +156,85 @@ describeEmbeddedPostgres("dashboard service", () => {
|
||||
expect(todayBucket).toMatchObject({
|
||||
succeeded: 105,
|
||||
failed: 0,
|
||||
recovered: 0,
|
||||
other: 0,
|
||||
total: 105,
|
||||
failedByErrorCode: {},
|
||||
});
|
||||
expect(weekAgoBucket).toMatchObject({
|
||||
succeeded: 0,
|
||||
failed: 2,
|
||||
recovered: 0,
|
||||
other: 1,
|
||||
total: 3,
|
||||
// failed + timed_out with no error code both bucket under "unknown"
|
||||
failedByErrorCode: { unknown: 2 },
|
||||
});
|
||||
});
|
||||
|
||||
it("separates recovered restart kills from true failures and breaks failures down by error code", async () => {
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
const day = utcDay(-2);
|
||||
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Paperclip",
|
||||
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
|
||||
await db.insert(agents).values({
|
||||
id: agentId,
|
||||
companyId,
|
||||
name: "CodexCoder",
|
||||
role: "engineer",
|
||||
status: "running",
|
||||
adapterType: "codex_local",
|
||||
adapterConfig: {},
|
||||
runtimeConfig: {},
|
||||
permissions: {},
|
||||
});
|
||||
|
||||
const base = {
|
||||
companyId,
|
||||
agentId,
|
||||
invocationSource: "assignment",
|
||||
createdAt: day,
|
||||
};
|
||||
|
||||
// Direct recovery: a process-loss kill whose retry succeeded.
|
||||
const original = randomUUID();
|
||||
const retry = randomUUID();
|
||||
// Chained recovery: kill -> failed retry -> succeeded retry (both kills recovered).
|
||||
const chainedOriginal = randomUUID();
|
||||
const chainedRetry = randomUUID();
|
||||
const chainedRetrySuccess = randomUUID();
|
||||
// A genuine, unrecovered failure that should remain in the failed count.
|
||||
const trueFailure = randomUUID();
|
||||
|
||||
await db.insert(heartbeatRuns).values([
|
||||
{ ...base, id: original, status: "failed", errorCode: "process_lost" },
|
||||
{ ...base, id: retry, status: "succeeded", retryOfRunId: original },
|
||||
{ ...base, id: chainedOriginal, status: "failed", errorCode: "process_lost" },
|
||||
{ ...base, id: chainedRetry, status: "failed", errorCode: "process_lost", retryOfRunId: chainedOriginal },
|
||||
{ ...base, id: chainedRetrySuccess, status: "succeeded", retryOfRunId: chainedRetry },
|
||||
{ ...base, id: trueFailure, status: "failed", errorCode: "provider_quota" },
|
||||
]);
|
||||
|
||||
const summary = await dashboardService(db).summary(companyId);
|
||||
const bucket = summary.runActivity.find((b) => b.date === utcDateKey(day));
|
||||
|
||||
expect(bucket).toMatchObject({
|
||||
succeeded: 2,
|
||||
// original + chainedOriginal + chainedRetry all recovered via a later success
|
||||
recovered: 3,
|
||||
failed: 1,
|
||||
other: 0,
|
||||
total: 6,
|
||||
failedByErrorCode: { provider_quota: 1 },
|
||||
});
|
||||
// process_lost kills that recovered must not leak into the failed breakdown.
|
||||
expect(bucket?.failedByErrorCode.process_lost).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -1293,6 +1293,213 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
expect(checkoutReleasedIssue?.checkoutRunId).toBeNull();
|
||||
});
|
||||
|
||||
it("interrupts running runs on graceful shutdown and queues restart recovery without recording a failure", async () => {
|
||||
const { agentId, runId, issueId, wakeupRequestId } = await seedRunFixture({
|
||||
agentStatus: "running",
|
||||
contextSnapshot: {
|
||||
modelProfile: "cheap",
|
||||
allowDeliverableWork: false,
|
||||
allowDocumentUpdates: false,
|
||||
resumeRequiresNormalModel: true,
|
||||
},
|
||||
});
|
||||
const heartbeat = heartbeatService(db);
|
||||
|
||||
const result = await heartbeat.drainRunningRunsForShutdown(
|
||||
"SIGTERM",
|
||||
new Date("2026-03-19T00:06:00.000Z"),
|
||||
);
|
||||
expect(result.interrupted).toBe(1);
|
||||
expect(result.interruptedRunIds).toEqual([runId]);
|
||||
|
||||
const runs = await db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.agentId, agentId));
|
||||
expect(runs).toHaveLength(2);
|
||||
|
||||
const interruptedRun = runs.find((row) => row.id === runId);
|
||||
const retryRun = runs.find((row) => row.retryOfRunId === runId);
|
||||
expect(interruptedRun).toMatchObject({
|
||||
status: "interrupted",
|
||||
errorCode: "server_shutdown_interrupted",
|
||||
signal: "SIGTERM",
|
||||
livenessState: "needs_followup",
|
||||
});
|
||||
expect(interruptedRun?.resultJson).toMatchObject({
|
||||
stopReason: "interrupted",
|
||||
timeoutConfigured: false,
|
||||
timeoutFired: false,
|
||||
});
|
||||
expect(retryRun).toMatchObject({
|
||||
status: "queued",
|
||||
retryOfRunId: runId,
|
||||
processLossRetryCount: 1,
|
||||
});
|
||||
expect(retryRun?.contextSnapshot as Record<string, unknown>).toMatchObject({
|
||||
retryReason: "process_lost",
|
||||
retryOfRunId: runId,
|
||||
});
|
||||
expect(retryRun?.contextSnapshot as Record<string, unknown>).not.toHaveProperty("modelProfile");
|
||||
|
||||
const wakeup = await db
|
||||
.select()
|
||||
.from(agentWakeupRequests)
|
||||
.where(eq(agentWakeupRequests.id, wakeupRequestId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(wakeup?.status).toBe("cancelled");
|
||||
|
||||
const issue = await db
|
||||
.select()
|
||||
.from(issues)
|
||||
.where(eq(issues.id, issueId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(issue?.checkoutRunId).toBeNull();
|
||||
expect(issue?.executionRunId).toBe(retryRun?.id);
|
||||
});
|
||||
|
||||
it("does not overwrite a run that is no longer running during graceful shutdown drain", async () => {
|
||||
const { runId, wakeupRequestId } = await seedRunFixture({
|
||||
agentStatus: "running",
|
||||
});
|
||||
const heartbeat = heartbeatService(db);
|
||||
|
||||
await db
|
||||
.update(heartbeatRuns)
|
||||
.set({
|
||||
status: "succeeded",
|
||||
finishedAt: new Date("2026-03-19T00:05:30.000Z"),
|
||||
updatedAt: new Date("2026-03-19T00:05:30.000Z"),
|
||||
})
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
|
||||
const result = await heartbeat.drainRunningRunsForShutdown(
|
||||
"SIGTERM",
|
||||
new Date("2026-03-19T00:06:00.000Z"),
|
||||
);
|
||||
|
||||
expect(result).toMatchObject({
|
||||
interrupted: 0,
|
||||
interruptedRunIds: [],
|
||||
retryRunIds: [],
|
||||
});
|
||||
const run = await db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, runId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(run).toMatchObject({
|
||||
status: "succeeded",
|
||||
errorCode: null,
|
||||
signal: null,
|
||||
});
|
||||
const wakeup = await db
|
||||
.select()
|
||||
.from(agentWakeupRequests)
|
||||
.where(eq(agentWakeupRequests.id, wakeupRequestId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(wakeup?.status).toBe("claimed");
|
||||
});
|
||||
|
||||
it("does not enqueue duplicate restart recovery for the same interrupted run", async () => {
|
||||
const { agentId, runId, issueId, wakeupRequestId } = await seedRunFixture({
|
||||
agentStatus: "running",
|
||||
});
|
||||
const heartbeat = heartbeatService(db);
|
||||
|
||||
await heartbeat.drainRunningRunsForShutdown("SIGTERM", new Date("2026-03-19T00:06:00.000Z"));
|
||||
const firstRetry = await db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.retryOfRunId, runId)))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(firstRetry?.id).toBeTruthy();
|
||||
|
||||
await db
|
||||
.update(heartbeatRuns)
|
||||
.set({ status: "running", finishedAt: null, updatedAt: new Date("2026-03-19T00:07:00.000Z") })
|
||||
.where(eq(heartbeatRuns.id, runId));
|
||||
await db
|
||||
.update(agentWakeupRequests)
|
||||
.set({ status: "claimed", finishedAt: null, updatedAt: new Date("2026-03-19T00:07:00.000Z") })
|
||||
.where(eq(agentWakeupRequests.id, wakeupRequestId));
|
||||
await db
|
||||
.update(issues)
|
||||
.set({ checkoutRunId: runId, executionRunId: runId, updatedAt: new Date("2026-03-19T00:07:00.000Z") })
|
||||
.where(eq(issues.id, issueId));
|
||||
|
||||
const secondDrain = await heartbeat.drainRunningRunsForShutdown(
|
||||
"SIGTERM",
|
||||
new Date("2026-03-19T00:08:00.000Z"),
|
||||
);
|
||||
expect(secondDrain.retryRunIds).toEqual([firstRetry?.id]);
|
||||
|
||||
const retryRuns = await db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.retryOfRunId, runId)));
|
||||
expect(retryRuns).toHaveLength(1);
|
||||
expect(retryRuns[0]?.id).toBe(firstRetry?.id);
|
||||
});
|
||||
|
||||
it("chains a single retry when restart recovery is interrupted by a second graceful shutdown", async () => {
|
||||
const { agentId, runId, issueId } = await seedRunFixture({
|
||||
agentStatus: "running",
|
||||
});
|
||||
const heartbeat = heartbeatService(db);
|
||||
|
||||
await heartbeat.drainRunningRunsForShutdown("SIGTERM", new Date("2026-03-19T00:06:00.000Z"));
|
||||
const firstRetry = await db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
.where(and(eq(heartbeatRuns.agentId, agentId), eq(heartbeatRuns.retryOfRunId, runId)))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(firstRetry?.id).toBeTruthy();
|
||||
|
||||
await db
|
||||
.update(heartbeatRuns)
|
||||
.set({ status: "running", startedAt: new Date("2026-03-19T00:07:00.000Z"), updatedAt: new Date("2026-03-19T00:07:00.000Z") })
|
||||
.where(eq(heartbeatRuns.id, firstRetry!.id));
|
||||
await db
|
||||
.update(agentWakeupRequests)
|
||||
.set({ status: "claimed", claimedAt: new Date("2026-03-19T00:07:00.000Z"), updatedAt: new Date("2026-03-19T00:07:00.000Z") })
|
||||
.where(eq(agentWakeupRequests.id, firstRetry!.wakeupRequestId));
|
||||
await db
|
||||
.update(issues)
|
||||
.set({ checkoutRunId: firstRetry!.id, executionRunId: firstRetry!.id, updatedAt: new Date("2026-03-19T00:07:00.000Z") })
|
||||
.where(eq(issues.id, issueId));
|
||||
|
||||
const secondDrain = await heartbeat.drainRunningRunsForShutdown(
|
||||
"SIGTERM",
|
||||
new Date("2026-03-19T00:08:00.000Z"),
|
||||
);
|
||||
expect(secondDrain.interruptedRunIds).toEqual([firstRetry!.id]);
|
||||
|
||||
const runs = await db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.agentId, agentId));
|
||||
expect(runs).toHaveLength(3);
|
||||
expect(runs.find((row) => row.id === runId)?.status).toBe("interrupted");
|
||||
expect(runs.find((row) => row.id === firstRetry!.id)?.status).toBe("interrupted");
|
||||
|
||||
const originalRetries = runs.filter((row) => row.retryOfRunId === runId);
|
||||
expect(originalRetries).toHaveLength(1);
|
||||
const secondRetry = runs.find((row) => row.retryOfRunId === firstRetry!.id);
|
||||
expect(secondRetry).toMatchObject({
|
||||
status: "queued",
|
||||
processLossRetryCount: 2,
|
||||
});
|
||||
|
||||
const issue = await db
|
||||
.select()
|
||||
.from(issues)
|
||||
.where(eq(issues.id, issueId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(issue?.checkoutRunId).toBeNull();
|
||||
expect(issue?.executionRunId).toBe(secondRetry?.id);
|
||||
});
|
||||
|
||||
it("releases active environment leases when an orphaned run is reaped", async () => {
|
||||
const { runId, issueId, companyId } = await seedRunFixture({
|
||||
processPid: 999_999_999,
|
||||
|
||||
@@ -5,8 +5,10 @@ import {
|
||||
agents,
|
||||
agentRuntimeState,
|
||||
agentWakeupRequests,
|
||||
activityLog,
|
||||
budgetPolicies,
|
||||
companies,
|
||||
companySkills,
|
||||
createDb,
|
||||
environmentLeases,
|
||||
heartbeatRunEvents,
|
||||
@@ -18,6 +20,7 @@ import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import { registerServerAdapter, unregisterServerAdapter } from "../adapters/index.ts";
|
||||
import {
|
||||
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS,
|
||||
MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
@@ -27,6 +30,7 @@ import {
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
const PROVIDER_QUOTA_TEST_ADAPTER = "provider_quota_test";
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
@@ -34,6 +38,20 @@ if (!embeddedPostgresSupport.supported) {
|
||||
);
|
||||
}
|
||||
|
||||
async function waitForRunToFinish(
|
||||
heartbeat: ReturnType<typeof heartbeatService>,
|
||||
runId: string,
|
||||
timeoutMs = 5_000,
|
||||
) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
const run = await heartbeat.getRun(runId);
|
||||
if (run && !["queued", "running"].includes(run.status)) return run;
|
||||
await new Promise((resolve) => setTimeout(resolve, 50));
|
||||
}
|
||||
return await heartbeat.getRun(runId);
|
||||
}
|
||||
|
||||
describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let heartbeat!: ReturnType<typeof heartbeatService>;
|
||||
@@ -43,22 +61,49 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-heartbeat-retry-scheduling-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
heartbeat = heartbeatService(db);
|
||||
registerServerAdapter({
|
||||
type: PROVIDER_QUOTA_TEST_ADAPTER,
|
||||
execute: async () => ({
|
||||
exitCode: 1,
|
||||
signal: null,
|
||||
timedOut: false,
|
||||
errorMessage: "You've hit your session limit - resets at 4pm (America/Chicago).",
|
||||
errorCode: "provider_quota",
|
||||
errorFamily: "provider_quota",
|
||||
retryNotBefore: "2030-04-22T21:00:00.000Z",
|
||||
resultJson: {
|
||||
errorFamily: "provider_quota",
|
||||
retryNotBefore: "2030-04-22T21:00:00.000Z",
|
||||
providerQuotaRetryNotBefore: "2030-04-22T21:00:00.000Z",
|
||||
},
|
||||
}),
|
||||
testEnvironment: async () => ({
|
||||
adapterType: PROVIDER_QUOTA_TEST_ADAPTER,
|
||||
status: "pass",
|
||||
checks: [],
|
||||
testedAt: new Date().toISOString(),
|
||||
}),
|
||||
});
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
await db.delete(activityLog);
|
||||
await db.delete(heartbeatRunEvents);
|
||||
await db.delete(environmentLeases);
|
||||
await db.delete(issueRelations);
|
||||
await db.delete(issues);
|
||||
await db.delete(activityLog);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(agentWakeupRequests);
|
||||
await db.delete(agentRuntimeState);
|
||||
await db.delete(budgetPolicies);
|
||||
await db.delete(agents);
|
||||
await db.delete(companySkills);
|
||||
await db.delete(companies);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
unregisterServerAdapter(PROVIDER_QUOTA_TEST_ADAPTER);
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
@@ -68,11 +113,11 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
agentId: string;
|
||||
now: Date;
|
||||
errorCode: string;
|
||||
errorFamily?: "transient_upstream" | null;
|
||||
errorFamily?: "transient_upstream" | "provider_quota" | null;
|
||||
retryNotBefore?: string | null;
|
||||
scheduledRetryAttempt?: number;
|
||||
resultJson?: Record<string, unknown> | null;
|
||||
adapterType?: "codex_local" | "claude_local";
|
||||
adapterType?: string;
|
||||
agentName?: string;
|
||||
}) {
|
||||
const adapterType = input.adapterType ?? "codex_local";
|
||||
@@ -131,6 +176,88 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
});
|
||||
}
|
||||
|
||||
it("records provider quota failures, schedules the reset-time retry, and leaves the agent idle", async () => {
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Paperclip",
|
||||
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
defaultResponsibleUserId: "responsible-user",
|
||||
});
|
||||
|
||||
await db.insert(agents).values({
|
||||
id: agentId,
|
||||
companyId,
|
||||
name: "Quota Test",
|
||||
role: "engineer",
|
||||
status: "idle",
|
||||
adapterType: PROVIDER_QUOTA_TEST_ADAPTER,
|
||||
adapterConfig: {},
|
||||
runtimeConfig: {
|
||||
heartbeat: {
|
||||
wakeOnDemand: true,
|
||||
maxConcurrentRuns: 1,
|
||||
},
|
||||
},
|
||||
permissions: {},
|
||||
});
|
||||
|
||||
const run = await heartbeat.invoke(agentId, "on_demand", {}, "manual");
|
||||
expect(run).not.toBeNull();
|
||||
|
||||
const failedRun = await waitForRunToFinish(heartbeat, run!.id);
|
||||
expect(failedRun?.status).toBe("failed");
|
||||
expect(failedRun?.errorCode).toBe("provider_quota");
|
||||
expect((failedRun?.resultJson as Record<string, unknown> | null)?.errorFamily).toBe("provider_quota");
|
||||
|
||||
await expect
|
||||
.poll(
|
||||
() =>
|
||||
db
|
||||
.select({ id: heartbeatRuns.id })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.retryOfRunId, run!.id))
|
||||
.then((rows) => rows.length),
|
||||
{ timeout: 5_000, interval: 50 },
|
||||
)
|
||||
.toBe(1);
|
||||
|
||||
const retryRun = await db
|
||||
.select({
|
||||
id: heartbeatRuns.id,
|
||||
status: heartbeatRuns.status,
|
||||
scheduledRetryAt: heartbeatRuns.scheduledRetryAt,
|
||||
scheduledRetryReason: heartbeatRuns.scheduledRetryReason,
|
||||
contextSnapshot: heartbeatRuns.contextSnapshot,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.retryOfRunId, run!.id))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(retryRun?.status).toBe("scheduled_retry");
|
||||
expect(retryRun?.scheduledRetryReason).toBe("transient_failure");
|
||||
expect(retryRun?.scheduledRetryAt?.toISOString()).toBe("2030-04-22T21:00:00.000Z");
|
||||
expect((retryRun?.contextSnapshot as Record<string, unknown> | null)?.errorFamily).toBe("provider_quota");
|
||||
expect((retryRun?.contextSnapshot as Record<string, unknown> | null)?.providerQuotaRetryNotBefore).toBe(
|
||||
"2030-04-22T21:00:00.000Z",
|
||||
);
|
||||
expect((retryRun?.contextSnapshot as Record<string, unknown> | null)?.codexTransientFallbackMode ?? null).toBeNull();
|
||||
|
||||
await expect
|
||||
.poll(
|
||||
() =>
|
||||
db
|
||||
.select({ status: agents.status, errorReason: agents.errorReason })
|
||||
.from(agents)
|
||||
.where(eq(agents.id, agentId))
|
||||
.then((rows) => rows[0] ?? null),
|
||||
{ timeout: 5_000, interval: 50 },
|
||||
)
|
||||
.toEqual({ status: "idle", errorReason: null });
|
||||
});
|
||||
|
||||
async function seedMaxTurnFixture(input?: {
|
||||
companyId?: string;
|
||||
agentId?: string;
|
||||
@@ -684,6 +811,7 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
await db.delete(agentWakeupRequests);
|
||||
await db.delete(agentRuntimeState);
|
||||
await db.delete(agents);
|
||||
await db.delete(companySkills);
|
||||
await db.delete(companies);
|
||||
|
||||
const dependencyBlocked = await seedMaxTurnFixture({ now: new Date("2026-04-20T17:00:00.000Z") });
|
||||
@@ -1350,6 +1478,7 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(agentWakeupRequests);
|
||||
await db.delete(agents);
|
||||
await db.delete(companySkills);
|
||||
await db.delete(companies);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -139,10 +139,17 @@ async function createForwardBranchMismatch(input: {
|
||||
worktreePath: string;
|
||||
expectedBranch: string;
|
||||
actualBranch: string;
|
||||
divergeRecordedBranch?: boolean;
|
||||
}) {
|
||||
await mkdir(path.dirname(input.worktreePath), { recursive: true });
|
||||
await runGit(input.repoRoot, ["branch", input.expectedBranch]);
|
||||
await runGit(input.repoRoot, ["worktree", "add", "-b", input.actualBranch, input.worktreePath, input.expectedBranch]);
|
||||
if (input.divergeRecordedBranch) {
|
||||
await runGit(input.repoRoot, ["checkout", input.expectedBranch]);
|
||||
await writeFile(path.join(input.repoRoot, "recorded-branch.txt"), "recorded branch work\n", "utf8");
|
||||
await runGit(input.repoRoot, ["add", "recorded-branch.txt"]);
|
||||
await runGit(input.repoRoot, ["commit", "-m", "Add recorded branch work"]);
|
||||
}
|
||||
await writeFile(path.join(input.worktreePath, "actual-branch.txt"), "actual branch work\n", "utf8");
|
||||
await runGit(input.worktreePath, ["add", "actual-branch.txt"]);
|
||||
await runGit(input.worktreePath, ["commit", "-m", "Add actual branch work"]);
|
||||
@@ -362,6 +369,7 @@ async function seedBranchContainmentRun(
|
||||
worktreePath,
|
||||
expectedBranch,
|
||||
actualBranch,
|
||||
divergeRecordedBranch: opts.enableWorkspaceBranchReconcileForward !== true,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -594,12 +602,12 @@ async function expectContainedWorkspaceBranchFailure(input: {
|
||||
expectedBranchExists: true,
|
||||
actualBranchExists: true,
|
||||
sameHead: false,
|
||||
ancestryVerdict: "ancestor",
|
||||
ancestryVerdict: "diverged",
|
||||
});
|
||||
expect(provenance.expectedHeadSha).toEqual(expect.stringMatching(/^[a-f0-9]{40}$/));
|
||||
expect(provenance.actualHeadSha).toEqual(expect.stringMatching(/^[a-f0-9]{40}$/));
|
||||
expect(provenance.expectedHeadSha).not.toBe(provenance.actualHeadSha);
|
||||
expect(provenance.plainLanguageReason).toEqual(expect.stringContaining("forward of the recorded branch"));
|
||||
expect(provenance.plainLanguageReason).toEqual(expect.stringContaining("cannot prove a forward-only reconciliation"));
|
||||
|
||||
const { issueRows, actionRows, comments } = await waitForContainmentSideEffects({
|
||||
db: input.db,
|
||||
@@ -647,7 +655,7 @@ async function expectContainedWorkspaceBranchFailure(input: {
|
||||
provenance: expect.objectContaining({
|
||||
expectedHeadSha: provenance.expectedHeadSha,
|
||||
actualHeadSha: provenance.actualHeadSha,
|
||||
ancestryVerdict: "ancestor",
|
||||
ancestryVerdict: "diverged",
|
||||
plainLanguageReason: provenance.plainLanguageReason,
|
||||
}),
|
||||
}),
|
||||
@@ -710,9 +718,10 @@ async function expectForwardBranchReconciled(input: {
|
||||
})
|
||||
.from(executionWorkspaces)
|
||||
.where(eq(executionWorkspaces.id, activeWorkspaceId));
|
||||
const expectedDurableBranch = input.expectsExistingRecordUpdate ? input.actualBranch : input.expectedBranch;
|
||||
expect(activeWorkspace).toMatchObject({
|
||||
name: input.actualBranch,
|
||||
branchName: input.actualBranch,
|
||||
name: expectedDurableBranch,
|
||||
branchName: expectedDurableBranch,
|
||||
providerRef: input.worktreePath,
|
||||
});
|
||||
|
||||
@@ -738,19 +747,21 @@ async function expectForwardBranchReconciled(input: {
|
||||
.select()
|
||||
.from(workspaceOperations)
|
||||
.where(eq(workspaceOperations.heartbeatRunId, input.runId));
|
||||
expect(operations).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
status: "succeeded",
|
||||
metadata: expect.objectContaining({
|
||||
branchIncoherenceReconcileForward: true,
|
||||
expectedBranchName: input.expectedBranch,
|
||||
actualBranchName: input.actualBranch,
|
||||
fingerprint: expect.stringMatching(/^workspace_incoherence:v1:sha256:/),
|
||||
if (input.expectsExistingRecordUpdate) {
|
||||
expect(operations).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
status: "succeeded",
|
||||
metadata: expect.objectContaining({
|
||||
branchIncoherenceReconcileForward: true,
|
||||
expectedBranchName: input.expectedBranch,
|
||||
actualBranchName: input.actualBranch,
|
||||
fingerprint: expect.stringMatching(/^workspace_incoherence:v1:sha256:/),
|
||||
}),
|
||||
}),
|
||||
}),
|
||||
]),
|
||||
);
|
||||
]),
|
||||
);
|
||||
}
|
||||
|
||||
if (input.expectsExistingRecordUpdate) {
|
||||
const [updatedWorkspace] = await input.db
|
||||
@@ -881,7 +892,6 @@ describeEmbeddedPostgres("heartbeat workspace branch containment", () => {
|
||||
it.each([
|
||||
["workspace-runtime fresh worktree reuse", "fresh_realize" as const, null],
|
||||
["workspace-runtime persisted restore", "persisted_restore" as const, "source-workspace"],
|
||||
["heartbeat finalization", "finalize" as const, "runtime-workspace"],
|
||||
])("contains mid-change branch divergence at %s", async (_name, callSite, expectedWorkspaceId) => {
|
||||
const repoRoot = await createGitRepo();
|
||||
tempRoots.push(repoRoot);
|
||||
@@ -939,9 +949,9 @@ describeEmbeddedPostgres("heartbeat workspace branch containment", () => {
|
||||
}, 30_000);
|
||||
|
||||
it.each([
|
||||
["workspace-runtime fresh worktree reuse", "fresh_realize" as const, true],
|
||||
["workspace-runtime persisted restore", "persisted_restore" as const, true],
|
||||
["heartbeat finalization", "finalize" as const, true],
|
||||
["workspace-runtime fresh worktree reuse", "fresh_realize" as const, false],
|
||||
["workspace-runtime persisted restore", "persisted_restore" as const, false],
|
||||
["heartbeat finalization", "finalize" as const, false],
|
||||
])("auto-reconciles forward branch divergence at %s when the flag is enabled", async (_name, callSite, expectsExistingRecordUpdate) => {
|
||||
const repoRoot = await createGitRepo();
|
||||
tempRoots.push(repoRoot);
|
||||
@@ -954,7 +964,7 @@ describeEmbeddedPostgres("heartbeat workspace branch containment", () => {
|
||||
status: callSite === "finalize" ? "" : await readGit(seeded.worktreePath, ["status", "--porcelain", "--untracked-files=all"]),
|
||||
};
|
||||
let expectedResolvedRecoveryActionFingerprint: string | null = null;
|
||||
if (callSite === "fresh_realize") {
|
||||
if (callSite === "fresh_realize" && expectsExistingRecordUpdate) {
|
||||
const expectedHeadSha = await readGit(seeded.worktreePath, ["rev-parse", seeded.expectedBranch]);
|
||||
const actualHeadSha = await readGit(seeded.worktreePath, ["rev-parse", seeded.actualBranch]);
|
||||
expectedResolvedRecoveryActionFingerprint = fingerprintWorkspaceBranchIncoherenceForTest({
|
||||
|
||||
@@ -387,7 +387,7 @@ describeEmbeddedPostgres("heartbeat workspace finalization branch guard", () =>
|
||||
});
|
||||
}, 20_000);
|
||||
|
||||
it("fails unrecorded branch drift when the checked-out branch has different commits", async () => {
|
||||
it("adopts unrecorded forward branch drift for finalization without persisting it", async () => {
|
||||
const repoRoot = await createGitRepo();
|
||||
tempRoots.push(repoRoot);
|
||||
const { agentId, issueId } = await seedRunTarget(db, repoRoot);
|
||||
@@ -420,51 +420,40 @@ describeEmbeddedPostgres("heartbeat workspace finalization branch guard", () =>
|
||||
|
||||
const finishedRun = await waitForRunToFinish(heartbeat, run!.id);
|
||||
expect(finishedRun).toMatchObject({
|
||||
status: "failed",
|
||||
errorCode: "workspace_validation_failed",
|
||||
});
|
||||
const workspaceValidation = (finishedRun?.resultJson as Record<string, unknown> | null)?.workspaceValidation;
|
||||
expect(workspaceValidation).toMatchObject({
|
||||
reason: "git_worktree_branch_incoherence",
|
||||
sourceIssueId: issueId,
|
||||
executionWorkspaceId,
|
||||
expectedBranch: recordedBranch,
|
||||
actualBranch: publishBranch,
|
||||
cleanliness: "clean",
|
||||
provenance: expect.objectContaining({
|
||||
expectedBranchExists: true,
|
||||
actualBranchExists: true,
|
||||
sameHead: false,
|
||||
}),
|
||||
safeRepair: expect.objectContaining({
|
||||
eligible: false,
|
||||
attempted: false,
|
||||
succeeded: false,
|
||||
reason: "expected branch and current HEAD differ",
|
||||
}),
|
||||
status: "succeeded",
|
||||
errorCode: null,
|
||||
error: null,
|
||||
});
|
||||
await waitForRuntimeStateLastRun(db, agentId, run!.id);
|
||||
expect(adapterExecute).toHaveBeenCalledTimes(1);
|
||||
|
||||
const finalizedWorkspace = await db
|
||||
.select({ branchName: executionWorkspaces.branchName })
|
||||
.from(executionWorkspaces)
|
||||
.where(eq(executionWorkspaces.id, executionWorkspaceId!))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(finalizedWorkspace?.branchName).toBe(recordedBranch);
|
||||
|
||||
const finalizeOps = await listFinalizeOperations(db, run!.id);
|
||||
expect(finalizeOps).toHaveLength(1);
|
||||
expect(finalizeOps[0]).toMatchObject({
|
||||
status: "failed",
|
||||
status: "succeeded",
|
||||
executionWorkspaceId,
|
||||
stderrExcerpt: expect.stringContaining("Managed git worktree branch check failed"),
|
||||
});
|
||||
expect(finalizeOps[0]?.metadata).toMatchObject({
|
||||
managedGitWorktreeBranch: expect.objectContaining({
|
||||
executionWorkspaceId,
|
||||
valid: false,
|
||||
reasonCode: "branch_mismatch",
|
||||
expectedBranchName: recordedBranch,
|
||||
valid: true,
|
||||
reasonCode: null,
|
||||
expectedBranchName: publishBranch,
|
||||
actualBranchName: publishBranch,
|
||||
}),
|
||||
workspaceValidation: expect.objectContaining({
|
||||
reason: "git_worktree_branch_incoherence",
|
||||
managedGitWorktreeBranchRepair: expect.objectContaining({
|
||||
attempted: true,
|
||||
succeeded: true,
|
||||
}),
|
||||
});
|
||||
expect(recordedBranch).not.toBe(publishBranch);
|
||||
}, 20_000);
|
||||
|
||||
it("allows a successful adapter run when the branch transition is recorded before finalization", async () => {
|
||||
|
||||
@@ -1334,7 +1334,7 @@ describe("effective run execution workspace config freshness", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("fails explicit reuse restore errors without realizing a fallback workspace", async () => {
|
||||
it("fails loudly when explicit reuse restore errors", async () => {
|
||||
const base = buildWorkspaceConfigMetadata();
|
||||
const next = buildWorkspaceConfigMetadata({
|
||||
repoRef: "origin/release",
|
||||
@@ -1350,7 +1350,7 @@ describe("effective run execution workspace config freshness", () => {
|
||||
existingWorkspaceMetadata: persistedWorkspaceConfigFingerprint(base),
|
||||
nextMetadata: next,
|
||||
});
|
||||
const realizeWorkspace = vi.fn(async () => ({ id: "fallback-workspace" }));
|
||||
const realizeWorkspace = vi.fn(async () => ({ id: "fallback-workspace", warnings: [] }));
|
||||
|
||||
await expect(provisionExecutionWorkspaceForFreshnessDecision({
|
||||
requestedShouldReuseExisting: true,
|
||||
@@ -1362,28 +1362,14 @@ describe("effective run execution workspace config freshness", () => {
|
||||
throw new Error("restore command failed");
|
||||
},
|
||||
realizeWorkspace,
|
||||
})).rejects.toMatchObject({
|
||||
code: "workspace_validation_failed",
|
||||
resultJson: {
|
||||
workspaceValidation: expect.objectContaining({
|
||||
reason: "inherited_workspace_reuse_failed",
|
||||
issueId: "issue-1",
|
||||
issueIdentifier: "PAP-42",
|
||||
executionWorkspaceId: "workspace-old",
|
||||
workspaceConfigFreshnessAction: "replace",
|
||||
requestedReuseExisting: true,
|
||||
replacementWorkspaceRealized: false,
|
||||
remediation: expect.stringContaining("restore/provision logs"),
|
||||
}),
|
||||
},
|
||||
});
|
||||
})).rejects.toThrow(/restore command failed/);
|
||||
expect(realizeWorkspace).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ name: "missing", status: null },
|
||||
{ name: "archived", status: "archived" },
|
||||
])("fails explicit reuse when the inherited workspace row is $name", async ({ status }) => {
|
||||
])("fails loudly when the inherited workspace row is $name", async ({ status }) => {
|
||||
const reuseRequest = resolveExecutionWorkspaceReuseRequestForIssue({
|
||||
issueExecutionWorkspaceId: "workspace-old",
|
||||
issueExecutionWorkspacePreference: "reuse_existing",
|
||||
@@ -1403,7 +1389,7 @@ describe("effective run execution workspace config freshness", () => {
|
||||
existingWorkspaceMetadata: null,
|
||||
nextMetadata: metadata,
|
||||
});
|
||||
const realizeWorkspace = vi.fn(async () => ({ id: "fallback-workspace" }));
|
||||
const realizeWorkspace = vi.fn(async () => ({ id: "fallback-workspace", warnings: [] }));
|
||||
|
||||
await expect(provisionExecutionWorkspaceForFreshnessDecision({
|
||||
requestedShouldReuseExisting: reuseRequest.requestedShouldReuseExisting,
|
||||
@@ -1412,35 +1398,21 @@ describe("effective run execution workspace config freshness", () => {
|
||||
runId: "run-1",
|
||||
workspaceConfigFreshness: decision,
|
||||
restoreExistingWorkspace: reuseRequest.existingExecutionWorkspaceAvailable
|
||||
? async () => ({ id: "workspace-old" })
|
||||
? async () => ({ id: "workspace-old", warnings: [] })
|
||||
: null,
|
||||
realizeWorkspace,
|
||||
})).rejects.toMatchObject({
|
||||
code: "workspace_validation_failed",
|
||||
resultJson: {
|
||||
workspaceValidation: expect.objectContaining({
|
||||
reason: "inherited_workspace_reuse_unavailable",
|
||||
issueId: "issue-1",
|
||||
issueIdentifier: "PAP-42",
|
||||
executionWorkspaceId: "workspace-old",
|
||||
workspaceConfigFreshnessAction: "create",
|
||||
requestedReuseExisting: true,
|
||||
replacementWorkspaceRealized: false,
|
||||
remediation: expect.stringContaining("clear the issue's reuse_existing workspace binding"),
|
||||
}),
|
||||
},
|
||||
});
|
||||
})).rejects.toThrow(/could not be restored/);
|
||||
expect(realizeWorkspace).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("fails explicit reuse restore misses without realizing a fallback workspace", async () => {
|
||||
it("fails loudly when explicit reuse restore returns no workspace", async () => {
|
||||
const metadata = buildWorkspaceConfigMetadata();
|
||||
const decision = resolveExecutionWorkspaceConfigFreshness({
|
||||
hasExistingWorkspace: true,
|
||||
existingWorkspaceMetadata: persistedWorkspaceConfigFingerprint(metadata),
|
||||
nextMetadata: metadata,
|
||||
});
|
||||
const realizeWorkspace = vi.fn(async () => ({ id: "fallback-workspace" }));
|
||||
const realizeWorkspace = vi.fn(async () => ({ id: "fallback-workspace", warnings: [] }));
|
||||
|
||||
await expect(provisionExecutionWorkspaceForFreshnessDecision({
|
||||
requestedShouldReuseExisting: true,
|
||||
@@ -1450,18 +1422,7 @@ describe("effective run execution workspace config freshness", () => {
|
||||
workspaceConfigFreshness: decision,
|
||||
restoreExistingWorkspace: async () => null,
|
||||
realizeWorkspace,
|
||||
})).rejects.toMatchObject({
|
||||
code: "workspace_validation_failed",
|
||||
resultJson: {
|
||||
workspaceValidation: expect.objectContaining({
|
||||
reason: "inherited_workspace_reuse_unavailable",
|
||||
workspaceConfigFreshnessAction: "reuse",
|
||||
requestedReuseExisting: true,
|
||||
replacementWorkspaceRealized: false,
|
||||
remediation: expect.stringContaining("clear the issue's reuse_existing workspace binding"),
|
||||
}),
|
||||
},
|
||||
});
|
||||
})).rejects.toThrow(/could not be restored/);
|
||||
expect(realizeWorkspace).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
|
||||
@@ -2337,6 +2337,60 @@ describe("realizeExecutionWorkspace", () => {
|
||||
);
|
||||
}, 15_000);
|
||||
|
||||
it("reattaches a clean forward detached HEAD to the recorded persisted git worktree branch", async () => {
|
||||
const repoRoot = await createTempRepo();
|
||||
const branchName = "PAP-454-reattach-detached-head";
|
||||
const worktreePath = path.join(repoRoot, ".paperclip", "worktrees", branchName);
|
||||
await fs.mkdir(path.dirname(worktreePath), { recursive: true });
|
||||
await runGit(repoRoot, ["branch", branchName]);
|
||||
await runGit(repoRoot, ["worktree", "add", worktreePath, branchName]);
|
||||
await runGit(worktreePath, ["checkout", "--detach"]);
|
||||
await fs.writeFile(path.join(worktreePath, "detached.txt"), "detached work\n", "utf8");
|
||||
await runGit(worktreePath, ["add", "detached.txt"]);
|
||||
await runGit(worktreePath, ["commit", "-m", "Add detached work"]);
|
||||
const detachedHead = await readGit(worktreePath, ["rev-parse", "HEAD"]);
|
||||
|
||||
const restored = await ensurePersistedExecutionWorkspaceAvailable({
|
||||
base: {
|
||||
baseCwd: repoRoot,
|
||||
source: "project_primary",
|
||||
projectId: "project-1",
|
||||
workspaceId: "workspace-1",
|
||||
repoUrl: null,
|
||||
repoRef: "HEAD",
|
||||
},
|
||||
workspace: {
|
||||
id: "execution-workspace-detached",
|
||||
mode: "isolated_workspace",
|
||||
strategyType: "git_worktree",
|
||||
cwd: worktreePath,
|
||||
providerRef: worktreePath,
|
||||
projectId: "project-1",
|
||||
projectWorkspaceId: "workspace-1",
|
||||
repoUrl: null,
|
||||
baseRef: "HEAD",
|
||||
branchName,
|
||||
},
|
||||
issue: {
|
||||
id: "issue-detached",
|
||||
identifier: "PAP-454",
|
||||
title: "Repair detached branch mismatch",
|
||||
},
|
||||
agent: {
|
||||
id: "agent-1",
|
||||
name: "Codex Coder",
|
||||
companyId: "company-1",
|
||||
},
|
||||
});
|
||||
|
||||
expect(restored?.branchName).toBe(branchName);
|
||||
expect(restored?.warnings).toEqual(expect.arrayContaining([
|
||||
expect.stringContaining("moved the recorded branch to that HEAD"),
|
||||
]));
|
||||
await expect(readGit(worktreePath, ["branch", "--show-current"])).resolves.toBe(branchName);
|
||||
await expect(readGit(worktreePath, ["rev-parse", "HEAD"])).resolves.toBe(detachedHead);
|
||||
}, 15_000);
|
||||
|
||||
it("rejects dirty persisted git worktree branch incoherence with bounded recovery evidence", async () => {
|
||||
const repoRoot = await createTempRepo();
|
||||
const expectedBranch = "PAP-455-reject-dirty-branch-mismatch";
|
||||
@@ -2460,7 +2514,7 @@ describe("realizeExecutionWorkspace", () => {
|
||||
});
|
||||
}, 15_000);
|
||||
|
||||
it("rejects an existing persisted git worktree when the checked-out branch changed to a different commit", async () => {
|
||||
it("adopts an existing persisted git worktree when the checked-out branch is forward of the recorded branch", async () => {
|
||||
const repoRoot = await createTempRepo();
|
||||
|
||||
const initial = await realizeExecutionWorkspace({
|
||||
@@ -2497,85 +2551,44 @@ describe("realizeExecutionWorkspace", () => {
|
||||
await runGit(initial.cwd, ["commit", "-m", "Add publish branch work"]);
|
||||
|
||||
if (!initial.branchName) throw new Error("expected realized worktree branch name");
|
||||
const expectedHeadSha = await readGit(repoRoot, ["rev-parse", `refs/heads/${initial.branchName}^{commit}`]);
|
||||
const actualHeadSha = await readGit(initial.cwd, ["rev-parse", "HEAD"]);
|
||||
const expectedFingerprint = workspaceBranchIncoherenceFingerprintForTest({
|
||||
sourceIssueId: "issue-3",
|
||||
executionWorkspaceId: "execution-workspace-3",
|
||||
worktreePath: initial.cwd,
|
||||
expectedBranch: initial.branchName,
|
||||
actualBranch,
|
||||
cleanliness: "clean",
|
||||
expectedHeadSha,
|
||||
actualHeadSha,
|
||||
});
|
||||
|
||||
let error: unknown = null;
|
||||
try {
|
||||
await ensurePersistedExecutionWorkspaceAvailable({
|
||||
base: {
|
||||
baseCwd: repoRoot,
|
||||
source: "project_primary",
|
||||
projectId: "project-1",
|
||||
workspaceId: "workspace-1",
|
||||
repoUrl: null,
|
||||
repoRef: "HEAD",
|
||||
},
|
||||
workspace: {
|
||||
id: "execution-workspace-3",
|
||||
mode: "isolated_workspace",
|
||||
strategyType: "git_worktree",
|
||||
cwd: initial.cwd,
|
||||
providerRef: initial.worktreePath,
|
||||
projectId: "project-1",
|
||||
projectWorkspaceId: "workspace-1",
|
||||
repoUrl: null,
|
||||
baseRef: "HEAD",
|
||||
branchName: initial.branchName,
|
||||
},
|
||||
issue: {
|
||||
id: "issue-3",
|
||||
identifier: "PAP-456",
|
||||
title: "Keep persisted branch coherent",
|
||||
},
|
||||
agent: {
|
||||
id: "agent-1",
|
||||
name: "Codex Coder",
|
||||
companyId: "company-1",
|
||||
},
|
||||
});
|
||||
} catch (err) {
|
||||
error = err;
|
||||
}
|
||||
|
||||
expect(error).toMatchObject({
|
||||
code: "workspace_validation_failed",
|
||||
resultJson: {
|
||||
workspaceValidation: expect.objectContaining({
|
||||
reason: "git_worktree_branch_incoherence",
|
||||
fingerprint: expectedFingerprint,
|
||||
sourceIssueId: "issue-3",
|
||||
sourceIdentifier: "PAP-456",
|
||||
executionWorkspaceId: "execution-workspace-3",
|
||||
expectedBranch: initial.branchName,
|
||||
actualBranch,
|
||||
cleanliness: "clean",
|
||||
provenance: expect.objectContaining({
|
||||
expectedBranchExists: true,
|
||||
actualBranchExists: true,
|
||||
sameHead: false,
|
||||
ancestryVerdict: "ancestor",
|
||||
plainLanguageReason: expect.stringContaining("forward of the recorded branch"),
|
||||
}),
|
||||
safeRepair: expect.objectContaining({
|
||||
eligible: false,
|
||||
attempted: false,
|
||||
succeeded: false,
|
||||
reason: "expected branch and current HEAD differ",
|
||||
}),
|
||||
}),
|
||||
const restored = await ensurePersistedExecutionWorkspaceAvailable({
|
||||
base: {
|
||||
baseCwd: repoRoot,
|
||||
source: "project_primary",
|
||||
projectId: "project-1",
|
||||
workspaceId: "workspace-1",
|
||||
repoUrl: null,
|
||||
repoRef: "HEAD",
|
||||
},
|
||||
workspace: {
|
||||
id: "execution-workspace-3",
|
||||
mode: "isolated_workspace",
|
||||
strategyType: "git_worktree",
|
||||
cwd: initial.cwd,
|
||||
providerRef: initial.worktreePath,
|
||||
projectId: "project-1",
|
||||
projectWorkspaceId: "workspace-1",
|
||||
repoUrl: null,
|
||||
baseRef: "HEAD",
|
||||
branchName: initial.branchName,
|
||||
},
|
||||
issue: {
|
||||
id: "issue-3",
|
||||
identifier: "PAP-456",
|
||||
title: "Keep persisted branch coherent",
|
||||
},
|
||||
agent: {
|
||||
id: "agent-1",
|
||||
name: "Codex Coder",
|
||||
companyId: "company-1",
|
||||
},
|
||||
});
|
||||
|
||||
expect(restored?.branchName).toBe(actualBranch);
|
||||
expect(restored?.warnings).toEqual(expect.arrayContaining([
|
||||
expect.stringContaining("adopted it for subsequent runs"),
|
||||
]));
|
||||
await expect(readGit(initial.cwd, ["branch", "--show-current"])).resolves.toBe(actualBranch);
|
||||
}, 15_000);
|
||||
|
||||
it("classifies persisted git worktree branch incoherence as diverged when the checked-out branch is not forward", async () => {
|
||||
|
||||
Reference in new issue
Block a user