From f1d25aca1423f1a7f69c11fa1fe9f344dcf8958e Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:20:29 -0600 Subject: [PATCH] test(restart-chaos): wait for full agent termination in hosted cleanup; count only real outages in continuity (#21176) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness has a hosted mode (#21153) that runs the release-swap proof on a real staging instance > - Its first 2-task smoke from master graded 6 / 6 tasks completed hands-off, but cleanup ended with an error and the API continuity count showed one failure that was not an outage > - Cleanup deleted the agent as soon as its `status` read `terminated`, while its `lifecycleState` was still `terminating`; the server refused with 409 "Complete termination before deleting the agent" > - The continuity count treated a deliberate 404 (an older build without the assigned-work invariant endpoint) as a failed request > - This pull request waits for full termination and counts only real outages > - The benefit is a clean exit code after a clean run and a continuity number that means "the instance did not answer" ## Linked Issues or Issue Description Refs #21153 **Describe the bug** Hosted cleanup reports `remove agent ...: 409 Complete termination before deleting the agent` and exits non-zero after a clean run. The API continuity line counts a deliberate 404 as a failed request. **Steps to reproduce** Run the hosted smoke (`--hosted-releases 0 --hosted-tasks 2`) against an instance that does not have `/api/companies/:id/assigned-work-invariant`. **Expected behavior** Cleanup waits until the agent is fully terminated. Continuity counts only no answer, server errors, and 401, 403, 408 or 429. ## What Changed - `hosted-setup.ts`: agent removal waits until both `status` and `lifecycleState` are `terminated` before the delete. - `hosted-api.ts`: a 400, 404, 409 or 422 answer is a deliberate refusal, not a continuity failure. No answer, 5xx, 401, 403, 408 and 429 still count. - `hosted-main.ts`: `finalizeHosted` passes the poll interval through, for the test. - `README.md`: states the continuity rule. ## Verification - `node server/node_modules/vitest/vitest.mjs run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/hosted-*.test.ts`: 34 tests pass. The setup-failure cleanup test now returns `terminated` with `lifecycleState: terminating` first and checks that the harness reads again before it deletes. The continuity test checks that a 404 is not a failure and a 401 is. - `node server/node_modules/typescript/bin/tsc -p tests/restart-chaos/tsconfig.json`: clean. - Live: the smoke `rc-hosted-smoke2` on a hosted staging instance hit the 409. A second `--hosted-cleanup-only` found nothing left: the agent is terminated and the product keeps its record (cost history), its tasks are hidden, its secret is deleted and its project is archived. ## Risks - Low risk: harness-only. Cleanup can wait up to 120 s per agent for termination, as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits). Default context window; no extended thinking mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- tests/restart-chaos/README.md | 5 +++-- tests/restart-chaos/hosted-api.ts | 3 ++- tests/restart-chaos/hosted-main.test.ts | 10 +++++++--- tests/restart-chaos/hosted-main.ts | 4 ++-- tests/restart-chaos/hosted-setup.ts | 2 +- 5 files changed, 15 insertions(+), 9 deletions(-) diff --git a/tests/restart-chaos/README.md b/tests/restart-chaos/README.md index a58e7afd613..e09f3fcc0e3 100644 --- a/tests/restart-chaos/README.md +++ b/tests/restart-chaos/README.md @@ -727,8 +727,9 @@ run's agents, the task view from `/api/issues/:id`, pending questions and approv approvals, failure chains and next events from `/api/heartbeat-runs/:id/events`, and duplicated comments and subtasks from comments and the issue list. Per release the table counts failed runs and the time from the old build's last answer to each surviving run's next event. Web and API continuity are reported for both the health -probe and every API request the harness made: requests, failures (any answer that is not 2xx, or no answer), slow -answers and the longest gap between successful answers. Runs come from the company run list, or from every task's +probe and every API request the harness made: requests, failures, slow answers and the longest gap between +successful answers. A health probe fails unless `/api/health` answers 200 with a readable body. An API request fails +on no answer, a server error, or 401, 403, 408 or 429; a deliberate 400, 404, 409 or 422 answer is not a failure. Runs come from the company run list, or from every task's run list when that list is full, so no task's early runs are lost. A queued or running run with no activity for 5 minutes is not a live owner, the same window the drain uses. diff --git a/tests/restart-chaos/hosted-api.ts b/tests/restart-chaos/hosted-api.ts index c197a1ad35f..47437b4bfe2 100644 --- a/tests/restart-chaos/hosted-api.ts +++ b/tests/restart-chaos/hosted-api.ts @@ -8,6 +8,7 @@ import type { HealthSample } from "./hosted-release.js"; export const HOSTED_TOKEN_ENV = "PAPERCLIP_HOSTED_TOKEN"; const TRANSIENT_STATUSES = new Set([502, 503, 504]); +export const ANSWERED_REFUSALS: ReadonlySet = new Set([400, 404, 409, 422]); export function hostedAuthPaths(env: NodeJS.ProcessEnv = process.env): string[] { const paths = [path.join(os.homedir(), ".paperclip", "auth.json")]; @@ -48,7 +49,7 @@ export class HostedApi { ...(body === undefined ? {} : { "content-type": "application/json" }) }, body: body === undefined ? undefined : JSON.stringify(body) }); const text = await response.text(); - this.requests.push({ at: started, ok: response.ok, reachable: response.status < 500, status: response.status, ms: Date.now() - started, method }); + this.requests.push({ at: started, ok: response.ok || ANSWERED_REFUSALS.has(response.status), reachable: response.status < 500, status: response.status, ms: Date.now() - started, method }); return { status: response.ok ? 200 : response.status, text, retryAfterMs: rateLimitDelayMs(response, text) }; } catch (error) { this.requests.push({ at: started, ok: false, reachable: false, status: null, ms: Date.now() - started, method }); diff --git a/tests/restart-chaos/hosted-main.test.ts b/tests/restart-chaos/hosted-main.test.ts index 0d283fe0937..1322dfd8aa7 100644 --- a/tests/restart-chaos/hosted-main.test.ts +++ b/tests/restart-chaos/hosted-main.test.ts @@ -59,11 +59,12 @@ describe("hosted run control", () => { const dir = await tempDir(); const writes: string[] = []; let agentStatus = "idle"; + let lifecycleReads = 0; const api = { get: async (route: string) => { if (route.endsWith("/environments")) return [{ id: "e1", driver: "sandbox" }] as T; if (route.endsWith("/agents")) return [{ id: "a1", name: "RC rc-x opencode" }] as T; - if (route === "/api/agents/a1") return { id: "a1", status: agentStatus } as T; + if (route === "/api/agents/a1") return { id: "a1", status: agentStatus, lifecycleState: agentStatus === "terminated" && lifecycleReads++ > 0 ? "terminated" : "terminating" } as T; if (route.endsWith("/secrets")) return [{ id: "s1", name: "restart-chaos rc-x OPENROUTER_API_KEY" }] as T; if (route.endsWith("/projects")) return [{ id: "p1", name: "Restart chaos rc-x" }] as T; return [] as T; @@ -85,7 +86,8 @@ describe("hosted run control", () => { await expect(setupHosted({ api, options, companyId: "c1", dir, env: { OPENROUTER_API_KEY: "k" }, record: () => {} })).rejects.toThrow(/agent creation failed/); expect(JSON.parse(await readFile(statePath(dir), "utf8"))).toMatchObject({ companyId: "c1", projectId: "p1", secrets: [{ id: "s1" }] }); writes.length = 0; - expect(await finalizeHosted({ api, keep: false, label: "rc-x", dir, state: null, record: () => {} })).toMatchObject({ errors: [] }); + expect(await finalizeHosted({ api, keep: false, label: "rc-x", dir, state: null, record: () => {}, terminatePollMs: 1 })).toMatchObject({ errors: [] }); + expect(lifecycleReads).toBeGreaterThan(1); expect(writes).toEqual(["PATCH /api/agents/a1", "POST /api/agents/a1/terminate", "DELETE /api/agents/a1", "DELETE /api/secrets/s1", "PATCH /api/projects/p1"]); expect(await finalizeHosted({ api, keep: true, label: "rc-x", dir, state: null, record: () => {} })).toBeNull(); }); @@ -151,11 +153,13 @@ describe("hosted readers collect complete history", () => { }); it("counts a refused request as a failure in continuity, not only server errors", async () => { - const statuses = [200, 401, 429, 503]; + const statuses = [200, 401, 404, 429, 503]; const api = new HostedApi(hostedTarget(`https://${host}`, host), "t", 1_000, (async () => new Response("{}", { status: statuses.shift()! })) as unknown as typeof fetch); await api.get("/api/health"); await api.get("/api/health").catch(() => undefined); expect(continuity(requestSamples(api.requests)).failures).toBeGreaterThanOrEqual(1); expect(api.requests[1]).toMatchObject({ ok: false, reachable: true, status: 401 }); + await api.get("/api/companies/c1/assigned-work-invariant").catch(() => undefined); + expect(api.requests[2]).toMatchObject({ ok: true, status: 404 }); }); }); diff --git a/tests/restart-chaos/hosted-main.ts b/tests/restart-chaos/hosted-main.ts index 1df1e1b5fa4..58d8d01361d 100644 --- a/tests/restart-chaos/hosted-main.ts +++ b/tests/restart-chaos/hosted-main.ts @@ -43,11 +43,11 @@ export function keepPolling(workload: Pick, pollMs = POL } export async function finalizeHosted(input: { api: Parameters[0]["api"]; keep: boolean; label: string; dir: string; state: HostedState | null; - record: (kind: string, data: Record) => void }): Promise<{ errors: string[] } | null> { + record: (kind: string, data: Record) => void; terminatePollMs?: number }): Promise<{ errors: string[] } | null> { if (input.keep) return null; const state = input.state ?? await loadState(input.dir); if (!state) return null; - return await cleanupHosted({ api: input.api, label: input.label, companyId: state.companyId, state, record: input.record }) + return await cleanupHosted({ api: input.api, label: input.label, companyId: state.companyId, state, record: input.record, terminatePollMs: input.terminatePollMs }) .catch(error => ({ errors: [String(error)] })); } diff --git a/tests/restart-chaos/hosted-setup.ts b/tests/restart-chaos/hosted-setup.ts index 1235829b2f7..5a70f986187 100644 --- a/tests/restart-chaos/hosted-setup.ts +++ b/tests/restart-chaos/hosted-setup.ts @@ -163,7 +163,7 @@ async function removeAgent(api: SetupApi, agentId: string, pollMs: number): Prom if (current.status !== "terminated") await api.post(`/api/agents/${agentId}/terminate`); for (const until = Date.now() + TERMINATE_WAIT_MS; ;) { const agent = await api.get(`/api/agents/${agentId}`); - if (agent.status === "terminated") break; + if (agent.status === "terminated" && (agent.lifecycleState === undefined || agent.lifecycleState === null || agent.lifecycleState === "terminated")) break; if (Date.now() >= until) throw new Error(`agent still ${String(agent.status)} after ${TERMINATE_WAIT_MS / 1000} s; run --hosted-cleanup-only again`); await delay(pollMs); }