test(restart-chaos): wait for full agent termination in hosted cleanup; count only real outages in continuity (#21176)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness has a hosted mode (#21153) that runs the
release-swap proof on a real staging instance
> - Its first 2-task smoke from master graded 6 / 6 tasks completed
hands-off, but cleanup ended with an error and the API continuity count
showed one failure that was not an outage
> - Cleanup deleted the agent as soon as its `status` read `terminated`,
while its `lifecycleState` was still `terminating`; the server refused
with 409 "Complete termination before deleting the agent"
> - The continuity count treated a deliberate 404 (an older build
without the assigned-work invariant endpoint) as a failed request
> - This pull request waits for full termination and counts only real
outages
> - The benefit is a clean exit code after a clean run and a continuity
number that means "the instance did not answer"

## Linked Issues or Issue Description

Refs #21153

**Describe the bug**
Hosted cleanup reports `remove agent ...: 409 Complete termination
before deleting the agent` and exits non-zero after a clean run. The API
continuity line counts a deliberate 404 as a failed request.

**Steps to reproduce**
Run the hosted smoke (`--hosted-releases 0 --hosted-tasks 2`) against an
instance that does not have
`/api/companies/:id/assigned-work-invariant`.

**Expected behavior**
Cleanup waits until the agent is fully terminated. Continuity counts
only no answer, server errors, and 401, 403, 408 or 429.

## What Changed

- `hosted-setup.ts`: agent removal waits until both `status` and
`lifecycleState` are `terminated` before the delete.
- `hosted-api.ts`: a 400, 404, 409 or 422 answer is a deliberate
refusal, not a continuity failure. No answer, 5xx, 401, 403, 408 and 429
still count.
- `hosted-main.ts`: `finalizeHosted` passes the poll interval through,
for the test.
- `README.md`: states the continuity rule.

## Verification

- `node server/node_modules/vitest/vitest.mjs run --config
tests/restart-chaos/vitest.config.ts
tests/restart-chaos/hosted-*.test.ts`: 34 tests pass. The setup-failure
cleanup test now returns `terminated` with `lifecycleState: terminating`
first and checks that the harness reads again before it deletes. The
continuity test checks that a 404 is not a failure and a 401 is.
- `node server/node_modules/typescript/bin/tsc -p
tests/restart-chaos/tsconfig.json`: clean.
- Live: the smoke `rc-hosted-smoke2` on a hosted staging instance hit
the 409. A second `--hosted-cleanup-only` found nothing left: the agent
is terminated and the product keeps its record (cost history), its tasks
are hidden, its secret is deleted and its project is archived.

## Risks

- Low risk: harness-only. Cleanup can wait up to 120 s per agent for
termination, as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits). Default context window; no
extended thinking mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 14:20:29 -06:00
1 parent 5412162716
commit f1d25aca14
5 files changed
+15 -9

No files matched your search

+3 -2
View File
@@ -727,8 +727,9 @@ run's agents, the task view from `/api/issues/:id`, pending questions and approv
approvals, failure chains and next events from `/api/heartbeat-runs/:id/events`, and duplicated comments and
subtasks from comments and the issue list. Per release the table counts failed runs and the time from the old
build's last answer to each surviving run's next event. Web and API continuity are reported for both the health
probe and every API request the harness made: requests, failures (any answer that is not 2xx, or no answer), slow
answers and the longest gap between successful answers. Runs come from the company run list, or from every task's
probe and every API request the harness made: requests, failures, slow answers and the longest gap between
successful answers. A health probe fails unless `/api/health` answers 200 with a readable body. An API request fails
on no answer, a server error, or 401, 403, 408 or 429; a deliberate 400, 404, 409 or 422 answer is not a failure. Runs come from the company run list, or from every task's
run list when that list is full, so no task's early runs are lost. A queued or running run with no activity for 5
minutes is not a live owner, the same window the drain uses.
+2 -1
View File
@@ -8,6 +8,7 @@ import type { HealthSample } from "./hosted-release.js";
export const HOSTED_TOKEN_ENV = "PAPERCLIP_HOSTED_TOKEN";
const TRANSIENT_STATUSES = new Set([502, 503, 504]);
export const ANSWERED_REFUSALS: ReadonlySet<number> = new Set([400, 404, 409, 422]);
export function hostedAuthPaths(env: NodeJS.ProcessEnv = process.env): string[] {
const paths = [path.join(os.homedir(), ".paperclip", "auth.json")];
@@ -48,7 +49,7 @@ export class HostedApi {
...(body === undefined ? {} : { "content-type": "application/json" }) },
body: body === undefined ? undefined : JSON.stringify(body) });
const text = await response.text();
this.requests.push({ at: started, ok: response.ok, reachable: response.status < 500, status: response.status, ms: Date.now() - started, method });
this.requests.push({ at: started, ok: response.ok || ANSWERED_REFUSALS.has(response.status), reachable: response.status < 500, status: response.status, ms: Date.now() - started, method });
return { status: response.ok ? 200 : response.status, text, retryAfterMs: rateLimitDelayMs(response, text) };
} catch (error) {
this.requests.push({ at: started, ok: false, reachable: false, status: null, ms: Date.now() - started, method });
+7 -3
View File
@@ -59,11 +59,12 @@ describe("hosted run control", () => {
const dir = await tempDir();
const writes: string[] = [];
let agentStatus = "idle";
let lifecycleReads = 0;
const api = {
get: async <T>(route: string) => {
if (route.endsWith("/environments")) return [{ id: "e1", driver: "sandbox" }] as T;
if (route.endsWith("/agents")) return [{ id: "a1", name: "RC rc-x opencode" }] as T;
if (route === "/api/agents/a1") return { id: "a1", status: agentStatus } as T;
if (route === "/api/agents/a1") return { id: "a1", status: agentStatus, lifecycleState: agentStatus === "terminated" && lifecycleReads++ > 0 ? "terminated" : "terminating" } as T;
if (route.endsWith("/secrets")) return [{ id: "s1", name: "restart-chaos rc-x OPENROUTER_API_KEY" }] as T;
if (route.endsWith("/projects")) return [{ id: "p1", name: "Restart chaos rc-x" }] as T;
return [] as T;
@@ -85,7 +86,8 @@ describe("hosted run control", () => {
await expect(setupHosted({ api, options, companyId: "c1", dir, env: { OPENROUTER_API_KEY: "k" }, record: () => {} })).rejects.toThrow(/agent creation failed/);
expect(JSON.parse(await readFile(statePath(dir), "utf8"))).toMatchObject({ companyId: "c1", projectId: "p1", secrets: [{ id: "s1" }] });
writes.length = 0;
expect(await finalizeHosted({ api, keep: false, label: "rc-x", dir, state: null, record: () => {} })).toMatchObject({ errors: [] });
expect(await finalizeHosted({ api, keep: false, label: "rc-x", dir, state: null, record: () => {}, terminatePollMs: 1 })).toMatchObject({ errors: [] });
expect(lifecycleReads).toBeGreaterThan(1);
expect(writes).toEqual(["PATCH /api/agents/a1", "POST /api/agents/a1/terminate", "DELETE /api/agents/a1", "DELETE /api/secrets/s1", "PATCH /api/projects/p1"]);
expect(await finalizeHosted({ api, keep: true, label: "rc-x", dir, state: null, record: () => {} })).toBeNull();
});
@@ -151,11 +153,13 @@ describe("hosted readers collect complete history", () => {
});
it("counts a refused request as a failure in continuity, not only server errors", async () => {
const statuses = [200, 401, 429, 503];
const statuses = [200, 401, 404, 429, 503];
const api = new HostedApi(hostedTarget(`https://${host}`, host), "t", 1_000, (async () => new Response("{}", { status: statuses.shift()! })) as unknown as typeof fetch);
await api.get("/api/health");
await api.get("/api/health").catch(() => undefined);
expect(continuity(requestSamples(api.requests)).failures).toBeGreaterThanOrEqual(1);
expect(api.requests[1]).toMatchObject({ ok: false, reachable: true, status: 401 });
await api.get("/api/companies/c1/assigned-work-invariant").catch(() => undefined);
expect(api.requests[2]).toMatchObject({ ok: true, status: 404 });
});
});
+2 -2
View File
@@ -43,11 +43,11 @@ export function keepPolling(workload: Pick<HostedWorkload, "poll">, pollMs = POL
}
export async function finalizeHosted(input: { api: Parameters<typeof cleanupHosted>[0]["api"]; keep: boolean; label: string; dir: string; state: HostedState | null;
record: (kind: string, data: Record<string, unknown>) => void }): Promise<{ errors: string[] } | null> {
record: (kind: string, data: Record<string, unknown>) => void; terminatePollMs?: number }): Promise<{ errors: string[] } | null> {
if (input.keep) return null;
const state = input.state ?? await loadState(input.dir);
if (!state) return null;
return await cleanupHosted({ api: input.api, label: input.label, companyId: state.companyId, state, record: input.record })
return await cleanupHosted({ api: input.api, label: input.label, companyId: state.companyId, state, record: input.record, terminatePollMs: input.terminatePollMs })
.catch(error => ({ errors: [String(error)] }));
}
+1 -1
View File
@@ -163,7 +163,7 @@ async function removeAgent(api: SetupApi, agentId: string, pollMs: number): Prom
if (current.status !== "terminated") await api.post(`/api/agents/${agentId}/terminate`);
for (const until = Date.now() + TERMINATE_WAIT_MS; ;) {
const agent = await api.get<Json>(`/api/agents/${agentId}`);
if (agent.status === "terminated") break;
if (agent.status === "terminated" && (agent.lifecycleState === undefined || agent.lifecycleState === null || agent.lifecycleState === "terminated")) break;
if (Date.now() >= until) throw new Error(`agent still ${String(agent.status)} after ${TERMINATE_WAIT_MS / 1000} s; run --hosted-cleanup-only again`);
await delay(pollMs);
}