mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
test(restart-chaos): wait for full agent termination in hosted cleanup; count only real outages in continuity (#21176)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness has a hosted mode (#21153) that runs the release-swap proof on a real staging instance > - Its first 2-task smoke from master graded 6 / 6 tasks completed hands-off, but cleanup ended with an error and the API continuity count showed one failure that was not an outage > - Cleanup deleted the agent as soon as its `status` read `terminated`, while its `lifecycleState` was still `terminating`; the server refused with 409 "Complete termination before deleting the agent" > - The continuity count treated a deliberate 404 (an older build without the assigned-work invariant endpoint) as a failed request > - This pull request waits for full termination and counts only real outages > - The benefit is a clean exit code after a clean run and a continuity number that means "the instance did not answer" ## Linked Issues or Issue Description Refs #21153 **Describe the bug** Hosted cleanup reports `remove agent ...: 409 Complete termination before deleting the agent` and exits non-zero after a clean run. The API continuity line counts a deliberate 404 as a failed request. **Steps to reproduce** Run the hosted smoke (`--hosted-releases 0 --hosted-tasks 2`) against an instance that does not have `/api/companies/:id/assigned-work-invariant`. **Expected behavior** Cleanup waits until the agent is fully terminated. Continuity counts only no answer, server errors, and 401, 403, 408 or 429. ## What Changed - `hosted-setup.ts`: agent removal waits until both `status` and `lifecycleState` are `terminated` before the delete. - `hosted-api.ts`: a 400, 404, 409 or 422 answer is a deliberate refusal, not a continuity failure. No answer, 5xx, 401, 403, 408 and 429 still count. - `hosted-main.ts`: `finalizeHosted` passes the poll interval through, for the test. - `README.md`: states the continuity rule. ## Verification - `node server/node_modules/vitest/vitest.mjs run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/hosted-*.test.ts`: 34 tests pass. The setup-failure cleanup test now returns `terminated` with `lifecycleState: terminating` first and checks that the harness reads again before it deletes. The continuity test checks that a 404 is not a failure and a 401 is. - `node server/node_modules/typescript/bin/tsc -p tests/restart-chaos/tsconfig.json`: clean. - Live: the smoke `rc-hosted-smoke2` on a hosted staging instance hit the 409. A second `--hosted-cleanup-only` found nothing left: the agent is terminated and the product keeps its record (cost history), its tasks are hidden, its secret is deleted and its project is archived. ## Risks - Low risk: harness-only. Cleanup can wait up to 120 s per agent for termination, as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits). Default context window; no extended thinking mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
5412162716
commit
f1d25aca14
5 files changed
+15
-9
No files matched your search
@@ -727,8 +727,9 @@ run's agents, the task view from `/api/issues/:id`, pending questions and approv
|
||||
approvals, failure chains and next events from `/api/heartbeat-runs/:id/events`, and duplicated comments and
|
||||
subtasks from comments and the issue list. Per release the table counts failed runs and the time from the old
|
||||
build's last answer to each surviving run's next event. Web and API continuity are reported for both the health
|
||||
probe and every API request the harness made: requests, failures (any answer that is not 2xx, or no answer), slow
|
||||
answers and the longest gap between successful answers. Runs come from the company run list, or from every task's
|
||||
probe and every API request the harness made: requests, failures, slow answers and the longest gap between
|
||||
successful answers. A health probe fails unless `/api/health` answers 200 with a readable body. An API request fails
|
||||
on no answer, a server error, or 401, 403, 408 or 429; a deliberate 400, 404, 409 or 422 answer is not a failure. Runs come from the company run list, or from every task's
|
||||
run list when that list is full, so no task's early runs are lost. A queued or running run with no activity for 5
|
||||
minutes is not a live owner, the same window the drain uses.
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ import type { HealthSample } from "./hosted-release.js";
|
||||
|
||||
export const HOSTED_TOKEN_ENV = "PAPERCLIP_HOSTED_TOKEN";
|
||||
const TRANSIENT_STATUSES = new Set([502, 503, 504]);
|
||||
export const ANSWERED_REFUSALS: ReadonlySet<number> = new Set([400, 404, 409, 422]);
|
||||
|
||||
export function hostedAuthPaths(env: NodeJS.ProcessEnv = process.env): string[] {
|
||||
const paths = [path.join(os.homedir(), ".paperclip", "auth.json")];
|
||||
@@ -48,7 +49,7 @@ export class HostedApi {
|
||||
...(body === undefined ? {} : { "content-type": "application/json" }) },
|
||||
body: body === undefined ? undefined : JSON.stringify(body) });
|
||||
const text = await response.text();
|
||||
this.requests.push({ at: started, ok: response.ok, reachable: response.status < 500, status: response.status, ms: Date.now() - started, method });
|
||||
this.requests.push({ at: started, ok: response.ok || ANSWERED_REFUSALS.has(response.status), reachable: response.status < 500, status: response.status, ms: Date.now() - started, method });
|
||||
return { status: response.ok ? 200 : response.status, text, retryAfterMs: rateLimitDelayMs(response, text) };
|
||||
} catch (error) {
|
||||
this.requests.push({ at: started, ok: false, reachable: false, status: null, ms: Date.now() - started, method });
|
||||
|
||||
@@ -59,11 +59,12 @@ describe("hosted run control", () => {
|
||||
const dir = await tempDir();
|
||||
const writes: string[] = [];
|
||||
let agentStatus = "idle";
|
||||
let lifecycleReads = 0;
|
||||
const api = {
|
||||
get: async <T>(route: string) => {
|
||||
if (route.endsWith("/environments")) return [{ id: "e1", driver: "sandbox" }] as T;
|
||||
if (route.endsWith("/agents")) return [{ id: "a1", name: "RC rc-x opencode" }] as T;
|
||||
if (route === "/api/agents/a1") return { id: "a1", status: agentStatus } as T;
|
||||
if (route === "/api/agents/a1") return { id: "a1", status: agentStatus, lifecycleState: agentStatus === "terminated" && lifecycleReads++ > 0 ? "terminated" : "terminating" } as T;
|
||||
if (route.endsWith("/secrets")) return [{ id: "s1", name: "restart-chaos rc-x OPENROUTER_API_KEY" }] as T;
|
||||
if (route.endsWith("/projects")) return [{ id: "p1", name: "Restart chaos rc-x" }] as T;
|
||||
return [] as T;
|
||||
@@ -85,7 +86,8 @@ describe("hosted run control", () => {
|
||||
await expect(setupHosted({ api, options, companyId: "c1", dir, env: { OPENROUTER_API_KEY: "k" }, record: () => {} })).rejects.toThrow(/agent creation failed/);
|
||||
expect(JSON.parse(await readFile(statePath(dir), "utf8"))).toMatchObject({ companyId: "c1", projectId: "p1", secrets: [{ id: "s1" }] });
|
||||
writes.length = 0;
|
||||
expect(await finalizeHosted({ api, keep: false, label: "rc-x", dir, state: null, record: () => {} })).toMatchObject({ errors: [] });
|
||||
expect(await finalizeHosted({ api, keep: false, label: "rc-x", dir, state: null, record: () => {}, terminatePollMs: 1 })).toMatchObject({ errors: [] });
|
||||
expect(lifecycleReads).toBeGreaterThan(1);
|
||||
expect(writes).toEqual(["PATCH /api/agents/a1", "POST /api/agents/a1/terminate", "DELETE /api/agents/a1", "DELETE /api/secrets/s1", "PATCH /api/projects/p1"]);
|
||||
expect(await finalizeHosted({ api, keep: true, label: "rc-x", dir, state: null, record: () => {} })).toBeNull();
|
||||
});
|
||||
@@ -151,11 +153,13 @@ describe("hosted readers collect complete history", () => {
|
||||
});
|
||||
|
||||
it("counts a refused request as a failure in continuity, not only server errors", async () => {
|
||||
const statuses = [200, 401, 429, 503];
|
||||
const statuses = [200, 401, 404, 429, 503];
|
||||
const api = new HostedApi(hostedTarget(`https://${host}`, host), "t", 1_000, (async () => new Response("{}", { status: statuses.shift()! })) as unknown as typeof fetch);
|
||||
await api.get("/api/health");
|
||||
await api.get("/api/health").catch(() => undefined);
|
||||
expect(continuity(requestSamples(api.requests)).failures).toBeGreaterThanOrEqual(1);
|
||||
expect(api.requests[1]).toMatchObject({ ok: false, reachable: true, status: 401 });
|
||||
await api.get("/api/companies/c1/assigned-work-invariant").catch(() => undefined);
|
||||
expect(api.requests[2]).toMatchObject({ ok: true, status: 404 });
|
||||
});
|
||||
});
|
||||
@@ -43,11 +43,11 @@ export function keepPolling(workload: Pick<HostedWorkload, "poll">, pollMs = POL
|
||||
}
|
||||
|
||||
export async function finalizeHosted(input: { api: Parameters<typeof cleanupHosted>[0]["api"]; keep: boolean; label: string; dir: string; state: HostedState | null;
|
||||
record: (kind: string, data: Record<string, unknown>) => void }): Promise<{ errors: string[] } | null> {
|
||||
record: (kind: string, data: Record<string, unknown>) => void; terminatePollMs?: number }): Promise<{ errors: string[] } | null> {
|
||||
if (input.keep) return null;
|
||||
const state = input.state ?? await loadState(input.dir);
|
||||
if (!state) return null;
|
||||
return await cleanupHosted({ api: input.api, label: input.label, companyId: state.companyId, state, record: input.record })
|
||||
return await cleanupHosted({ api: input.api, label: input.label, companyId: state.companyId, state, record: input.record, terminatePollMs: input.terminatePollMs })
|
||||
.catch(error => ({ errors: [String(error)] }));
|
||||
}
|
||||
|
||||
|
||||
@@ -163,7 +163,7 @@ async function removeAgent(api: SetupApi, agentId: string, pollMs: number): Prom
|
||||
if (current.status !== "terminated") await api.post(`/api/agents/${agentId}/terminate`);
|
||||
for (const until = Date.now() + TERMINATE_WAIT_MS; ;) {
|
||||
const agent = await api.get<Json>(`/api/agents/${agentId}`);
|
||||
if (agent.status === "terminated") break;
|
||||
if (agent.status === "terminated" && (agent.lifecycleState === undefined || agent.lifecycleState === null || agent.lifecycleState === "terminated")) break;
|
||||
if (Date.now() >= until) throw new Error(`agent still ${String(agent.status)} after ${TERMINATE_WAIT_MS / 1000} s; run --hosted-cleanup-only again`);
|
||||
await delay(pollMs);
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user