Files
PaperClipAI/tests/e2e/disconnect-write-recovery.spec.ts
T
07c6df260b fix(server): schedule archived runner evidence from saved deadlines (#17925)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/85/changes/paperclip/85
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Retired native sessions leave continuity and quarantine evidence on
disk.
> - The old hourly loop mixes evidence retention with backup expiry,
which has different database eligibility rules.
> - Evidence already has saved creation timestamps and local publication
signals.
> - This pull request gives evidence its own registered worker and
actual next deadline.
> - The worker stops filesystem and SQL activity when its evidence
obligations settle.

## Linked Issues or Issue Description

Stack slice 65. Parent: #17923
(`codex/sleep-64-skill-version-retention`). Related foundation: #15625.

**What happened?**

At parent `7b94aa185c29673372646bbf2e07a2fbd16bc394`, an hourly
backup/evidence tick still inspects native state roots. Evidence's
private due time is checked by that recurring tick instead of the shared
scheduler. Empty evidence and future retention windows have no reason to
be rediscovered every hour.

**Expected behavior**

Reconstruct evidence once. Wake from the existing publication signal.
Register the earliest saved retention deadline. Keep busy, failed, or
incomplete evidence recoverable. Preserve backup eligibility as a
separate duty.

**Steps to reproduce**

Start retention with an empty evidence root or archives inside the
seven-day window. The old timer still visits the combined owner. The new
evidence worker disarms when empty, or registers the saved expiry with
no evidence filesystem or SQL calls between relevant changes and that
deadline.

## What Changed

- Add an evidence worker through the delivery coordinator and shared
scheduler. Track startup reconstruction without delaying HTTP readiness.
Unsubscribe local publication before delivery shutdown.
- Select `backups` for the existing backup timer and `evidence` for the
new worker. Existing callers that omit the selection retain combined
behavior. Backup expiration remains the next separate migration.
- Use hourly retries for concrete busy or failed evidence, with earlier
saved expiries and one-second continuations independent. Restore
candidate/removal/byte summaries. Schedule publication at the first
removable instant; preserve legacy manual-tick semantics.
- Expose the existing earliest evidence deadline. Preserve one-second
bounded continuation, concrete failure/busy retries, cleanup-source
archive exclusions, symlink protection, late activity checks, atomic
rename, and interrupted removal recovery.
- Preserve non-ENOENT inspection errors as recovery obligations. Retry a
failed session root in a new evidence pass instead of continuing past it
and forgetting it.
- Register the first removable instant after the inclusive retention
boundary. Keep the existing manual dry-run publication semantics.
- Add real filesystem/isolated PostgreSQL lifecycle tests and
disabled/held-startup wiring cases. Add no table or migration.

The archive and quarantine writers already call
`noteNativeRunnerEvidence`; this PR connects that existing signal to the
scheduler. Shared code owns scheduling, while evidence rules stay with
the retention service. See
[`doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md`](doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md).

## Verification

- Adopted actual prerequisite squash
`b06ea111a254c62e0a55199b010d82e0adb33bf0` from #17923 through normal
merges. Qualified runtime/package/UI/map trees remain unchanged. The
final nine-path diff includes one browser-spec readiness repair.
- Runtime qualification remains **652 passing TypeScript cases across 22
complete files**, plus **29 passing Rust cases** in native-backend,
runtime-notification and configured-environment targets. One actual
closed Pi snapshot launch is intentionally skipped because its external
distribution root is not provided. Accepted source is
`b80eee32f344ae0218d8b791c14f2ea174da09cc`; all runtime and earlier
sleep-test bytes are preserved.
- The complete outage-recovery browser file passes **all five journeys**
on a fresh isolated database after four additive rendered-history
readiness assertions. Every original assertion, outage control and
timeout remains byte-identical when those new lines are removed. The
intentionally unresolved first-session scenario keeps its pre-auth
interaction and delayed-owner coverage. The prior CI/local timeout
evidence and one rejected broader precondition remain in the audit.
- Adapter/Runner/database builds, server/Runner typechecking, Rust
formatting/workspace checks, server build, module boundaries, migration
order and generated TypeScript/Rust profiles pass. PostgreSQL 17
fixtures apply all 413 journal entries through index 414; source hashes
and temporary fixture imports restore exactly.
- **Inherited validation exception:** Copilot profile 81's separate
source-identity check fails on the merged Rust configured-environment
hash. Its inputs and attestations equal upstream and stay unchanged
here. Grok's identity check passes. No release attestation is
regenerated or new live-provider qualification claimed. Required
exact-head CI/security and Greptile 5/5 still must pass.
- [The evidence
audit](doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md)
retains historical failures, counts and limitations. No full local
repository-suite, live provider, external-alarm, measured suspension or
fleet-savings claim. Fresh final-head CI/security, exact-head Greptile
5/5 and actionable Ponytail findings remain separate merge gates.

## Risks

- Live publication is the existing server-owned executor in one process.
Independent filesystem writers need an explicit signal contract. Startup
reconstructs restart state.
- Busy sessions and concrete inspection/removal failures retain retries.
Unknown cleanup-source archives remain protected.
- The existing scan enumerates directory names before applying its
entry/removal budget. This PR preserves that behavior and does not claim
a bounded total directory-listing allocation.
- An in-process deadline cannot wake stopped compute. Cloud sleep-safety
and external-alarm coverage stay unchanged. Physical evidence
reclamation can wait for a supported host wake.
- Backup expiration still has an hourly database eligibility scan. This
slice removes only the archived-evidence duty from that loop and does
not claim full instance suspension.

## Model Used

OpenAI Codex (GPT-6-based coding agent), using repository inspection,
normal Git merges, composition proofs and complete
PostgreSQL/filesystem/startup suites. No subagents were used in this
reconciliation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 11:12:16 -05:00

211 lines
13 KiB
TypeScript

import { test, expect, type APIResponse } from "@playwright/test";
async function json(response: APIResponse) {
const body = await response.text();
expect(response.ok(), `${response.url()}: ${response.status()} ${body}`).toBe(true);
return JSON.parse(body);
}
test("a task title survives an outage and saves after server recovery", async ({ page, request }) => {
const company = await json(await request.post("/api/companies", { data: { name: `Write recovery ${Date.now()}` } }));
try {
const issue = await json(await request.post(`/api/companies/${company.id}/issues`, {
data: { title: "Original task title", status: "backlog" },
}));
const other = await json(await request.post(`/api/companies/${company.id}/issues`, {
data: { title: "Another task stays unchanged", status: "backlog" },
}));
await page.goto(`/${company.issuePrefix}/issues/${issue.identifier}`);
await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 });
const header = page.getByTestId("issue-detail-header");
await expect(header.getByRole("heading", { name: issue.title, exact: true })).toBeVisible();
let unavailable = true;
let failedWrites = 0;
await page.route("**/api/**", async route => {
if (!unavailable) return route.continue();
if (route.request().method() === "PATCH" && [`/api/issues/${issue.id}`, `/api/issues/${issue.identifier}`].includes(new URL(route.request().url()).pathname)) {
failedWrites += 1;
}
if (!failedWrites) return route.continue();
await route.fulfill({ status: 503, contentType: "application/json", body: JSON.stringify({ error: "tenant_app_unavailable" }) });
});
await header.getByRole("heading", { name: issue.title, exact: true }).click();
const title = header.locator("textarea");
await title.fill("A title retained through the outage");
await title.press("Enter");
await expect.poll(() => failedWrites).toBeGreaterThan(0);
await expect(page.locator('[data-connection-status="reconnecting"]')).toBeVisible();
expect((await json(await request.get(`/api/issues/${issue.id}`))).title).toBe(issue.title);
await expect(header).toBeVisible();
unavailable = false;
await expect.poll(async () => (await json(await request.get(`/api/issues/${issue.id}`))).title,
{ timeout: 30_000 }).toBe("A title retained through the outage");
expect((await json(await request.get(`/api/issues/${other.id}`))).title).toBe(other.title);
await page.reload();
await expect(header.getByRole("heading", { name: "A title retained through the outage", exact: true })).toBeVisible();
expect((await json(await request.get(`/api/issues/${issue.id}`))).status).toBe("backlog");
} finally {
await request.patch(`/api/companies/${company.id}`, { data: { status: "archived" } });
}
});
test("two cached tasks keep separate title saves while the browser is offline", async ({ page, request }) => {
test.setTimeout(120_000);
const company = await json(await request.post("/api/companies", { data: { name: `Queued writes ${Date.now()}` } }));
try {
const first = await json(await request.post(`/api/companies/${company.id}/issues`, { data: { title: "First queued task", status: "todo" } }));
const second = await json(await request.post(`/api/companies/${company.id}/issues`, { data: { title: "Second queued task", status: "todo" } }));
await page.goto(`/${company.issuePrefix}/issues/${second.id}`);
await expect(page.getByRole("heading", { name: second.title, exact: true })).toBeVisible();
await page.getByRole("link", { name: "Tasks", exact: true }).first().click();
await page.getByRole("link").filter({ hasText: first.title }).first().click();
await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 });
const header = page.getByTestId("issue-detail-header");
await expect(header.getByRole("heading", { name: first.title, exact: true })).toBeVisible();
const dismissAnnouncement = page.getByRole("button", { name: "Dismiss announcement" });
if (await dismissAnnouncement.isVisible()) await dismissAnnouncement.click();
const secondLink = page.getByRole("link").filter({ hasText: second.title }).first();
await expect(secondLink).toBeVisible();
await page.context().setOffline(true);
await expect(page.locator('[data-connection-status="offline"]')).toBeVisible();
await header.getByRole("heading", { name: first.title, exact: true }).click();
await header.locator("textarea").fill("First title retained offline");
await header.locator("textarea").press("Enter");
await secondLink.focus();
await secondLink.press("Enter", { timeout: 5_000 });
await expect(page).toHaveURL(new RegExp(`${second.id}$`));
await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 });
await expect(header.getByRole("heading", { name: second.title, exact: true })).toBeVisible();
await header.getByRole("heading", { name: second.title, exact: true }).click();
await expect(header.locator("textarea")).toHaveValue(second.title);
await header.locator("textarea").fill("Second title retained offline");
await header.locator("textarea").press("Enter");
expect((await json(await request.get(`/api/issues/${first.id}`))).title).toBe(first.title);
expect((await json(await request.get(`/api/issues/${second.id}`))).title).toBe(second.title);
await page.context().setOffline(false);
await expect.poll(async () => (await json(await request.get(`/api/issues/${first.id}`))).title, { timeout: 30_000 }).toBe("First title retained offline");
await expect.poll(async () => (await json(await request.get(`/api/issues/${second.id}`))).title, { timeout: 30_000 }).toBe("Second title retained offline");
await page.reload();
await expect(header.getByRole("heading", { name: "Second title retained offline", exact: true })).toBeVisible();
} finally {
await page.context().setOffline(false).catch(() => {});
await request.patch(`/api/companies/${company.id}`, { data: { status: "archived" } });
}
});
test("a queued startup save survives the first session lookup during an API outage", async ({ page, request }) => {
const company = await json(await request.post("/api/companies", { data: { name: `Startup recovery ${Date.now()}` } }));
const session = await json(await request.get("/api/auth/get-session"));
let releaseSession!: () => void;
const sessionReady = new Promise<void>(resolve => { releaseSession = resolve; });
const savedTitle = "Saved through first session lookup";
let apiUnavailable = false;
try {
const issue = await json(await request.post(`/api/companies/${company.id}/issues`, {
data: { title: "Startup original title", status: "backlog" },
}));
await page.route("**/api/**", async route => {
const path = new URL(route.request().url()).pathname;
if (apiUnavailable && (path === "/api/health" || (route.request().method() === "PATCH" && path.startsWith("/api/issues/")))) {
return route.fulfill({ status: 503, contentType: "application/json", body: JSON.stringify({ error: "tenant_app_unavailable" }) });
}
await route.continue();
});
await page.route("**/api/auth/get-session", async route => {
await sessionReady;
await route.fulfill({ status: 200, contentType: "application/json", body: JSON.stringify(session) });
});
await page.goto(`/${company.issuePrefix}/issues/${issue.identifier}`);
const header = page.getByTestId("issue-detail-header");
await expect(header.getByRole("heading", { name: issue.title, exact: true })).toBeVisible();
apiUnavailable = true;
await page.evaluate(() => window.dispatchEvent(new Event("offline")));
await expect(page.locator('[data-connection-status="offline"]')).toBeVisible();
await header.getByRole("heading", { name: issue.title, exact: true }).click();
await header.locator("textarea").fill(savedTitle);
await header.locator("textarea").press("Enter");
expect((await json(await request.get(`/api/issues/${issue.id}`))).title).toBe(issue.title);
releaseSession();
await expect.poll(() => page.evaluate(({ owner, title }) => Object.keys(localStorage).some(key =>
key.startsWith(`paperclip:inline-draft:v2:${owner}:`) && JSON.parse(localStorage.getItem(key) ?? "null")?.draft === title),
{ owner: session.user.id, title: savedTitle })).toBe(true);
apiUnavailable = false;
await page.evaluate(() => window.dispatchEvent(new Event("online")));
await expect.poll(async () => (await json(await request.get(`/api/issues/${issue.id}`))).title,
{ timeout: 15_000 }).toBe(savedTitle);
await page.reload();
await expect(header.getByRole("heading", { name: savedTitle, exact: true })).toBeVisible();
} finally {
releaseSession();
apiUnavailable = false;
await page.evaluate(() => window.dispatchEvent(new Event("online"))).catch(() => {});
await request.patch(`/api/companies/${company.id}`, { data: { status: "archived" } });
}
});
test("replaying an accepted comment does not reopen completed work", async ({ page, request }) => {
const company = await json(await request.post("/api/companies", { data: { name: `Comment recovery ${Date.now()}` } }));
try {
const issue = await json(await request.post(`/api/companies/${company.id}/issues`, {
data: { title: "Completed work stays completed", status: "backlog", assigneeAgentId: null, assigneeUserId: null },
}));
const body = "A comment whose response was lost";
const clientRequestId = crypto.randomUUID();
const first = await json(await request.post(`/api/issues/${issue.id}/comments`, { data: { body, clientRequestId } }));
const agent = await json(await request.post(`/api/companies/${company.id}/agents`, {
data: { name: "Completed comment worker", role: "engineer", adapterType: "process", runner: "legacy",
adapterConfig: { command: "/usr/bin/true" }, runtimeConfig: { heartbeat: { enabled: false } } },
}));
await json(await request.patch(`/api/issues/${issue.id}`, { data: { status: "done", assigneeAgentId: agent.id } }));
const receipt = await json(await request.post(`/api/issues/${issue.id}/comments`, { data: { body, clientRequestId } }));
expect(receipt.id).toBe(first.id);
expect((await json(await request.get(`/api/issues/${issue.id}`))).status).toBe("done");
const conflict = await request.post(`/api/issues/${issue.id}/comments`, { data: { body: "Different request", clientRequestId } });
expect(conflict.status()).toBe(409);
expect((await json(await request.get(`/api/issues/${issue.id}`))).status).toBe("done");
const comments = await json(await request.get(`/api/issues/${issue.id}/comments`));
expect(comments.filter((comment: { clientRequestId?: string }) => comment.clientRequestId === clientRequestId)).toHaveLength(1);
const activity = await json(await request.get(`/api/issues/${issue.id}/activity`));
expect(activity.filter((entry: { action: string; details?: { commentId?: string } }) =>
entry.action === "issue.comment_added" && entry.details?.commentId === first.id)).toHaveLength(1);
await page.goto(`/${company.issuePrefix}/issues/${issue.id}`);
await expect(page.getByRole("heading", { name: issue.title, exact: true })).toBeVisible();
await expect(page.getByText(body, { exact: true })).toBeVisible();
} finally {
await request.patch(`/api/companies/${company.id}`, { data: { status: "archived" } });
}
});
test("startup work retries a failed session lookup while other APIs stay healthy", async ({ page, request }) => {
const company = await json(await request.post("/api/companies", { data: { name: `Session recovery ${Date.now()}` } }));
const session = await json(await request.get("/api/auth/get-session"));
let healthy = false, saving = false, failedAfterSave = 0;
try {
const issue = await json(await request.post(`/api/companies/${company.id}/issues`, {
data: { title: "Before session recovery", status: "backlog" },
}));
await page.route("**/api/auth/get-session", async route => {
if (healthy) return route.fulfill({ status: 200, json: session });
if (saving) failedAfterSave += 1;
await route.fulfill({ status: 500, json: { error: "session_lookup_failed" } });
});
await page.goto(`/${company.issuePrefix}/issues/${issue.id}`);
await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 });
const header = page.getByTestId("issue-detail-header");
await header.getByRole("heading", { name: issue.title, exact: true }).click();
await header.locator("textarea").fill("Saved after session recovery");
saving = true;
await header.locator("textarea").press("Enter");
await expect.poll(() => failedAfterSave).toBeGreaterThan(0);
healthy = true;
await expect.poll(async () => (await json(await request.get(`/api/issues/${issue.id}`))).title,
{ timeout: 15_000 }).toBe("Saved after session recovery");
await page.reload();
await expect(header.getByRole("heading", { name: "Saved after session recovery", exact: true })).toBeVisible();
} finally {
await request.patch(`/api/companies/${company.id}`, { data: { status: "archived" } });
}
});