diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index a55eed6b7d5..b9fba65827b 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -3525,3 +3525,15 @@ writer inventory, module contracts, local acceptance, and limitations. Remote review and checks remain pending. The user's October 27 instruction now authorizes merging qualifying sleep PRs in dependency order after Greptile 5/5 and any present Ponytail review passes, with all required CI checks passing. + +## Slice 65 — archived runner evidence retention (2026-10-27) + +Split evidence retention from native-harness backup eligibility in their shared +legacy service. Evidence now reconstructs once, wakes from the existing archive/ +quarantine publication signal, and uses saved retention timestamps. Busy or failed +items retain recovery and bounded continuation. The backup timer selects only its +own duty and remains for the next slice. See the +[evidence audit](2026-10-27-runner-evidence-retention-sleep-audit.md) for producer, +clock-boundary, interrupted-cursor, writer-model, and verification evidence. +Local acceptance passes 158 cases plus server typechecking/build. Remote review +and CI are pending; this is not a deployed sleep claim. diff --git a/doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md b/doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md new file mode 100644 index 00000000000..ae9688ba345 --- /dev/null +++ b/doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md @@ -0,0 +1,290 @@ +# Runner evidence retention scheduling + +Slice 65, parent slice 64 (#17923) at +`d2ccbcdfc9118b8841299da65e94e487807e73ab` before its review repair. +This slice owns archived runner evidence. Native-harness backup expiration is a +separate remaining duty, although the old implementation inspects both in one loop. +The default branch was observed at `50b932a142216331b8d476042b454bc123739c40`. +The stack's pinned integration baseline remains `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`; +the later default-branch changes require final stack integration and verification. + +## Obligation and existing persistence + +Continuity-break archives and identity quarantine directories preserve evidence +from a retired native session. Their names contain the creation timestamp. The +existing retention service falls back to mtime for a timestamp in the future, +preserves cleanup-source quarantine archives, refuses symlinks, checks session +activity twice before removal, and moves eligible evidence to a unique expired +name before recursive removal. Interrupted expired directories are recovered. + +These filesystem facts already persist the retention obligation. There is no +reason to add a database table. No database work is needed for evidence inspection. +The existing hourly tick still invokes the backup-expiration path, whose eligibility +also depends on issues, runs, finalizations, and leases. Moving those two duties +independently keeps each PR's trigger and correctness claim reviewable. + +## Triggers and module ownership + +- Startup reconstructs evidence once using the existing bounded session/archive/ + quarantine continuation state. +- `quarantineRunnerdStateRoot` publishes `noteNativeRunnerEvidence` after the + quarantine rename. `archiveContinuityState` publishes the same signal when it + creates the continuity archive, before it moves the retired state. The saved + retention deadline is days later; the signal does not authorize immediate + deletion or claim that the archive's write is complete. +- A publication adds its timestamp to the existing earliest evidence deadline and + wakes the registered worker. Earlier deadlines continue to win. The next due + inspection reconstructs all retained evidence, including multiple writes. +- A recent item contributes its saved creation time plus the configured retention + window and one millisecond, because the existing protection comparison is + inclusive at the window boundary. +- Busy sessions, incomplete scans, and failed inspection/removal retain actual + recovery obligations. Bounded continuations run after one second. Concrete + failed or busy evidence retries use a one-hour worker retry; an earlier saved + expiry or new publication can still wake the worker sooner. +- Empty evidence, ignored cleanup-source archives, and fully processed history + leave no evidence deadline or recurring filesystem/SQL calls. + +The existing retention owner exposes `nextEvidenceAt()` and a per-instance duty +selection. Legacy callers that omit the selection keep combined behavior. Startup +selects `backups` for the existing backup timer and registers an `evidence` instance +with the delivery coordinator. The shared scheduler/coordinator owns overlap, +admission, drain, retries, deadlines, and shutdown. No feature logic moves into +`index.ts`. The worker's first promise stays tracked without delaying readiness; +shutdown unsubscribes the local signal before stopping/joining delivery work. + +The backup interval remains deliberately visible for the next independent slice. +This PR removes evidence from that interval; it does not claim that the instance +as a whole is idle while backup and other recurring duties remain. + +Supported publication is the existing server-owned session executor in one +process. Unrelated filesystem writers need an explicit signal/ingress contract. +Startup reconstructs restart state; it is not cross-process live notification. +Cloud sleep gates and external alarm coverage are unchanged. Expiration can wait +for a supported host wake when compute is stopped. + +## Failure record and local evidence + +Two regressions failed before repair: the saved deadline was one millisecond +before the first removable instant, and an EACCES reading an archive directory +was treated as an empty directory and disarmed retention. Missing paths remain +quiet; other inspection failures now retain recovery. + +A further controlled EACCES at the session root reproduced a lost cursor: retry +continued after the failed scope and left its evidence behind. The owner now +records that concrete failed inspection for a new full evidence pass. Healthy +scopes continue, and the failed scope is revisited after the bounded retry. + +An intermediate full run exposed a compatibility regression in the existing +manual dry-run notification case. The publication timestamp's original due +semantics were restored; the inclusive boundary adjustment applies to the +observed recent item's next inspection. The old test and deletion protections +were preserved rather than weakened. + +Final local acceptance: 158 cases pass across native-harness/evidence retention, +retention configuration, and startup. The retention suite uses a guarded isolated +PostgreSQL database; all selected cases execute. Tests measure quiet filesystem +and SQL behavior, saved deadlines, local publication, restart, busy/failure retry, +first work during drain, one-entry continuation, absence of backup SQL in the +evidence worker, disabled registration, and tracked startup after HTTP readiness. +Existing backup, manifest, lease, issue, dry-run, symlink, and cleanup-archive +cases still pass. Server TypeScript checking and the full server build also pass. +Remote review/CI and whole-stack acceptance remain pending. + +## Ponytail repair: quiet handled retries, summaries, and publication boundary + +Controlled tests reproduced a busy archive causing whole-tree inspection every +thirty seconds, that overdue retry masking another archive's ten-second saved +expiry, missing deletion/byte summaries, and a publication deadline one millisecond +before the first removable instant. All four fail before this repair. + +The evidence-only owner now records an hourly retry for handled busy/inspection/ +removal failures. Genuine earlier saved expiries remain the minimum deadline. +One-second truncated continuations remain independent. Unexpected state-base +inspection failure is also retained as an hourly worker retry; a new publication +can reactivate it immediately. Admission/drain reconciliation still uses the +coordinator's existing short memory-only retry. Legacy combined/manual callers +retain their original explicit-tick behavior. + +The worker again logs candidate, removal, reclaimed-byte, failure, and dry-run +summary fields when it processes candidates or failures. Publication now schedules +the first removable instant in evidence-only mode; the old manual notification +case is unchanged. An added base-directory fault test proves five minutes of quiet +before a publication resumes processing. Final owning acceptance passes 162 cases +across retention, configuration, and startup, plus server typechecking. Earlier +receipts remain tied to their recorded heads; integrated verification follows the +parent skill-retention repair. + +## October 29 master reconciliation and current acceptance + +This existing archived-evidence slice was reconciled with the qualified retention +parent through normal merges. The expiration service now exposes both the new +saved evidence deadline and the parent's separate startup checkpoint methods. +The finite checkpoint failure budget remains intact. The native retention fixture +keeps its explicit external-database override and migrations while using the +parent's fresh recovery-database helper by default. Identical logger/coordinator +imports introduced by automatic composition were deduplicated. Every other path +matches the automatic merge, and packages remain byte-identical to the parent. + +Current local acceptance is **285 passing unique cases across 11 complete files**, +without skips. The original combined run passed 283 cases but failed two new +startup fixtures because their mocked service omitted the preserved +`registerCheckpointExpiration` method. Production startup correctly called that +method. The fixture repair supplies the actual API and adds explicit assertions +that enabled and disabled evidence configurations retain checkpoint registration, +including while evidence reconstruction is held after HTTP readiness. The full +100-case startup file then passed in 7.55 seconds, without weakening any prior +assertion or timeout. The other ten complete files passed 185 cases and their +source remained unchanged. Accepted source after that fixture-only repair is +`da44dea2f3cd86cc72a25f2a419c63acafda1f7a`. + +Coverage includes real archived-evidence expiry, filesystem and path safety, +inclusive retention boundaries, retry and paged continuation, preserved finite +checkpoint startup work, backup cleanup/configuration, server wiring, shared +deadline/coordinator and idle-work behavior, plus retained skill-cache behavior +and its own-company expiry regression. Database build, production server +typechecking/build, module boundaries, target migration order and Runner profile +identity passed before the fixture-only correction; runtime source did not change +afterward. Two fresh PostgreSQL 17 schemas plus per-file fixtures applied all 412 +journal entries through index 413. All temporary local fixture imports were +restored exactly, and every selected test hash is bound in the acceptance record. + +The archive worker uses the evidence duty and performs no backup SQL. The backup +duty and startup checkpoint owner remain distinct. Saved publication/retention +boundaries and one-hour busy/failure retries retain nearer expiries and bounded +one-second continuation pages. Unsubscription still precedes coordinator stop. +Publication must adopt the actual merged skill-retention prerequisite and obtain +fresh exact-head CI/security and reviews. No full local repository-suite, live +provider, external wake, measured instance/database suspension or fleet-cost claim +is made. Historical results and limitations above remain tied to their revisions. + +## Requalification with later upstream changes + +Normal reconciliation preserved the final skill-retention repair and all 95 +unrelated master paths from the actual timer squash, task activity index and task +read changes. On source `2fa458833fa43b47eddc728a5e57f5ffcf5fa3e4`, all **285 +cases across 11 complete files passed together**, with no skips, in 47.26 seconds. +This fresh full selection includes the preserved checkpoint APIs and their +repaired startup fixtures. Database build, production server typechecking/build, +module boundaries, target migration order and generated TypeScript/Rust release +profile consistency passed. Fresh PostgreSQL 17 schemas applied all 413 journal +entries through index 414. Every selected source hash and temporary fixture import +restored exactly. The final parent audit was then adopted through a doc-only +normal merge without changing any accepted runtime/test/package tree. + +The inherited upstream Copilot source-identity failure is recorded in the +[parent retention audit](2026-10-27-skill-version-retention-sleep-audit.md#final-current-master-qualification-before-publication). +This slice leaves its release attestation and all identity inputs unchanged. +Generated admission still matches the active release manifest; the separate +source-identity computation is an explicit inherited exception, not a pass or a +claim of new live-provider qualification. Required fresh exact-head CI/security +and reviews remain merge gates. Actual parent squash adoption is still required +before publication; any new upstream behavior must remain preserved and receive +appropriate combined-context checks. + +## Workspace-seed upstream composition + +The next upstream merge (`27fbdb2a74e13141065555b6a2fa5acdd8da7964`) +optimizes verified workspace seed reuse. It is preserved by a normal merge, +including its adapter implementation, tests and native workspace history. On +combined source `a288a58287047193774eda54601ac9eda732d2e8`, **321 cases +across 14 complete files passed together**, without skips, in 95.73 seconds. +This includes the previous complete 11-file retention selection and the full +workspace-seed reuse, GitHub launcher and native workspace-history files. These +results replace, rather than add to, the earlier 285-case acceptance total for +this combined source. Adapter/database builds, server typechecking/build, module +boundaries, target migration order and generated release profiles pass. Every +selected source hash and temporary fixture import restores exactly. Provider +identity inputs remain unchanged; the inherited exception above still applies. + +Actual skill-retention squash adoption remains a publication prerequisite. If +its tree carries only these already accepted upstream changes, tree identity +will prove that adoption changes no accepted runtime. Any additional behavior +will receive scoped combined-context validation before publication. + +## Runner-authority upstream qualification + +Master `27be8590442fc02d29a6c257a4e9dbe5bbbd66d2` then introduced Runner +authority for runtime activity and updated provider manifests. Normal composition +through `b80eee32f344ae0218d8b791c14f2ea174da09cc` preserves its complete +package tree byte for byte, the accepted workspace changes and the parent browser +readiness repair. The server retention implementation is unchanged. + +On that combined source, **652 TypeScript cases across 22 complete files passed**. +These include the complete 14-file evidence-retention/workspace selection (321 +cases) and eight affected Runner files (331 cases). One actual closed Pi snapshot +launch is intentionally skipped because its external distribution root is not +provided; this is not a live-distribution qualification. **29 Rust cases passed** +in the full native-backend target (26), full runtime-notification target (one), +and both configured-environment unit cases. Adapter/Runner/database builds, Runner +and server typechecking, Rust formatting/workspace checking, server build, module +boundaries, migration order and generated TypeScript/Rust profiles all passed. +Fresh PostgreSQL 17 schemas apply all 413 entries through index 414. All selected +source hashes and temporary imports restore exactly. These fresh counts replace +rather than add to the prior selections. + +The new upstream manifest uses Copilot profile 81, but its separate source-identity +check still fails: the active fixture's `configuredEnvironmentRustSourceSha256` +differs from the actual merged Rust file. Every bound source and the entire +package tree equal master. This preserves the earlier inherited OpenCode-key +identity drift through the upstream profile update; no retention edit causes it. +Grok's separate identity check passes. The Copilot exception remains explicit, +and no release attestation is regenerated or new live-provider qualification +claimed. Required fresh exact-head CI/security and reviews remain merge gates. + +Actual parent squash adoption remains necessary before final publication. The +new parent browser-repair commit and its original assertions are carried exactly; +the browser gate itself must still be green on the final published head. + +## Actual prerequisite adoption for final publication + +The skill-retention prerequisite [#17923](https://github.com/paperclipai/paperclip/pull/17923) +merged at `b06ea111a254c62e0a55199b010d82e0adb33bf0`, after all 58 current +checks were clean and the exact reviewed head `269935f6ccd7685866fd94bea66f7f803c48058d` +had Greptile and Ponytail 5/5. Its separate, evidence-supported failed-job retry +passed; no failing gate or required review finding was waived. + +This last slice adopts that actual squash through normal merge +`3c40eaad954f7705144e6f378d93be2adff0c54a`. Duplicate ancestry conflicts were +resolved to the proven composed tree. The complete pre-adoption held tree is +unchanged, including the 652-case TypeScript and 29-case Rust accepted sources, +all selected tests and the parent's browser-readiness repair. Package, UI, E2E +and map trees equal the actual prerequisite. The final diff has eight owned +paths. This adoption changes ancestry rather than runtime behavior, so accepted +checks are not repeated; this paragraph and the PR body only record provenance. +The original remote head remains an ancestor, and no history is force-pushed. + +Fresh required CI/security, exact-head Greptile 5/5 and any actionable Ponytail +findings are still gates on this slice's final publication. The inherited source +identity exception and all bounded-acceptance limitations above remain explicit. + +## Final outage-journey readiness repair + +The first final publication, `762d48605f93351e47f9fd1249d9293625834242`, +received exact-head Greptile 5/5 and Ponytail 5/5 with no actionable findings. +Its required checks completed cleanly except the second browser shard and its +aggregate. The title-outage journey reached the 60-second global test timeout; +its failure artifact still shows the original title, and no issue write was +observed. The same file then passed that case locally but reproduced a timeout +in the failed-session startup case (four passing cases and one failure). These +failures and the original assertions remain recorded; no failing gate is waived. + +A visible initial task heading alone does not establish that the conversation +layout has settled after the upstream loading changes. The test now requires +visible conversation history before the ordinary title-edit flows, including +the cached offline task switch. Four additive readiness assertions are the only +spec changes. Removing those four lines reproduces the exact prior file bytes: +all original recovery assertions, outage controls, and timeout budgets remain. +The deliberately held first-session-lookup case retains its original pre-auth +readiness, so it still queues a write before the owner is known. An initial +broader readiness attempt blocked that intentional startup state; that extra +condition was removed rather than changing the scenario. + +All **five complete outage-recovery browser journeys pass** afterward on a new +isolated PostgreSQL 17 database. They still verify failed-save retention, actual +retry after recovery, separation of two offline drafts, delayed owner binding, +comment replay idempotency and failed-session retry. The accepted 652-case +TypeScript and 29-case Rust runtime sources, all prior sleep regression files, +packages and UI are unchanged. The final nine-path diff adds this spec-readiness +repair to the eight-path feature. Fresh exact-head CI/security and reviews are +required after publication; older-head review passes are not reused. diff --git a/server/src/__tests__/native-harness-backup-expiration.test.ts b/server/src/__tests__/native-harness-backup-expiration.test.ts index 764539c2454..b24c12a80c9 100644 --- a/server/src/__tests__/native-harness-backup-expiration.test.ts +++ b/server/src/__tests__/native-harness-backup-expiration.test.ts @@ -1,3 +1,8 @@ +import { applyPendingMigrations } from "@paperclipai/db"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { registerNativeRunnerEvidenceRetentionWork } from "../services/native-runtime/native-runner-evidence-retention-work.js"; +import { noteNativeRunnerEvidence } from "../services/native-runtime/native-runner-evidence-signal.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; import { createHash, randomUUID } from "node:crypto"; import os from "node:os"; import path from "node:path"; @@ -20,9 +25,10 @@ import { nativeHarnessBackupExpirationService } from "../services/native-runtime import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; import { logger } from "../middleware/logger.js"; -const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_RETENTION_TEST_DATABASE_URL"); +const describeEmbeddedPostgres = external || embeddedPostgresSupport.supported ? describe : describe.skip; -if (!embeddedPostgresSupport.supported) { +if (!external && !embeddedPostgresSupport.supported) { console.warn( `Skipping embedded Postgres native harness backup expiration tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`, ); @@ -41,11 +47,17 @@ describeEmbeddedPostgres("native harness backup expiration", () => { const now = new Date(); beforeAll(async () => { - tempDb = await startRecoveryPostgres("paperclip-native-backup-expiration-"); - db = createDb(tempDb.connectionString); + if (!external) tempDb = await startRecoveryPostgres("paperclip-native-backup-expiration-"); + if (external) await applyPendingMigrations(external); + db = createDb(external ?? tempDb!.connectionString); }, 20_000); + let coordinator: ReturnType | undefined; + let worker: ReturnType | undefined; + let admitted = true; afterEach(async () => { + worker?.unsubscribe(); await coordinator?.stop(); coordinator = undefined; worker = undefined; admitted = true; + vi.useRealTimers(); vi.restoreAllMocks(); await db.delete(agentTaskSessions); await db.delete(nativeRunFinalizations); await db.delete(environmentLeases); @@ -58,6 +70,7 @@ describeEmbeddedPostgres("native harness backup expiration", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); @@ -522,6 +535,146 @@ describeEmbeddedPostgres("native harness backup expiration", () => { return createHash("sha256").update(randomUUID()).digest("hex"); } + async function startEvidence(options: Parameters[1] = {}) { + if (!vi.isFakeTimers()) { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + vi.setSystemTime(now); + } + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => true, + onError: error => { throw error; } }); + worker = registerNativeRunnerEvidenceRetentionWork(coordinator, db, { stateBase, isSessionBusy: () => false, ...options }, () => admitted); + await worker.ready; + } + async function settled(deadline: number | null = null) { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBe(deadline); + }, { timeout: 5000 }); + } + async function quiet(ms = 3 * 60 * 60_000) { + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"), directories = vi.spyOn(fs, "readdir"); + await vi.advanceTimersByTimeAsync(ms); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); expect(directories).not.toHaveBeenCalled(); + reads.mockRestore(); writes.mockRestore(); directories.mockRestore(); + } + + it("leaves empty evidence and retained backups quiet without invoking backup SQL", async () => { + const session = await seedSession(); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"); + await startEvidence(); await settled(); expect(await exists(session.current)).toBe(true); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + reads.mockRestore(); writes.mockRestore(); await quiet(); + }); + + it("reconstructs and replaces the saved evidence deadline after a publication signal", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const first = await seedArchive(session.root, { stampedDaysAgo: 2 }); + const firstDeadline = now.getTime() + 5 * DAY_MS + 1; + await startEvidence(); await settled(firstDeadline); await quiet(); + const writtenAt = Date.now(); + const next = await seedArchive(session.root, { name: `${writtenAt}-${randomUUID()}` }); + noteNativeRunnerEvidence(writtenAt); + await settled(firstDeadline); + await vi.advanceTimersByTimeAsync(firstDeadline - Date.now()); + await settled(writtenAt + 7 * DAY_MS + 1); expect(await exists(first)).toBe(false); + await vi.advanceTimersByTimeAsync(writtenAt + 7 * DAY_MS + 1 - Date.now()); await settled(); + expect(await exists(next)).toBe(false); expect(await exists(session.current)).toBe(true); await quiet(); + }); + + it.each(["session root", "archive directory"])("preserves an evidence inspection failure at %s until its concrete retry succeeds", async location => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root); + const original = fs.lstat; + const denied = vi.spyOn(fs, "lstat").mockImplementation(async (...args) => { + if (String(args[0]) === (location === "session root" ? session.root : path.join(session.root, "continuity-breaks"))) throw Object.assign(new Error("unavailable"), { code: "EACCES" }); + return original(...args); + }); + await startEvidence(); expect(coordinator!.nextWakeAt()).not.toBeNull(); + const retryAt = coordinator!.nextWakeAt()!; expect(await exists(archive)).toBe(true); + denied.mockRestore(); await quiet(5000); + await vi.advanceTimersByTimeAsync(retryAt - Date.now()); await settled(); + expect(await exists(archive)).toBe(false); await quiet(); + }); + + it("retries a busy evidence obligation and disarms after the session releases it", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root); let busy = true; + await startEvidence({ isSessionBusy: () => busy }); + const retryAt = coordinator!.nextWakeAt()!; expect(retryAt - Date.now()).toBeGreaterThanOrEqual(60 * 60_000); + await quiet(5000); busy = false; + await vi.advanceTimersByTimeAsync(retryAt - Date.now()); await settled(); + expect(await exists(archive)).toBe(false); await quiet(); + }); + + it("retains the first publication during drain and cleans up on resume", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + await startEvidence(); await settled(); admitted = false; + const archive = await seedArchive(session.root); + noteNativeRunnerEvidence(now.getTime() - 10 * DAY_MS); + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(1000); expect(idleWorkSnapshot().active).toBe(0); }); + await quiet(60_000); expect(await exists(archive)).toBe(true); + admitted = true; await settled(); expect(await exists(archive)).toBe(false); await quiet(); + }); + + it("reconstructs the same retained deadline after restart", async () => { + const scope = await emptyStateBase(); await seedQuarantine(scope, { stampedDaysAgo: 6 }); + const deadline = now.getTime() + DAY_MS + 1; + await startEvidence(); await settled(deadline); + worker!.unsubscribe(); await coordinator!.stop(); + await startEvidence(); await settled(deadline); await quiet(60_000); + expect(coordinator!.nextWakeAt()).toBe(deadline); + }); + + it("backs off a failed state-base inspection and lets publication reactivate it", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root), read = fs.readdir; + const denied = vi.spyOn(fs, "readdir").mockImplementation(async (...args) => { + if (args[0] === stateBase) throw Object.assign(new Error("state base unavailable"), { code: "EACCES" }); + return read(...args); + }); + await startEvidence(); expect(coordinator!.nextWakeAt()! - Date.now()).toBeGreaterThanOrEqual(60 * 60_000); + denied.mockRestore(); await quiet(5 * 60_000); expect(await exists(archive)).toBe(true); + noteNativeRunnerEvidence(now.getTime() - 10 * DAY_MS); + await settled(); expect(await exists(archive)).toBe(false); await quiet(); + }); + + it("lets healthy saved expiry preempt a long-busy evidence retry", async () => { + const busy = await seedSession({ issueStatus: "in_progress" }), healthy = await seedSession({ issueStatus: "in_progress" }); + const held = await seedArchive(busy.root); + const dueAt = now.getTime() + 10_001; + const available = await seedArchive(healthy.root, { name: `${dueAt - 7 * DAY_MS - 1}-${randomUUID()}` }); + await startEvidence({ isSessionBusy: scope => scope === busy.scope }); await settled(dueAt); + await vi.advanceTimersByTimeAsync(dueAt - Date.now()); + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(100); expect(await exists(available)).toBe(false); }); + expect(await exists(held)).toBe(true); expect(coordinator!.nextWakeAt()! - Date.now()).toBeGreaterThan(59 * 60_000); + await quiet(60_000); + }); + + it("reports reclaimed evidence bytes and dry-run mode", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root); const reported = vi.spyOn(logger, "info"); + await startEvidence(); await settled(); + expect(reported.mock.calls).toEqual(expect.arrayContaining([[expect.objectContaining({ evidenceCandidates: 1, + evidenceRemoved: 1, evidenceReclaimedBytes: expect.any(Number), dryRun: false }), "native runner evidence retention"]])); + expect(await exists(archive)).toBe(false); await quiet(); + }); + + it("schedules publication at the first removable instant after its retention boundary", async () => { + const scope = await emptyStateBase(); await startEvidence(); await settled(); await quiet(); + const writtenAt = Date.now(); + await seedQuarantine(scope, { name: `${scope}.identity_indeterminate.${writtenAt}.4242.${randomUUID()}` }); + noteNativeRunnerEvidence(writtenAt); await settled(writtenAt + 7 * DAY_MS + 1); + }); + + it("uses bounded continuations without recounting an evidence session forever", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archives = [await seedArchive(session.root), await seedArchive(session.root)]; + const quarantine = await seedQuarantine(session.scope); + await startEvidence({ maxEntries: 1 }); await settled(); + for (const archive of [...archives, quarantine]) expect(await exists(archive)).toBe(false); + await quiet(); + }); + it("removes continuity archives older than the retention window and keeps recent ones", async () => { const session = await seedSession({ issueStatus: "in_progress" }); const old = await seedArchive(session.root, { stampedDaysAgo: 9 }); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index dc0e05e249c..420a18bbc86 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -1,3 +1,4 @@ +import { nativeHarnessBackupExpirationService } from "../services/native-runtime/native-harness-backup-expiration.js"; import { skillVersionRetentionService } from "../services/skill-version-retention.js"; import { approvalQueueSeedingService } from "../services/approval-queue-seeding-work.js"; import { joinRequestRetentionService } from "../services/join-request-retention.js"; @@ -539,6 +540,18 @@ vi.mock("../board-claim.js", () => ({ initializeBoardClaimChallenge: vi.fn(async () => undefined), })); +vi.mock("../services/native-runtime/native-harness-backup-expiration.js", () => ({ + NATIVE_HARNESS_BACKUP_EXPIRATION_INTERVAL_MS: 60 * 60_000, + NATIVE_HARNESS_BACKUP_EXPIRATION_STARTUP_DELAY_MS: 2 * 60_000, + NATIVE_HARNESS_BACKUP_EXPIRATION_DEFAULT_DAYS: 7, + NATIVE_RUNNER_EVIDENCE_RETENTION_DEFAULT_DAYS: 7, + nativeHarnessBackupExpirationService: vi.fn(() => ({ + tick: vi.fn(async () => ({ checked: 0, candidates: 0, evidenceCandidates: 0, failed: 0, truncated: false })), + noteEvidence: vi.fn(), nextEvidenceAt: vi.fn(() => null), + tickCheckpoints: vi.fn(), registerCheckpointExpiration: vi.fn(), + })), +})); + vi.mock("../services/skill-version-retention.js", () => ({ skillVersionRetentionService: vi.fn(() => ({ listCompanies: vi.fn(async () => []), pruneCompany: vi.fn(async () => ({ checked: 0, truncated: false, nextRunAt: null })), @@ -582,6 +595,31 @@ describe("startServer feedback export wiring", () => { process.env.BETTER_AUTH_SECRET = "test-secret"; }); + it.each([false, true])("registers runner evidence independently from the backup scan (%s)", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ nativeHarnessBackupExpirationEnabled: enabled })); + await startServer(); await settleStartupHousekeeping(); + expect(deliveryWorkRegisterMock.mock.calls.filter(([queue]) => queue === DELIVERY_QUEUES.nativeRunnerEvidenceRetention)).toHaveLength(enabled ? 1 : 0); + expect(vi.mocked(nativeHarnessBackupExpirationService).mock.calls.map(([, options]) => options?.duty)).toEqual(enabled ? ["backups", "evidence"] : ["backups"]); + const backups = vi.mocked(nativeHarnessBackupExpirationService).mock.results[0]!.value; + expect(backups.registerCheckpointExpiration).toHaveBeenCalledTimes(enabled ? 1 : 0); + }); + + it("tracks held evidence reconstruction after HTTP readiness", async () => { + loadConfigMock.mockReturnValue(buildTestConfig({ nativeHarnessBackupExpirationEnabled: true })); + const pending = Promise.withResolvers<{ truncated: boolean }>(); + vi.mocked(nativeHarnessBackupExpirationService).mockReturnValueOnce({ tick: vi.fn(), noteEvidence: vi.fn(), nextEvidenceAt: () => null, + tickCheckpoints: vi.fn(), registerCheckpointExpiration: vi.fn() } as never); + vi.mocked(nativeHarnessBackupExpirationService).mockReturnValueOnce({ tick: vi.fn(() => pending.promise), noteEvidence: vi.fn(), nextEvidenceAt: () => null, + tickCheckpoints: vi.fn(), registerCheckpointExpiration: vi.fn() } as never); + try { + await startServer(); await settleStartupHousekeeping(); + expect(getStartupRecoveryState().phase).toBe("ready"); expect(idleWorkSnapshot().active).toBe(1); + expect(vi.mocked(nativeHarnessBackupExpirationService).mock.results[0]!.value.registerCheckpointExpiration).toHaveBeenCalledOnce(); + } finally { pending.resolve({ truncated: false }); await settleStartupHousekeeping(); } + expect(idleWorkSnapshot().active).toBe(0); + expect(deliveryWorkErrors.filter(error => error.queue === DELIVERY_QUEUES.nativeRunnerEvidenceRetention)).toEqual([]); + }); + it.each([false, true])("registers skill cache retention only when enabled (%s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ skillVersionRetentionEnabled: enabled })); await startServer(); await settleStartupHousekeeping(); diff --git a/server/src/index.ts b/server/src/index.ts index 76ec3cf072b..1c51bfb6a8b 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -32,7 +32,7 @@ import { NATIVE_HARNESS_BACKUP_EXPIRATION_STARTUP_DELAY_MS, nativeHarnessBackupExpirationService, } from "./services/native-runtime/native-harness-backup-expiration.js"; -import { onNativeRunnerEvidence } from "./services/native-runtime/native-runner-evidence-signal.js"; +import { registerNativeRunnerEvidenceRetentionWork } from "./services/native-runtime/native-runner-evidence-retention-work.js"; import { gitRestorePointRetentionService, registerGitRestorePointWork } from "./services/git-restore-point-ledger.js"; import { nativeWorkspaceSyncRetentionService, @@ -1259,12 +1259,12 @@ async function startServerWithDatabaseTeardown( }; const skillSourceSync = skillSourceSyncService(db as any); const nativeHarnessBackupExpiration = nativeHarnessBackupExpirationService(db as any, { + duty: "backups", dryRun: config.nativeHarnessBackupExpirationDryRun, closedForDays: config.nativeHarnessBackupExpirationDays, evidenceRetentionDays: config.nativeRunnerEvidenceRetentionDays, shouldContinue: () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby(), }); - const unsubscribeNativeRunnerEvidence = onNativeRunnerEvidence(nativeHarnessBackupExpiration.noteEvidence); let nativeHarnessBackupExpirationInFlight = false; let nativeHarnessBackupExpirationContinuation: ReturnType | undefined; const sweepNativeHarnessBackups = () => { @@ -1344,6 +1344,13 @@ async function startServerWithDatabaseTeardown( server.on("close", () => { void taskDrainExpiryWork.stop(); }); const executionSchedule = executionControlSchedule(db); const workspaceRecoveryRuntime = environmentRuntimeService(db, { pluginWorkerManager }); + const nativeRunnerEvidenceRetentionWorker = config.nativeHarnessBackupExpirationEnabled + ? registerNativeRunnerEvidenceRetentionWork(deliveryWork, db as any, { + dryRun: config.nativeHarnessBackupExpirationDryRun, + evidenceRetentionDays: config.nativeRunnerEvidenceRetentionDays, + }, () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby()) + : null; + if (nativeRunnerEvidenceRetentionWorker) trackHeartbeatSchedulerWork(nativeRunnerEvidenceRetentionWorker.ready); const skillVersionRetentionWorker = config.skillVersionRetentionEnabled ? registerSkillVersionRetentionWork(deliveryWork, skillVersionRetentionService(db as any, { dryRun: config.skillVersionRetentionDryRun, @@ -2191,7 +2198,7 @@ async function startServerWithDatabaseTeardown( clearInterval(nativeHarnessBackupExpirationInterval); clearTimeout(nativeHarnessBackupExpirationStartup); clearTimeout(nativeHarnessBackupExpirationContinuation); - unsubscribeNativeRunnerEvidence(); + nativeRunnerEvidenceRetentionWorker?.unsubscribe(); await skillSourceSync.stop(); unsubscribeChatCompletions(); unsubscribeNativeResponsePresentation?.(); diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index ef031e443ef..0e8748ad226 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; /** Topics for existing durable queues; register intent before their writes. */ export const DELIVERY_QUEUES = { + nativeRunnerEvidenceRetention: "native-runner-evidence-retention", skillVersionRetention: "skill-version-retention", databaseBackups: "database-backups", importSpoolCleanup: "import-spool-cleanup", diff --git a/server/src/services/native-runtime/native-harness-backup-expiration.ts b/server/src/services/native-runtime/native-harness-backup-expiration.ts index 750085d1e09..8ea9a242629 100644 --- a/server/src/services/native-runtime/native-harness-backup-expiration.ts +++ b/server/src/services/native-runtime/native-harness-backup-expiration.ts @@ -40,6 +40,7 @@ const MAX_CONTROL_PLANE_STATE_BYTES = 4 * 1024 * 1024; const CHECKPOINT_EXPIRATION_MAX_FAILED_PASSES = 3; export type NativeHarnessBackupExpirationOptions = { + duty?: "backups" | "evidence"; closedForDays?: number; evidenceRetentionDays?: number; dryRun?: boolean; @@ -77,6 +78,7 @@ type EvidenceContext = { retainedBefore: number; canInspect: () => boolean; dueAt: (at: number) => void; + retry: () => void; }; type SessionIdentity = { @@ -105,8 +107,13 @@ function emptyResult(): NativeHarnessBackupExpirationResult { }; } +function missingPath(error: NodeJS.ErrnoException) { + if (error.code === "ENOENT") return null; + throw error; +} + async function isRealDirectory(target: string): Promise { - const stat = await fs.lstat(target).catch(() => null); + const stat = await fs.lstat(target).catch(missingPath); return Boolean(stat?.isDirectory() && !stat.isSymbolicLink()); } @@ -262,6 +269,8 @@ export function nativeHarnessBackupExpirationService( db: Db, defaults: NativeHarnessBackupExpirationOptions = {}, ) { + const includeBackups = defaults.duty !== "evidence"; + const includeEvidence = defaults.duty !== "backups"; let cursor: string | null = null; let resume: { scope: string; archive: string } | null = null; let quarantineCursor: string | null = null; @@ -421,7 +430,10 @@ export function nativeHarnessBackupExpirationService( } async function removeExpiredLeftovers(parent: string, context: EvidenceContext, result: NativeHarnessBackupExpirationResult) { - for (const name of await fs.readdir(parent).catch(() => [] as string[])) { + for (const name of await fs.readdir(parent).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return [] as string[]; + throw error; + })) { if (!name.startsWith(EXPIRED_PREFIX) || !UUID_PATTERN.test(name.slice(EXPIRED_PREFIX.length))) continue; if (!context.canInspect()) { context.dueAt(context.now); @@ -444,17 +456,17 @@ export function nativeHarnessBackupExpirationService( result.skipped[reason] = (result.skipped[reason] ?? 0) + 1; }; const target = path.join(parent, name); - const stat = await fs.lstat(target).catch(() => null); + const stat = await fs.lstat(target).catch(missingPath); if (path.dirname(target) !== parent || !stat?.isDirectory()) { return skip("unsafe_path"); } const createdAt = stampedAt > 0 && stampedAt <= context.now ? stampedAt : stat.mtimeMs; if (createdAt >= context.retainedBefore) { - context.dueAt(createdAt + context.retentionMs); + context.dueAt(createdAt + context.retentionMs + 1); return skip("recent_evidence"); } if (context.isSessionBusy(scopeDigest)) { - context.dueAt(context.now); + context.retry(); return skip("session_active"); } const bytes = await allocatedBytes(target); @@ -462,7 +474,7 @@ export function nativeHarnessBackupExpirationService( result.evidenceCandidateBytes += bytes; if (context.dryRun) return; if (context.isSessionBusy(scopeDigest)) { - context.dueAt(context.now); + context.retry(); return skip("session_active"); } const trash = path.join(parent, `${EXPIRED_PREFIX}${randomUUID()}`); @@ -480,7 +492,7 @@ export function nativeHarnessBackupExpirationService( result: NativeHarnessBackupExpirationResult, ): Promise { const parent = path.join(root, "continuity-breaks"); - const stat = await fs.lstat(parent).catch(() => null); + const stat = await fs.lstat(parent).catch(missingPath); if (!stat) return null; if (!stat.isDirectory()) { result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; @@ -499,7 +511,7 @@ export function nativeHarnessBackupExpirationService( await retireEvidence(parent, name, scopeDigest, Number(match[1]), context, result); } catch (err) { result.failed++; - context.dueAt(context.now); + context.retry(); logger.warn({ err, scope: scopeDigest }, "native runner evidence retention skipped a continuity archive"); } } @@ -513,7 +525,7 @@ export function nativeHarnessBackupExpirationService( result: NativeHarnessBackupExpirationResult, ): Promise { const parent = path.join(base, "quarantine"); - const stat = await fs.lstat(parent).catch(() => null); + const stat = await fs.lstat(parent).catch(missingPath); if (!stat) return true; if (!stat.isDirectory()) { result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; @@ -537,7 +549,7 @@ export function nativeHarnessBackupExpirationService( await retireEvidence(parent, name, match[1]!, Number(match[2]), context, result); } catch (err) { result.failed++; - context.dueAt(context.now); + context.retry(); logger.warn({ err, scope: match[1] }, "native runner evidence retention skipped a quarantine entry"); } } @@ -546,7 +558,7 @@ export function nativeHarnessBackupExpirationService( } function noteEvidence(writtenAt: number) { - evidenceDueAt = Math.min(evidenceDueAt, writtenAt + evidenceRetentionMs()); + evidenceDueAt = Math.min(evidenceDueAt, writtenAt + evidenceRetentionMs() + (defaults.duty === "evidence" ? 1 : 0)); } function evidenceRetentionMs(options: NativeHarnessBackupExpirationOptions = {}) { @@ -572,9 +584,10 @@ export function nativeHarnessBackupExpirationService( const closedBefore = new Date(now - closedForDays * DAY_MS); const result = emptyResult(); const hasBudget = () => shouldContinue() && result.candidates + result.evidenceCandidates < maxRemovals; + if (!passActive && !includeBackups && evidenceDueAt > now) return result; if (!passActive) { passActive = true; - evidencePass = evidenceDueAt <= now; + evidencePass = includeEvidence && evidenceDueAt <= now; if (evidencePass) { evidenceDueAt = Number.POSITIVE_INFINITY; passDueAt = Number.POSITIVE_INFINITY; @@ -590,10 +603,16 @@ export function nativeHarnessBackupExpirationService( dueAt: (at) => { passDueAt = Math.min(passDueAt, at); }, + retry: () => { + passDueAt = Math.min(passDueAt, now + (defaults.duty === "evidence" ? 60 * 60_000 : 0)); + }, }; if (phase === "sessions") { - const names = (await fs.readdir(base).catch(() => [] as string[])) + const names = (await fs.readdir(base).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return [] as string[]; + throw error; + })) .filter((name) => SCOPE_PATTERN.test(name)) .sort(); const pending = cursor ? names.filter((name) => name > cursor!) : names; @@ -608,11 +627,18 @@ export function nativeHarnessBackupExpirationService( const previous = cursor; cursor = name; const root = path.join(base, name); - if (path.dirname(root) !== base || !(await isRealDirectory(root))) { - result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; + try { + if (path.dirname(root) !== base || !(await isRealDirectory(root))) { + result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; + continue; + } + } catch (err) { + result.failed++; + context.retry(); + logger.warn({ err, scope: name }, "native runner retention could not inspect a session root"); continue; } - if (resumeAfter === null) { + if (resumeAfter === null && includeBackups) { try { await expireSession(root, name, { dryRun, isSessionBusy, closedBefore }, result); } catch (err) { @@ -626,7 +652,7 @@ export function nativeHarnessBackupExpirationService( stoppedAt = await retireContinuityBreaks(root, name, resumeAfter, context, result); } catch (err) { result.failed++; - context.dueAt(now); + context.retry(); logger.warn({ err, scope: name }, "native runner evidence retention skipped a session"); } if (stoppedAt !== null) { @@ -651,7 +677,7 @@ export function nativeHarnessBackupExpirationService( } } catch (err) { result.failed++; - context.dueAt(now); + context.retry(); quarantineCursor = null; logger.warn({ err }, "native runner evidence retention skipped quarantine"); } @@ -662,6 +688,11 @@ export function nativeHarnessBackupExpirationService( return result; } + function nextEvidenceAt() { + if (passActive) return Date.now(); + return Number.isFinite(evidenceDueAt) ? evidenceDueAt : null; + } + async function tickCheckpoints(options: NativeHarnessBackupExpirationOptions = {}) { const closedForDays = Math.max(1, options.closedForDays ?? defaults.closedForDays ?? NATIVE_HARNESS_BACKUP_EXPIRATION_DEFAULT_DAYS); const result = emptyResult(); @@ -714,5 +745,5 @@ export function nativeHarnessBackupExpirationService( }); } - return { tick, noteEvidence, tickCheckpoints, registerCheckpointExpiration }; + return { tick, noteEvidence, nextEvidenceAt, tickCheckpoints, registerCheckpointExpiration }; } diff --git a/server/src/services/native-runtime/native-runner-evidence-retention-work.ts b/server/src/services/native-runtime/native-runner-evidence-retention-work.ts new file mode 100644 index 00000000000..b43dfaa60f7 --- /dev/null +++ b/server/src/services/native-runtime/native-runner-evidence-retention-work.ts @@ -0,0 +1,34 @@ +import { logger } from "../../middleware/logger.js"; +import { registerDeadlineWork } from "../deadline-work.js"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; +import { nativeHarnessBackupExpirationService, type NativeHarnessBackupExpirationOptions } from "./native-harness-backup-expiration.js"; +import { onNativeRunnerEvidence } from "./native-runner-evidence-signal.js"; + +export function registerNativeRunnerEvidenceRetentionWork( + coordinator: ReturnType, + db: Parameters[0], + options: NativeHarnessBackupExpirationOptions, + canRun: () => boolean, +) { + const service = nativeHarnessBackupExpirationService(db, { ...options, duty: "evidence" }); + let continuation: number | null = null; + const worker = registerDeadlineWork(coordinator, DELIVERY_QUEUES.nativeRunnerEvidenceRetention, { + retryMs: 30_000, canRun, + run: async signal => { + try { + const result = await service.tick({ shouldContinue: () => !signal.aborted && canRun() }); + if (result.evidenceCandidates > 0 || result.failed > 0) { + logger.info({ ...result, dryRun: options.dryRun ?? false }, "native runner evidence retention"); + } + continuation = result.truncated ? Date.now() + 1000 : null; + } catch (err) { + continuation = Date.now() + 60 * 60_000; + logger.warn({ err }, "native runner evidence inspection failed"); + } + }, + next: async () => continuation ?? service.nextEvidenceAt(), + }); + const unsubscribe = onNativeRunnerEvidence(at => { service.noteEvidence(at); worker.wake(); }); + return { ...worker, unsubscribe }; +} diff --git a/tests/e2e/disconnect-write-recovery.spec.ts b/tests/e2e/disconnect-write-recovery.spec.ts index e533361e775..8f81ee178dc 100644 --- a/tests/e2e/disconnect-write-recovery.spec.ts +++ b/tests/e2e/disconnect-write-recovery.spec.ts @@ -16,6 +16,7 @@ test("a task title survives an outage and saves after server recovery", async ({ data: { title: "Another task stays unchanged", status: "backlog" }, })); await page.goto(`/${company.issuePrefix}/issues/${issue.identifier}`); + await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 }); const header = page.getByTestId("issue-detail-header"); await expect(header.getByRole("heading", { name: issue.title, exact: true })).toBeVisible(); @@ -60,6 +61,7 @@ test("two cached tasks keep separate title saves while the browser is offline", await expect(page.getByRole("heading", { name: second.title, exact: true })).toBeVisible(); await page.getByRole("link", { name: "Tasks", exact: true }).first().click(); await page.getByRole("link").filter({ hasText: first.title }).first().click(); + await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 }); const header = page.getByTestId("issue-detail-header"); await expect(header.getByRole("heading", { name: first.title, exact: true })).toBeVisible(); const dismissAnnouncement = page.getByRole("button", { name: "Dismiss announcement" }); @@ -74,6 +76,7 @@ test("two cached tasks keep separate title saves while the browser is offline", await secondLink.focus(); await secondLink.press("Enter", { timeout: 5_000 }); await expect(page).toHaveURL(new RegExp(`${second.id}$`)); + await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 }); await expect(header.getByRole("heading", { name: second.title, exact: true })).toBeVisible(); await header.getByRole("heading", { name: second.title, exact: true }).click(); await expect(header.locator("textarea")).toHaveValue(second.title); @@ -189,6 +192,7 @@ test("startup work retries a failed session lookup while other APIs stay healthy await route.fulfill({ status: 500, json: { error: "session_lookup_failed" } }); }); await page.goto(`/${company.issuePrefix}/issues/${issue.id}`); + await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 }); const header = page.getByTestId("issue-detail-header"); await header.getByRole("heading", { name: issue.title, exact: true }).click(); await header.locator("textarea").fill("Saved after session recovery");