From 07c6df260bbf0dbdcf84d8a0a75d8c88f3a0128a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 29 Oct 2026 11:12:16 -0500 Subject: [PATCH] fix(server): schedule archived runner evidence from saved deadlines (#17925) https://entire.com/gh/paperclipai/-/trails/85/changes/paperclip/85 ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Retired native sessions leave continuity and quarantine evidence on disk. > - The old hourly loop mixes evidence retention with backup expiry, which has different database eligibility rules. > - Evidence already has saved creation timestamps and local publication signals. > - This pull request gives evidence its own registered worker and actual next deadline. > - The worker stops filesystem and SQL activity when its evidence obligations settle. ## Linked Issues or Issue Description Stack slice 65. Parent: #17923 (`codex/sleep-64-skill-version-retention`). Related foundation: #15625. **What happened?** At parent `7b94aa185c29673372646bbf2e07a2fbd16bc394`, an hourly backup/evidence tick still inspects native state roots. Evidence's private due time is checked by that recurring tick instead of the shared scheduler. Empty evidence and future retention windows have no reason to be rediscovered every hour. **Expected behavior** Reconstruct evidence once. Wake from the existing publication signal. Register the earliest saved retention deadline. Keep busy, failed, or incomplete evidence recoverable. Preserve backup eligibility as a separate duty. **Steps to reproduce** Start retention with an empty evidence root or archives inside the seven-day window. The old timer still visits the combined owner. The new evidence worker disarms when empty, or registers the saved expiry with no evidence filesystem or SQL calls between relevant changes and that deadline. ## What Changed - Add an evidence worker through the delivery coordinator and shared scheduler. Track startup reconstruction without delaying HTTP readiness. Unsubscribe local publication before delivery shutdown. - Select `backups` for the existing backup timer and `evidence` for the new worker. Existing callers that omit the selection retain combined behavior. Backup expiration remains the next separate migration. - Use hourly retries for concrete busy or failed evidence, with earlier saved expiries and one-second continuations independent. Restore candidate/removal/byte summaries. Schedule publication at the first removable instant; preserve legacy manual-tick semantics. - Expose the existing earliest evidence deadline. Preserve one-second bounded continuation, concrete failure/busy retries, cleanup-source archive exclusions, symlink protection, late activity checks, atomic rename, and interrupted removal recovery. - Preserve non-ENOENT inspection errors as recovery obligations. Retry a failed session root in a new evidence pass instead of continuing past it and forgetting it. - Register the first removable instant after the inclusive retention boundary. Keep the existing manual dry-run publication semantics. - Add real filesystem/isolated PostgreSQL lifecycle tests and disabled/held-startup wiring cases. Add no table or migration. The archive and quarantine writers already call `noteNativeRunnerEvidence`; this PR connects that existing signal to the scheduler. Shared code owns scheduling, while evidence rules stay with the retention service. See [`doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md`](doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md). ## Verification - Adopted actual prerequisite squash `b06ea111a254c62e0a55199b010d82e0adb33bf0` from #17923 through normal merges. Qualified runtime/package/UI/map trees remain unchanged. The final nine-path diff includes one browser-spec readiness repair. - Runtime qualification remains **652 passing TypeScript cases across 22 complete files**, plus **29 passing Rust cases** in native-backend, runtime-notification and configured-environment targets. One actual closed Pi snapshot launch is intentionally skipped because its external distribution root is not provided. Accepted source is `b80eee32f344ae0218d8b791c14f2ea174da09cc`; all runtime and earlier sleep-test bytes are preserved. - The complete outage-recovery browser file passes **all five journeys** on a fresh isolated database after four additive rendered-history readiness assertions. Every original assertion, outage control and timeout remains byte-identical when those new lines are removed. The intentionally unresolved first-session scenario keeps its pre-auth interaction and delayed-owner coverage. The prior CI/local timeout evidence and one rejected broader precondition remain in the audit. - Adapter/Runner/database builds, server/Runner typechecking, Rust formatting/workspace checks, server build, module boundaries, migration order and generated TypeScript/Rust profiles pass. PostgreSQL 17 fixtures apply all 413 journal entries through index 414; source hashes and temporary fixture imports restore exactly. - **Inherited validation exception:** Copilot profile 81's separate source-identity check fails on the merged Rust configured-environment hash. Its inputs and attestations equal upstream and stay unchanged here. Grok's identity check passes. No release attestation is regenerated or new live-provider qualification claimed. Required exact-head CI/security and Greptile 5/5 still must pass. - [The evidence audit](doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md) retains historical failures, counts and limitations. No full local repository-suite, live provider, external-alarm, measured suspension or fleet-savings claim. Fresh final-head CI/security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - Live publication is the existing server-owned executor in one process. Independent filesystem writers need an explicit signal contract. Startup reconstructs restart state. - Busy sessions and concrete inspection/removal failures retain retries. Unknown cleanup-source archives remain protected. - The existing scan enumerates directory names before applying its entry/removal budget. This PR preserves that behavior and does not claim a bounded total directory-listing allocation. - An in-process deadline cannot wake stopped compute. Cloud sleep-safety and external-alarm coverage stay unchanged. Physical evidence reclamation can wait for a supported host wake. - Backup expiration still has an hourly database eligibility scan. This slice removes only the archived-evidence duty from that loop and does not claim full instance suspension. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, normal Git merges, composition proofs and complete PostgreSQL/filesystem/startup suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip Co-authored-by: Codex --- ...2026-10-24-recurring-server-sleep-stack.md | 12 + ...7-runner-evidence-retention-sleep-audit.md | 290 ++++++++++++++++++ .../native-harness-backup-expiration.test.ts | 161 +++++++++- .../server-startup-feedback-export.test.ts | 38 +++ server/src/index.ts | 13 +- .../services/delivery-work-notifications.ts | 1 + .../native-harness-backup-expiration.ts | 69 +++-- .../native-runner-evidence-retention-work.ts | 34 ++ tests/e2e/disconnect-write-recovery.spec.ts | 4 + 9 files changed, 596 insertions(+), 26 deletions(-) create mode 100644 doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md create mode 100644 server/src/services/native-runtime/native-runner-evidence-retention-work.ts diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index a55eed6b7d5..b9fba65827b 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -3525,3 +3525,15 @@ writer inventory, module contracts, local acceptance, and limitations. Remote review and checks remain pending. The user's October 27 instruction now authorizes merging qualifying sleep PRs in dependency order after Greptile 5/5 and any present Ponytail review passes, with all required CI checks passing. + +## Slice 65 — archived runner evidence retention (2026-10-27) + +Split evidence retention from native-harness backup eligibility in their shared +legacy service. Evidence now reconstructs once, wakes from the existing archive/ +quarantine publication signal, and uses saved retention timestamps. Busy or failed +items retain recovery and bounded continuation. The backup timer selects only its +own duty and remains for the next slice. See the +[evidence audit](2026-10-27-runner-evidence-retention-sleep-audit.md) for producer, +clock-boundary, interrupted-cursor, writer-model, and verification evidence. +Local acceptance passes 158 cases plus server typechecking/build. Remote review +and CI are pending; this is not a deployed sleep claim. diff --git a/doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md b/doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md new file mode 100644 index 00000000000..ae9688ba345 --- /dev/null +++ b/doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md @@ -0,0 +1,290 @@ +# Runner evidence retention scheduling + +Slice 65, parent slice 64 (#17923) at +`d2ccbcdfc9118b8841299da65e94e487807e73ab` before its review repair. +This slice owns archived runner evidence. Native-harness backup expiration is a +separate remaining duty, although the old implementation inspects both in one loop. +The default branch was observed at `50b932a142216331b8d476042b454bc123739c40`. +The stack's pinned integration baseline remains `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`; +the later default-branch changes require final stack integration and verification. + +## Obligation and existing persistence + +Continuity-break archives and identity quarantine directories preserve evidence +from a retired native session. Their names contain the creation timestamp. The +existing retention service falls back to mtime for a timestamp in the future, +preserves cleanup-source quarantine archives, refuses symlinks, checks session +activity twice before removal, and moves eligible evidence to a unique expired +name before recursive removal. Interrupted expired directories are recovered. + +These filesystem facts already persist the retention obligation. There is no +reason to add a database table. No database work is needed for evidence inspection. +The existing hourly tick still invokes the backup-expiration path, whose eligibility +also depends on issues, runs, finalizations, and leases. Moving those two duties +independently keeps each PR's trigger and correctness claim reviewable. + +## Triggers and module ownership + +- Startup reconstructs evidence once using the existing bounded session/archive/ + quarantine continuation state. +- `quarantineRunnerdStateRoot` publishes `noteNativeRunnerEvidence` after the + quarantine rename. `archiveContinuityState` publishes the same signal when it + creates the continuity archive, before it moves the retired state. The saved + retention deadline is days later; the signal does not authorize immediate + deletion or claim that the archive's write is complete. +- A publication adds its timestamp to the existing earliest evidence deadline and + wakes the registered worker. Earlier deadlines continue to win. The next due + inspection reconstructs all retained evidence, including multiple writes. +- A recent item contributes its saved creation time plus the configured retention + window and one millisecond, because the existing protection comparison is + inclusive at the window boundary. +- Busy sessions, incomplete scans, and failed inspection/removal retain actual + recovery obligations. Bounded continuations run after one second. Concrete + failed or busy evidence retries use a one-hour worker retry; an earlier saved + expiry or new publication can still wake the worker sooner. +- Empty evidence, ignored cleanup-source archives, and fully processed history + leave no evidence deadline or recurring filesystem/SQL calls. + +The existing retention owner exposes `nextEvidenceAt()` and a per-instance duty +selection. Legacy callers that omit the selection keep combined behavior. Startup +selects `backups` for the existing backup timer and registers an `evidence` instance +with the delivery coordinator. The shared scheduler/coordinator owns overlap, +admission, drain, retries, deadlines, and shutdown. No feature logic moves into +`index.ts`. The worker's first promise stays tracked without delaying readiness; +shutdown unsubscribes the local signal before stopping/joining delivery work. + +The backup interval remains deliberately visible for the next independent slice. +This PR removes evidence from that interval; it does not claim that the instance +as a whole is idle while backup and other recurring duties remain. + +Supported publication is the existing server-owned session executor in one +process. Unrelated filesystem writers need an explicit signal/ingress contract. +Startup reconstructs restart state; it is not cross-process live notification. +Cloud sleep gates and external alarm coverage are unchanged. Expiration can wait +for a supported host wake when compute is stopped. + +## Failure record and local evidence + +Two regressions failed before repair: the saved deadline was one millisecond +before the first removable instant, and an EACCES reading an archive directory +was treated as an empty directory and disarmed retention. Missing paths remain +quiet; other inspection failures now retain recovery. + +A further controlled EACCES at the session root reproduced a lost cursor: retry +continued after the failed scope and left its evidence behind. The owner now +records that concrete failed inspection for a new full evidence pass. Healthy +scopes continue, and the failed scope is revisited after the bounded retry. + +An intermediate full run exposed a compatibility regression in the existing +manual dry-run notification case. The publication timestamp's original due +semantics were restored; the inclusive boundary adjustment applies to the +observed recent item's next inspection. The old test and deletion protections +were preserved rather than weakened. + +Final local acceptance: 158 cases pass across native-harness/evidence retention, +retention configuration, and startup. The retention suite uses a guarded isolated +PostgreSQL database; all selected cases execute. Tests measure quiet filesystem +and SQL behavior, saved deadlines, local publication, restart, busy/failure retry, +first work during drain, one-entry continuation, absence of backup SQL in the +evidence worker, disabled registration, and tracked startup after HTTP readiness. +Existing backup, manifest, lease, issue, dry-run, symlink, and cleanup-archive +cases still pass. Server TypeScript checking and the full server build also pass. +Remote review/CI and whole-stack acceptance remain pending. + +## Ponytail repair: quiet handled retries, summaries, and publication boundary + +Controlled tests reproduced a busy archive causing whole-tree inspection every +thirty seconds, that overdue retry masking another archive's ten-second saved +expiry, missing deletion/byte summaries, and a publication deadline one millisecond +before the first removable instant. All four fail before this repair. + +The evidence-only owner now records an hourly retry for handled busy/inspection/ +removal failures. Genuine earlier saved expiries remain the minimum deadline. +One-second truncated continuations remain independent. Unexpected state-base +inspection failure is also retained as an hourly worker retry; a new publication +can reactivate it immediately. Admission/drain reconciliation still uses the +coordinator's existing short memory-only retry. Legacy combined/manual callers +retain their original explicit-tick behavior. + +The worker again logs candidate, removal, reclaimed-byte, failure, and dry-run +summary fields when it processes candidates or failures. Publication now schedules +the first removable instant in evidence-only mode; the old manual notification +case is unchanged. An added base-directory fault test proves five minutes of quiet +before a publication resumes processing. Final owning acceptance passes 162 cases +across retention, configuration, and startup, plus server typechecking. Earlier +receipts remain tied to their recorded heads; integrated verification follows the +parent skill-retention repair. + +## October 29 master reconciliation and current acceptance + +This existing archived-evidence slice was reconciled with the qualified retention +parent through normal merges. The expiration service now exposes both the new +saved evidence deadline and the parent's separate startup checkpoint methods. +The finite checkpoint failure budget remains intact. The native retention fixture +keeps its explicit external-database override and migrations while using the +parent's fresh recovery-database helper by default. Identical logger/coordinator +imports introduced by automatic composition were deduplicated. Every other path +matches the automatic merge, and packages remain byte-identical to the parent. + +Current local acceptance is **285 passing unique cases across 11 complete files**, +without skips. The original combined run passed 283 cases but failed two new +startup fixtures because their mocked service omitted the preserved +`registerCheckpointExpiration` method. Production startup correctly called that +method. The fixture repair supplies the actual API and adds explicit assertions +that enabled and disabled evidence configurations retain checkpoint registration, +including while evidence reconstruction is held after HTTP readiness. The full +100-case startup file then passed in 7.55 seconds, without weakening any prior +assertion or timeout. The other ten complete files passed 185 cases and their +source remained unchanged. Accepted source after that fixture-only repair is +`da44dea2f3cd86cc72a25f2a419c63acafda1f7a`. + +Coverage includes real archived-evidence expiry, filesystem and path safety, +inclusive retention boundaries, retry and paged continuation, preserved finite +checkpoint startup work, backup cleanup/configuration, server wiring, shared +deadline/coordinator and idle-work behavior, plus retained skill-cache behavior +and its own-company expiry regression. Database build, production server +typechecking/build, module boundaries, target migration order and Runner profile +identity passed before the fixture-only correction; runtime source did not change +afterward. Two fresh PostgreSQL 17 schemas plus per-file fixtures applied all 412 +journal entries through index 413. All temporary local fixture imports were +restored exactly, and every selected test hash is bound in the acceptance record. + +The archive worker uses the evidence duty and performs no backup SQL. The backup +duty and startup checkpoint owner remain distinct. Saved publication/retention +boundaries and one-hour busy/failure retries retain nearer expiries and bounded +one-second continuation pages. Unsubscription still precedes coordinator stop. +Publication must adopt the actual merged skill-retention prerequisite and obtain +fresh exact-head CI/security and reviews. No full local repository-suite, live +provider, external wake, measured instance/database suspension or fleet-cost claim +is made. Historical results and limitations above remain tied to their revisions. + +## Requalification with later upstream changes + +Normal reconciliation preserved the final skill-retention repair and all 95 +unrelated master paths from the actual timer squash, task activity index and task +read changes. On source `2fa458833fa43b47eddc728a5e57f5ffcf5fa3e4`, all **285 +cases across 11 complete files passed together**, with no skips, in 47.26 seconds. +This fresh full selection includes the preserved checkpoint APIs and their +repaired startup fixtures. Database build, production server typechecking/build, +module boundaries, target migration order and generated TypeScript/Rust release +profile consistency passed. Fresh PostgreSQL 17 schemas applied all 413 journal +entries through index 414. Every selected source hash and temporary fixture import +restored exactly. The final parent audit was then adopted through a doc-only +normal merge without changing any accepted runtime/test/package tree. + +The inherited upstream Copilot source-identity failure is recorded in the +[parent retention audit](2026-10-27-skill-version-retention-sleep-audit.md#final-current-master-qualification-before-publication). +This slice leaves its release attestation and all identity inputs unchanged. +Generated admission still matches the active release manifest; the separate +source-identity computation is an explicit inherited exception, not a pass or a +claim of new live-provider qualification. Required fresh exact-head CI/security +and reviews remain merge gates. Actual parent squash adoption is still required +before publication; any new upstream behavior must remain preserved and receive +appropriate combined-context checks. + +## Workspace-seed upstream composition + +The next upstream merge (`27fbdb2a74e13141065555b6a2fa5acdd8da7964`) +optimizes verified workspace seed reuse. It is preserved by a normal merge, +including its adapter implementation, tests and native workspace history. On +combined source `a288a58287047193774eda54601ac9eda732d2e8`, **321 cases +across 14 complete files passed together**, without skips, in 95.73 seconds. +This includes the previous complete 11-file retention selection and the full +workspace-seed reuse, GitHub launcher and native workspace-history files. These +results replace, rather than add to, the earlier 285-case acceptance total for +this combined source. Adapter/database builds, server typechecking/build, module +boundaries, target migration order and generated release profiles pass. Every +selected source hash and temporary fixture import restores exactly. Provider +identity inputs remain unchanged; the inherited exception above still applies. + +Actual skill-retention squash adoption remains a publication prerequisite. If +its tree carries only these already accepted upstream changes, tree identity +will prove that adoption changes no accepted runtime. Any additional behavior +will receive scoped combined-context validation before publication. + +## Runner-authority upstream qualification + +Master `27be8590442fc02d29a6c257a4e9dbe5bbbd66d2` then introduced Runner +authority for runtime activity and updated provider manifests. Normal composition +through `b80eee32f344ae0218d8b791c14f2ea174da09cc` preserves its complete +package tree byte for byte, the accepted workspace changes and the parent browser +readiness repair. The server retention implementation is unchanged. + +On that combined source, **652 TypeScript cases across 22 complete files passed**. +These include the complete 14-file evidence-retention/workspace selection (321 +cases) and eight affected Runner files (331 cases). One actual closed Pi snapshot +launch is intentionally skipped because its external distribution root is not +provided; this is not a live-distribution qualification. **29 Rust cases passed** +in the full native-backend target (26), full runtime-notification target (one), +and both configured-environment unit cases. Adapter/Runner/database builds, Runner +and server typechecking, Rust formatting/workspace checking, server build, module +boundaries, migration order and generated TypeScript/Rust profiles all passed. +Fresh PostgreSQL 17 schemas apply all 413 entries through index 414. All selected +source hashes and temporary imports restore exactly. These fresh counts replace +rather than add to the prior selections. + +The new upstream manifest uses Copilot profile 81, but its separate source-identity +check still fails: the active fixture's `configuredEnvironmentRustSourceSha256` +differs from the actual merged Rust file. Every bound source and the entire +package tree equal master. This preserves the earlier inherited OpenCode-key +identity drift through the upstream profile update; no retention edit causes it. +Grok's separate identity check passes. The Copilot exception remains explicit, +and no release attestation is regenerated or new live-provider qualification +claimed. Required fresh exact-head CI/security and reviews remain merge gates. + +Actual parent squash adoption remains necessary before final publication. The +new parent browser-repair commit and its original assertions are carried exactly; +the browser gate itself must still be green on the final published head. + +## Actual prerequisite adoption for final publication + +The skill-retention prerequisite [#17923](https://github.com/paperclipai/paperclip/pull/17923) +merged at `b06ea111a254c62e0a55199b010d82e0adb33bf0`, after all 58 current +checks were clean and the exact reviewed head `269935f6ccd7685866fd94bea66f7f803c48058d` +had Greptile and Ponytail 5/5. Its separate, evidence-supported failed-job retry +passed; no failing gate or required review finding was waived. + +This last slice adopts that actual squash through normal merge +`3c40eaad954f7705144e6f378d93be2adff0c54a`. Duplicate ancestry conflicts were +resolved to the proven composed tree. The complete pre-adoption held tree is +unchanged, including the 652-case TypeScript and 29-case Rust accepted sources, +all selected tests and the parent's browser-readiness repair. Package, UI, E2E +and map trees equal the actual prerequisite. The final diff has eight owned +paths. This adoption changes ancestry rather than runtime behavior, so accepted +checks are not repeated; this paragraph and the PR body only record provenance. +The original remote head remains an ancestor, and no history is force-pushed. + +Fresh required CI/security, exact-head Greptile 5/5 and any actionable Ponytail +findings are still gates on this slice's final publication. The inherited source +identity exception and all bounded-acceptance limitations above remain explicit. + +## Final outage-journey readiness repair + +The first final publication, `762d48605f93351e47f9fd1249d9293625834242`, +received exact-head Greptile 5/5 and Ponytail 5/5 with no actionable findings. +Its required checks completed cleanly except the second browser shard and its +aggregate. The title-outage journey reached the 60-second global test timeout; +its failure artifact still shows the original title, and no issue write was +observed. The same file then passed that case locally but reproduced a timeout +in the failed-session startup case (four passing cases and one failure). These +failures and the original assertions remain recorded; no failing gate is waived. + +A visible initial task heading alone does not establish that the conversation +layout has settled after the upstream loading changes. The test now requires +visible conversation history before the ordinary title-edit flows, including +the cached offline task switch. Four additive readiness assertions are the only +spec changes. Removing those four lines reproduces the exact prior file bytes: +all original recovery assertions, outage controls, and timeout budgets remain. +The deliberately held first-session-lookup case retains its original pre-auth +readiness, so it still queues a write before the owner is known. An initial +broader readiness attempt blocked that intentional startup state; that extra +condition was removed rather than changing the scenario. + +All **five complete outage-recovery browser journeys pass** afterward on a new +isolated PostgreSQL 17 database. They still verify failed-save retention, actual +retry after recovery, separation of two offline drafts, delayed owner binding, +comment replay idempotency and failed-session retry. The accepted 652-case +TypeScript and 29-case Rust runtime sources, all prior sleep regression files, +packages and UI are unchanged. The final nine-path diff adds this spec-readiness +repair to the eight-path feature. Fresh exact-head CI/security and reviews are +required after publication; older-head review passes are not reused. diff --git a/server/src/__tests__/native-harness-backup-expiration.test.ts b/server/src/__tests__/native-harness-backup-expiration.test.ts index 764539c2454..b24c12a80c9 100644 --- a/server/src/__tests__/native-harness-backup-expiration.test.ts +++ b/server/src/__tests__/native-harness-backup-expiration.test.ts @@ -1,3 +1,8 @@ +import { applyPendingMigrations } from "@paperclipai/db"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { registerNativeRunnerEvidenceRetentionWork } from "../services/native-runtime/native-runner-evidence-retention-work.js"; +import { noteNativeRunnerEvidence } from "../services/native-runtime/native-runner-evidence-signal.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; import { createHash, randomUUID } from "node:crypto"; import os from "node:os"; import path from "node:path"; @@ -20,9 +25,10 @@ import { nativeHarnessBackupExpirationService } from "../services/native-runtime import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; import { logger } from "../middleware/logger.js"; -const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_RETENTION_TEST_DATABASE_URL"); +const describeEmbeddedPostgres = external || embeddedPostgresSupport.supported ? describe : describe.skip; -if (!embeddedPostgresSupport.supported) { +if (!external && !embeddedPostgresSupport.supported) { console.warn( `Skipping embedded Postgres native harness backup expiration tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`, ); @@ -41,11 +47,17 @@ describeEmbeddedPostgres("native harness backup expiration", () => { const now = new Date(); beforeAll(async () => { - tempDb = await startRecoveryPostgres("paperclip-native-backup-expiration-"); - db = createDb(tempDb.connectionString); + if (!external) tempDb = await startRecoveryPostgres("paperclip-native-backup-expiration-"); + if (external) await applyPendingMigrations(external); + db = createDb(external ?? tempDb!.connectionString); }, 20_000); + let coordinator: ReturnType | undefined; + let worker: ReturnType | undefined; + let admitted = true; afterEach(async () => { + worker?.unsubscribe(); await coordinator?.stop(); coordinator = undefined; worker = undefined; admitted = true; + vi.useRealTimers(); vi.restoreAllMocks(); await db.delete(agentTaskSessions); await db.delete(nativeRunFinalizations); await db.delete(environmentLeases); @@ -58,6 +70,7 @@ describeEmbeddedPostgres("native harness backup expiration", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); @@ -522,6 +535,146 @@ describeEmbeddedPostgres("native harness backup expiration", () => { return createHash("sha256").update(randomUUID()).digest("hex"); } + async function startEvidence(options: Parameters[1] = {}) { + if (!vi.isFakeTimers()) { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + vi.setSystemTime(now); + } + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => true, + onError: error => { throw error; } }); + worker = registerNativeRunnerEvidenceRetentionWork(coordinator, db, { stateBase, isSessionBusy: () => false, ...options }, () => admitted); + await worker.ready; + } + async function settled(deadline: number | null = null) { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBe(deadline); + }, { timeout: 5000 }); + } + async function quiet(ms = 3 * 60 * 60_000) { + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"), directories = vi.spyOn(fs, "readdir"); + await vi.advanceTimersByTimeAsync(ms); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); expect(directories).not.toHaveBeenCalled(); + reads.mockRestore(); writes.mockRestore(); directories.mockRestore(); + } + + it("leaves empty evidence and retained backups quiet without invoking backup SQL", async () => { + const session = await seedSession(); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"); + await startEvidence(); await settled(); expect(await exists(session.current)).toBe(true); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + reads.mockRestore(); writes.mockRestore(); await quiet(); + }); + + it("reconstructs and replaces the saved evidence deadline after a publication signal", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const first = await seedArchive(session.root, { stampedDaysAgo: 2 }); + const firstDeadline = now.getTime() + 5 * DAY_MS + 1; + await startEvidence(); await settled(firstDeadline); await quiet(); + const writtenAt = Date.now(); + const next = await seedArchive(session.root, { name: `${writtenAt}-${randomUUID()}` }); + noteNativeRunnerEvidence(writtenAt); + await settled(firstDeadline); + await vi.advanceTimersByTimeAsync(firstDeadline - Date.now()); + await settled(writtenAt + 7 * DAY_MS + 1); expect(await exists(first)).toBe(false); + await vi.advanceTimersByTimeAsync(writtenAt + 7 * DAY_MS + 1 - Date.now()); await settled(); + expect(await exists(next)).toBe(false); expect(await exists(session.current)).toBe(true); await quiet(); + }); + + it.each(["session root", "archive directory"])("preserves an evidence inspection failure at %s until its concrete retry succeeds", async location => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root); + const original = fs.lstat; + const denied = vi.spyOn(fs, "lstat").mockImplementation(async (...args) => { + if (String(args[0]) === (location === "session root" ? session.root : path.join(session.root, "continuity-breaks"))) throw Object.assign(new Error("unavailable"), { code: "EACCES" }); + return original(...args); + }); + await startEvidence(); expect(coordinator!.nextWakeAt()).not.toBeNull(); + const retryAt = coordinator!.nextWakeAt()!; expect(await exists(archive)).toBe(true); + denied.mockRestore(); await quiet(5000); + await vi.advanceTimersByTimeAsync(retryAt - Date.now()); await settled(); + expect(await exists(archive)).toBe(false); await quiet(); + }); + + it("retries a busy evidence obligation and disarms after the session releases it", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root); let busy = true; + await startEvidence({ isSessionBusy: () => busy }); + const retryAt = coordinator!.nextWakeAt()!; expect(retryAt - Date.now()).toBeGreaterThanOrEqual(60 * 60_000); + await quiet(5000); busy = false; + await vi.advanceTimersByTimeAsync(retryAt - Date.now()); await settled(); + expect(await exists(archive)).toBe(false); await quiet(); + }); + + it("retains the first publication during drain and cleans up on resume", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + await startEvidence(); await settled(); admitted = false; + const archive = await seedArchive(session.root); + noteNativeRunnerEvidence(now.getTime() - 10 * DAY_MS); + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(1000); expect(idleWorkSnapshot().active).toBe(0); }); + await quiet(60_000); expect(await exists(archive)).toBe(true); + admitted = true; await settled(); expect(await exists(archive)).toBe(false); await quiet(); + }); + + it("reconstructs the same retained deadline after restart", async () => { + const scope = await emptyStateBase(); await seedQuarantine(scope, { stampedDaysAgo: 6 }); + const deadline = now.getTime() + DAY_MS + 1; + await startEvidence(); await settled(deadline); + worker!.unsubscribe(); await coordinator!.stop(); + await startEvidence(); await settled(deadline); await quiet(60_000); + expect(coordinator!.nextWakeAt()).toBe(deadline); + }); + + it("backs off a failed state-base inspection and lets publication reactivate it", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root), read = fs.readdir; + const denied = vi.spyOn(fs, "readdir").mockImplementation(async (...args) => { + if (args[0] === stateBase) throw Object.assign(new Error("state base unavailable"), { code: "EACCES" }); + return read(...args); + }); + await startEvidence(); expect(coordinator!.nextWakeAt()! - Date.now()).toBeGreaterThanOrEqual(60 * 60_000); + denied.mockRestore(); await quiet(5 * 60_000); expect(await exists(archive)).toBe(true); + noteNativeRunnerEvidence(now.getTime() - 10 * DAY_MS); + await settled(); expect(await exists(archive)).toBe(false); await quiet(); + }); + + it("lets healthy saved expiry preempt a long-busy evidence retry", async () => { + const busy = await seedSession({ issueStatus: "in_progress" }), healthy = await seedSession({ issueStatus: "in_progress" }); + const held = await seedArchive(busy.root); + const dueAt = now.getTime() + 10_001; + const available = await seedArchive(healthy.root, { name: `${dueAt - 7 * DAY_MS - 1}-${randomUUID()}` }); + await startEvidence({ isSessionBusy: scope => scope === busy.scope }); await settled(dueAt); + await vi.advanceTimersByTimeAsync(dueAt - Date.now()); + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(100); expect(await exists(available)).toBe(false); }); + expect(await exists(held)).toBe(true); expect(coordinator!.nextWakeAt()! - Date.now()).toBeGreaterThan(59 * 60_000); + await quiet(60_000); + }); + + it("reports reclaimed evidence bytes and dry-run mode", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archive = await seedArchive(session.root); const reported = vi.spyOn(logger, "info"); + await startEvidence(); await settled(); + expect(reported.mock.calls).toEqual(expect.arrayContaining([[expect.objectContaining({ evidenceCandidates: 1, + evidenceRemoved: 1, evidenceReclaimedBytes: expect.any(Number), dryRun: false }), "native runner evidence retention"]])); + expect(await exists(archive)).toBe(false); await quiet(); + }); + + it("schedules publication at the first removable instant after its retention boundary", async () => { + const scope = await emptyStateBase(); await startEvidence(); await settled(); await quiet(); + const writtenAt = Date.now(); + await seedQuarantine(scope, { name: `${scope}.identity_indeterminate.${writtenAt}.4242.${randomUUID()}` }); + noteNativeRunnerEvidence(writtenAt); await settled(writtenAt + 7 * DAY_MS + 1); + }); + + it("uses bounded continuations without recounting an evidence session forever", async () => { + const session = await seedSession({ issueStatus: "in_progress" }); + const archives = [await seedArchive(session.root), await seedArchive(session.root)]; + const quarantine = await seedQuarantine(session.scope); + await startEvidence({ maxEntries: 1 }); await settled(); + for (const archive of [...archives, quarantine]) expect(await exists(archive)).toBe(false); + await quiet(); + }); + it("removes continuity archives older than the retention window and keeps recent ones", async () => { const session = await seedSession({ issueStatus: "in_progress" }); const old = await seedArchive(session.root, { stampedDaysAgo: 9 }); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index dc0e05e249c..420a18bbc86 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -1,3 +1,4 @@ +import { nativeHarnessBackupExpirationService } from "../services/native-runtime/native-harness-backup-expiration.js"; import { skillVersionRetentionService } from "../services/skill-version-retention.js"; import { approvalQueueSeedingService } from "../services/approval-queue-seeding-work.js"; import { joinRequestRetentionService } from "../services/join-request-retention.js"; @@ -539,6 +540,18 @@ vi.mock("../board-claim.js", () => ({ initializeBoardClaimChallenge: vi.fn(async () => undefined), })); +vi.mock("../services/native-runtime/native-harness-backup-expiration.js", () => ({ + NATIVE_HARNESS_BACKUP_EXPIRATION_INTERVAL_MS: 60 * 60_000, + NATIVE_HARNESS_BACKUP_EXPIRATION_STARTUP_DELAY_MS: 2 * 60_000, + NATIVE_HARNESS_BACKUP_EXPIRATION_DEFAULT_DAYS: 7, + NATIVE_RUNNER_EVIDENCE_RETENTION_DEFAULT_DAYS: 7, + nativeHarnessBackupExpirationService: vi.fn(() => ({ + tick: vi.fn(async () => ({ checked: 0, candidates: 0, evidenceCandidates: 0, failed: 0, truncated: false })), + noteEvidence: vi.fn(), nextEvidenceAt: vi.fn(() => null), + tickCheckpoints: vi.fn(), registerCheckpointExpiration: vi.fn(), + })), +})); + vi.mock("../services/skill-version-retention.js", () => ({ skillVersionRetentionService: vi.fn(() => ({ listCompanies: vi.fn(async () => []), pruneCompany: vi.fn(async () => ({ checked: 0, truncated: false, nextRunAt: null })), @@ -582,6 +595,31 @@ describe("startServer feedback export wiring", () => { process.env.BETTER_AUTH_SECRET = "test-secret"; }); + it.each([false, true])("registers runner evidence independently from the backup scan (%s)", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ nativeHarnessBackupExpirationEnabled: enabled })); + await startServer(); await settleStartupHousekeeping(); + expect(deliveryWorkRegisterMock.mock.calls.filter(([queue]) => queue === DELIVERY_QUEUES.nativeRunnerEvidenceRetention)).toHaveLength(enabled ? 1 : 0); + expect(vi.mocked(nativeHarnessBackupExpirationService).mock.calls.map(([, options]) => options?.duty)).toEqual(enabled ? ["backups", "evidence"] : ["backups"]); + const backups = vi.mocked(nativeHarnessBackupExpirationService).mock.results[0]!.value; + expect(backups.registerCheckpointExpiration).toHaveBeenCalledTimes(enabled ? 1 : 0); + }); + + it("tracks held evidence reconstruction after HTTP readiness", async () => { + loadConfigMock.mockReturnValue(buildTestConfig({ nativeHarnessBackupExpirationEnabled: true })); + const pending = Promise.withResolvers<{ truncated: boolean }>(); + vi.mocked(nativeHarnessBackupExpirationService).mockReturnValueOnce({ tick: vi.fn(), noteEvidence: vi.fn(), nextEvidenceAt: () => null, + tickCheckpoints: vi.fn(), registerCheckpointExpiration: vi.fn() } as never); + vi.mocked(nativeHarnessBackupExpirationService).mockReturnValueOnce({ tick: vi.fn(() => pending.promise), noteEvidence: vi.fn(), nextEvidenceAt: () => null, + tickCheckpoints: vi.fn(), registerCheckpointExpiration: vi.fn() } as never); + try { + await startServer(); await settleStartupHousekeeping(); + expect(getStartupRecoveryState().phase).toBe("ready"); expect(idleWorkSnapshot().active).toBe(1); + expect(vi.mocked(nativeHarnessBackupExpirationService).mock.results[0]!.value.registerCheckpointExpiration).toHaveBeenCalledOnce(); + } finally { pending.resolve({ truncated: false }); await settleStartupHousekeeping(); } + expect(idleWorkSnapshot().active).toBe(0); + expect(deliveryWorkErrors.filter(error => error.queue === DELIVERY_QUEUES.nativeRunnerEvidenceRetention)).toEqual([]); + }); + it.each([false, true])("registers skill cache retention only when enabled (%s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ skillVersionRetentionEnabled: enabled })); await startServer(); await settleStartupHousekeeping(); diff --git a/server/src/index.ts b/server/src/index.ts index 76ec3cf072b..1c51bfb6a8b 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -32,7 +32,7 @@ import { NATIVE_HARNESS_BACKUP_EXPIRATION_STARTUP_DELAY_MS, nativeHarnessBackupExpirationService, } from "./services/native-runtime/native-harness-backup-expiration.js"; -import { onNativeRunnerEvidence } from "./services/native-runtime/native-runner-evidence-signal.js"; +import { registerNativeRunnerEvidenceRetentionWork } from "./services/native-runtime/native-runner-evidence-retention-work.js"; import { gitRestorePointRetentionService, registerGitRestorePointWork } from "./services/git-restore-point-ledger.js"; import { nativeWorkspaceSyncRetentionService, @@ -1259,12 +1259,12 @@ async function startServerWithDatabaseTeardown( }; const skillSourceSync = skillSourceSyncService(db as any); const nativeHarnessBackupExpiration = nativeHarnessBackupExpirationService(db as any, { + duty: "backups", dryRun: config.nativeHarnessBackupExpirationDryRun, closedForDays: config.nativeHarnessBackupExpirationDays, evidenceRetentionDays: config.nativeRunnerEvidenceRetentionDays, shouldContinue: () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby(), }); - const unsubscribeNativeRunnerEvidence = onNativeRunnerEvidence(nativeHarnessBackupExpiration.noteEvidence); let nativeHarnessBackupExpirationInFlight = false; let nativeHarnessBackupExpirationContinuation: ReturnType | undefined; const sweepNativeHarnessBackups = () => { @@ -1344,6 +1344,13 @@ async function startServerWithDatabaseTeardown( server.on("close", () => { void taskDrainExpiryWork.stop(); }); const executionSchedule = executionControlSchedule(db); const workspaceRecoveryRuntime = environmentRuntimeService(db, { pluginWorkerManager }); + const nativeRunnerEvidenceRetentionWorker = config.nativeHarnessBackupExpirationEnabled + ? registerNativeRunnerEvidenceRetentionWork(deliveryWork, db as any, { + dryRun: config.nativeHarnessBackupExpirationDryRun, + evidenceRetentionDays: config.nativeRunnerEvidenceRetentionDays, + }, () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby()) + : null; + if (nativeRunnerEvidenceRetentionWorker) trackHeartbeatSchedulerWork(nativeRunnerEvidenceRetentionWorker.ready); const skillVersionRetentionWorker = config.skillVersionRetentionEnabled ? registerSkillVersionRetentionWork(deliveryWork, skillVersionRetentionService(db as any, { dryRun: config.skillVersionRetentionDryRun, @@ -2191,7 +2198,7 @@ async function startServerWithDatabaseTeardown( clearInterval(nativeHarnessBackupExpirationInterval); clearTimeout(nativeHarnessBackupExpirationStartup); clearTimeout(nativeHarnessBackupExpirationContinuation); - unsubscribeNativeRunnerEvidence(); + nativeRunnerEvidenceRetentionWorker?.unsubscribe(); await skillSourceSync.stop(); unsubscribeChatCompletions(); unsubscribeNativeResponsePresentation?.(); diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index ef031e443ef..0e8748ad226 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; /** Topics for existing durable queues; register intent before their writes. */ export const DELIVERY_QUEUES = { + nativeRunnerEvidenceRetention: "native-runner-evidence-retention", skillVersionRetention: "skill-version-retention", databaseBackups: "database-backups", importSpoolCleanup: "import-spool-cleanup", diff --git a/server/src/services/native-runtime/native-harness-backup-expiration.ts b/server/src/services/native-runtime/native-harness-backup-expiration.ts index 750085d1e09..8ea9a242629 100644 --- a/server/src/services/native-runtime/native-harness-backup-expiration.ts +++ b/server/src/services/native-runtime/native-harness-backup-expiration.ts @@ -40,6 +40,7 @@ const MAX_CONTROL_PLANE_STATE_BYTES = 4 * 1024 * 1024; const CHECKPOINT_EXPIRATION_MAX_FAILED_PASSES = 3; export type NativeHarnessBackupExpirationOptions = { + duty?: "backups" | "evidence"; closedForDays?: number; evidenceRetentionDays?: number; dryRun?: boolean; @@ -77,6 +78,7 @@ type EvidenceContext = { retainedBefore: number; canInspect: () => boolean; dueAt: (at: number) => void; + retry: () => void; }; type SessionIdentity = { @@ -105,8 +107,13 @@ function emptyResult(): NativeHarnessBackupExpirationResult { }; } +function missingPath(error: NodeJS.ErrnoException) { + if (error.code === "ENOENT") return null; + throw error; +} + async function isRealDirectory(target: string): Promise { - const stat = await fs.lstat(target).catch(() => null); + const stat = await fs.lstat(target).catch(missingPath); return Boolean(stat?.isDirectory() && !stat.isSymbolicLink()); } @@ -262,6 +269,8 @@ export function nativeHarnessBackupExpirationService( db: Db, defaults: NativeHarnessBackupExpirationOptions = {}, ) { + const includeBackups = defaults.duty !== "evidence"; + const includeEvidence = defaults.duty !== "backups"; let cursor: string | null = null; let resume: { scope: string; archive: string } | null = null; let quarantineCursor: string | null = null; @@ -421,7 +430,10 @@ export function nativeHarnessBackupExpirationService( } async function removeExpiredLeftovers(parent: string, context: EvidenceContext, result: NativeHarnessBackupExpirationResult) { - for (const name of await fs.readdir(parent).catch(() => [] as string[])) { + for (const name of await fs.readdir(parent).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return [] as string[]; + throw error; + })) { if (!name.startsWith(EXPIRED_PREFIX) || !UUID_PATTERN.test(name.slice(EXPIRED_PREFIX.length))) continue; if (!context.canInspect()) { context.dueAt(context.now); @@ -444,17 +456,17 @@ export function nativeHarnessBackupExpirationService( result.skipped[reason] = (result.skipped[reason] ?? 0) + 1; }; const target = path.join(parent, name); - const stat = await fs.lstat(target).catch(() => null); + const stat = await fs.lstat(target).catch(missingPath); if (path.dirname(target) !== parent || !stat?.isDirectory()) { return skip("unsafe_path"); } const createdAt = stampedAt > 0 && stampedAt <= context.now ? stampedAt : stat.mtimeMs; if (createdAt >= context.retainedBefore) { - context.dueAt(createdAt + context.retentionMs); + context.dueAt(createdAt + context.retentionMs + 1); return skip("recent_evidence"); } if (context.isSessionBusy(scopeDigest)) { - context.dueAt(context.now); + context.retry(); return skip("session_active"); } const bytes = await allocatedBytes(target); @@ -462,7 +474,7 @@ export function nativeHarnessBackupExpirationService( result.evidenceCandidateBytes += bytes; if (context.dryRun) return; if (context.isSessionBusy(scopeDigest)) { - context.dueAt(context.now); + context.retry(); return skip("session_active"); } const trash = path.join(parent, `${EXPIRED_PREFIX}${randomUUID()}`); @@ -480,7 +492,7 @@ export function nativeHarnessBackupExpirationService( result: NativeHarnessBackupExpirationResult, ): Promise { const parent = path.join(root, "continuity-breaks"); - const stat = await fs.lstat(parent).catch(() => null); + const stat = await fs.lstat(parent).catch(missingPath); if (!stat) return null; if (!stat.isDirectory()) { result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; @@ -499,7 +511,7 @@ export function nativeHarnessBackupExpirationService( await retireEvidence(parent, name, scopeDigest, Number(match[1]), context, result); } catch (err) { result.failed++; - context.dueAt(context.now); + context.retry(); logger.warn({ err, scope: scopeDigest }, "native runner evidence retention skipped a continuity archive"); } } @@ -513,7 +525,7 @@ export function nativeHarnessBackupExpirationService( result: NativeHarnessBackupExpirationResult, ): Promise { const parent = path.join(base, "quarantine"); - const stat = await fs.lstat(parent).catch(() => null); + const stat = await fs.lstat(parent).catch(missingPath); if (!stat) return true; if (!stat.isDirectory()) { result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; @@ -537,7 +549,7 @@ export function nativeHarnessBackupExpirationService( await retireEvidence(parent, name, match[1]!, Number(match[2]), context, result); } catch (err) { result.failed++; - context.dueAt(context.now); + context.retry(); logger.warn({ err, scope: match[1] }, "native runner evidence retention skipped a quarantine entry"); } } @@ -546,7 +558,7 @@ export function nativeHarnessBackupExpirationService( } function noteEvidence(writtenAt: number) { - evidenceDueAt = Math.min(evidenceDueAt, writtenAt + evidenceRetentionMs()); + evidenceDueAt = Math.min(evidenceDueAt, writtenAt + evidenceRetentionMs() + (defaults.duty === "evidence" ? 1 : 0)); } function evidenceRetentionMs(options: NativeHarnessBackupExpirationOptions = {}) { @@ -572,9 +584,10 @@ export function nativeHarnessBackupExpirationService( const closedBefore = new Date(now - closedForDays * DAY_MS); const result = emptyResult(); const hasBudget = () => shouldContinue() && result.candidates + result.evidenceCandidates < maxRemovals; + if (!passActive && !includeBackups && evidenceDueAt > now) return result; if (!passActive) { passActive = true; - evidencePass = evidenceDueAt <= now; + evidencePass = includeEvidence && evidenceDueAt <= now; if (evidencePass) { evidenceDueAt = Number.POSITIVE_INFINITY; passDueAt = Number.POSITIVE_INFINITY; @@ -590,10 +603,16 @@ export function nativeHarnessBackupExpirationService( dueAt: (at) => { passDueAt = Math.min(passDueAt, at); }, + retry: () => { + passDueAt = Math.min(passDueAt, now + (defaults.duty === "evidence" ? 60 * 60_000 : 0)); + }, }; if (phase === "sessions") { - const names = (await fs.readdir(base).catch(() => [] as string[])) + const names = (await fs.readdir(base).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return [] as string[]; + throw error; + })) .filter((name) => SCOPE_PATTERN.test(name)) .sort(); const pending = cursor ? names.filter((name) => name > cursor!) : names; @@ -608,11 +627,18 @@ export function nativeHarnessBackupExpirationService( const previous = cursor; cursor = name; const root = path.join(base, name); - if (path.dirname(root) !== base || !(await isRealDirectory(root))) { - result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; + try { + if (path.dirname(root) !== base || !(await isRealDirectory(root))) { + result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1; + continue; + } + } catch (err) { + result.failed++; + context.retry(); + logger.warn({ err, scope: name }, "native runner retention could not inspect a session root"); continue; } - if (resumeAfter === null) { + if (resumeAfter === null && includeBackups) { try { await expireSession(root, name, { dryRun, isSessionBusy, closedBefore }, result); } catch (err) { @@ -626,7 +652,7 @@ export function nativeHarnessBackupExpirationService( stoppedAt = await retireContinuityBreaks(root, name, resumeAfter, context, result); } catch (err) { result.failed++; - context.dueAt(now); + context.retry(); logger.warn({ err, scope: name }, "native runner evidence retention skipped a session"); } if (stoppedAt !== null) { @@ -651,7 +677,7 @@ export function nativeHarnessBackupExpirationService( } } catch (err) { result.failed++; - context.dueAt(now); + context.retry(); quarantineCursor = null; logger.warn({ err }, "native runner evidence retention skipped quarantine"); } @@ -662,6 +688,11 @@ export function nativeHarnessBackupExpirationService( return result; } + function nextEvidenceAt() { + if (passActive) return Date.now(); + return Number.isFinite(evidenceDueAt) ? evidenceDueAt : null; + } + async function tickCheckpoints(options: NativeHarnessBackupExpirationOptions = {}) { const closedForDays = Math.max(1, options.closedForDays ?? defaults.closedForDays ?? NATIVE_HARNESS_BACKUP_EXPIRATION_DEFAULT_DAYS); const result = emptyResult(); @@ -714,5 +745,5 @@ export function nativeHarnessBackupExpirationService( }); } - return { tick, noteEvidence, tickCheckpoints, registerCheckpointExpiration }; + return { tick, noteEvidence, nextEvidenceAt, tickCheckpoints, registerCheckpointExpiration }; } diff --git a/server/src/services/native-runtime/native-runner-evidence-retention-work.ts b/server/src/services/native-runtime/native-runner-evidence-retention-work.ts new file mode 100644 index 00000000000..b43dfaa60f7 --- /dev/null +++ b/server/src/services/native-runtime/native-runner-evidence-retention-work.ts @@ -0,0 +1,34 @@ +import { logger } from "../../middleware/logger.js"; +import { registerDeadlineWork } from "../deadline-work.js"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; +import { nativeHarnessBackupExpirationService, type NativeHarnessBackupExpirationOptions } from "./native-harness-backup-expiration.js"; +import { onNativeRunnerEvidence } from "./native-runner-evidence-signal.js"; + +export function registerNativeRunnerEvidenceRetentionWork( + coordinator: ReturnType, + db: Parameters[0], + options: NativeHarnessBackupExpirationOptions, + canRun: () => boolean, +) { + const service = nativeHarnessBackupExpirationService(db, { ...options, duty: "evidence" }); + let continuation: number | null = null; + const worker = registerDeadlineWork(coordinator, DELIVERY_QUEUES.nativeRunnerEvidenceRetention, { + retryMs: 30_000, canRun, + run: async signal => { + try { + const result = await service.tick({ shouldContinue: () => !signal.aborted && canRun() }); + if (result.evidenceCandidates > 0 || result.failed > 0) { + logger.info({ ...result, dryRun: options.dryRun ?? false }, "native runner evidence retention"); + } + continuation = result.truncated ? Date.now() + 1000 : null; + } catch (err) { + continuation = Date.now() + 60 * 60_000; + logger.warn({ err }, "native runner evidence inspection failed"); + } + }, + next: async () => continuation ?? service.nextEvidenceAt(), + }); + const unsubscribe = onNativeRunnerEvidence(at => { service.noteEvidence(at); worker.wake(); }); + return { ...worker, unsubscribe }; +} diff --git a/tests/e2e/disconnect-write-recovery.spec.ts b/tests/e2e/disconnect-write-recovery.spec.ts index e533361e775..8f81ee178dc 100644 --- a/tests/e2e/disconnect-write-recovery.spec.ts +++ b/tests/e2e/disconnect-write-recovery.spec.ts @@ -16,6 +16,7 @@ test("a task title survives an outage and saves after server recovery", async ({ data: { title: "Another task stays unchanged", status: "backlog" }, })); await page.goto(`/${company.issuePrefix}/issues/${issue.identifier}`); + await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 }); const header = page.getByTestId("issue-detail-header"); await expect(header.getByRole("heading", { name: issue.title, exact: true })).toBeVisible(); @@ -60,6 +61,7 @@ test("two cached tasks keep separate title saves while the browser is offline", await expect(page.getByRole("heading", { name: second.title, exact: true })).toBeVisible(); await page.getByRole("link", { name: "Tasks", exact: true }).first().click(); await page.getByRole("link").filter({ hasText: first.title }).first().click(); + await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 }); const header = page.getByTestId("issue-detail-header"); await expect(header.getByRole("heading", { name: first.title, exact: true })).toBeVisible(); const dismissAnnouncement = page.getByRole("button", { name: "Dismiss announcement" }); @@ -74,6 +76,7 @@ test("two cached tasks keep separate title saves while the browser is offline", await secondLink.focus(); await secondLink.press("Enter", { timeout: 5_000 }); await expect(page).toHaveURL(new RegExp(`${second.id}$`)); + await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 }); await expect(header.getByRole("heading", { name: second.title, exact: true })).toBeVisible(); await header.getByRole("heading", { name: second.title, exact: true }).click(); await expect(header.locator("textarea")).toHaveValue(second.title); @@ -189,6 +192,7 @@ test("startup work retries a failed session lookup while other APIs stay healthy await route.fulfill({ status: 500, json: { error: "session_lookup_failed" } }); }); await page.goto(`/${company.issuePrefix}/issues/${issue.id}`); + await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 }); const header = page.getByTestId("issue-detail-header"); await header.getByRole("heading", { name: issue.title, exact: true }).click(); await header.locator("textarea").fill("Saved after session recovery");