fix(server): schedule archived runner evidence from saved deadlines (#17925)

<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/85/changes/paperclip/85
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Retired native sessions leave continuity and quarantine evidence on
disk.
> - The old hourly loop mixes evidence retention with backup expiry,
which has different database eligibility rules.
> - Evidence already has saved creation timestamps and local publication
signals.
> - This pull request gives evidence its own registered worker and
actual next deadline.
> - The worker stops filesystem and SQL activity when its evidence
obligations settle.

## Linked Issues or Issue Description

Stack slice 65. Parent: #17923
(`codex/sleep-64-skill-version-retention`). Related foundation: #15625.

**What happened?**

At parent `7b94aa185c29673372646bbf2e07a2fbd16bc394`, an hourly
backup/evidence tick still inspects native state roots. Evidence's
private due time is checked by that recurring tick instead of the shared
scheduler. Empty evidence and future retention windows have no reason to
be rediscovered every hour.

**Expected behavior**

Reconstruct evidence once. Wake from the existing publication signal.
Register the earliest saved retention deadline. Keep busy, failed, or
incomplete evidence recoverable. Preserve backup eligibility as a
separate duty.

**Steps to reproduce**

Start retention with an empty evidence root or archives inside the
seven-day window. The old timer still visits the combined owner. The new
evidence worker disarms when empty, or registers the saved expiry with
no evidence filesystem or SQL calls between relevant changes and that
deadline.

## What Changed

- Add an evidence worker through the delivery coordinator and shared
scheduler. Track startup reconstruction without delaying HTTP readiness.
Unsubscribe local publication before delivery shutdown.
- Select `backups` for the existing backup timer and `evidence` for the
new worker. Existing callers that omit the selection retain combined
behavior. Backup expiration remains the next separate migration.
- Use hourly retries for concrete busy or failed evidence, with earlier
saved expiries and one-second continuations independent. Restore
candidate/removal/byte summaries. Schedule publication at the first
removable instant; preserve legacy manual-tick semantics.
- Expose the existing earliest evidence deadline. Preserve one-second
bounded continuation, concrete failure/busy retries, cleanup-source
archive exclusions, symlink protection, late activity checks, atomic
rename, and interrupted removal recovery.
- Preserve non-ENOENT inspection errors as recovery obligations. Retry a
failed session root in a new evidence pass instead of continuing past it
and forgetting it.
- Register the first removable instant after the inclusive retention
boundary. Keep the existing manual dry-run publication semantics.
- Add real filesystem/isolated PostgreSQL lifecycle tests and
disabled/held-startup wiring cases. Add no table or migration.

The archive and quarantine writers already call
`noteNativeRunnerEvidence`; this PR connects that existing signal to the
scheduler. Shared code owns scheduling, while evidence rules stay with
the retention service. See
[`doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md`](doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md).

## Verification

- Adopted actual prerequisite squash
`b06ea111a254c62e0a55199b010d82e0adb33bf0` from #17923 through normal
merges. Qualified runtime/package/UI/map trees remain unchanged. The
final nine-path diff includes one browser-spec readiness repair.
- Runtime qualification remains **652 passing TypeScript cases across 22
complete files**, plus **29 passing Rust cases** in native-backend,
runtime-notification and configured-environment targets. One actual
closed Pi snapshot launch is intentionally skipped because its external
distribution root is not provided. Accepted source is
`b80eee32f344ae0218d8b791c14f2ea174da09cc`; all runtime and earlier
sleep-test bytes are preserved.
- The complete outage-recovery browser file passes **all five journeys**
on a fresh isolated database after four additive rendered-history
readiness assertions. Every original assertion, outage control and
timeout remains byte-identical when those new lines are removed. The
intentionally unresolved first-session scenario keeps its pre-auth
interaction and delayed-owner coverage. The prior CI/local timeout
evidence and one rejected broader precondition remain in the audit.
- Adapter/Runner/database builds, server/Runner typechecking, Rust
formatting/workspace checks, server build, module boundaries, migration
order and generated TypeScript/Rust profiles pass. PostgreSQL 17
fixtures apply all 413 journal entries through index 414; source hashes
and temporary fixture imports restore exactly.
- **Inherited validation exception:** Copilot profile 81's separate
source-identity check fails on the merged Rust configured-environment
hash. Its inputs and attestations equal upstream and stay unchanged
here. Grok's identity check passes. No release attestation is
regenerated or new live-provider qualification claimed. Required
exact-head CI/security and Greptile 5/5 still must pass.
- [The evidence
audit](doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md)
retains historical failures, counts and limitations. No full local
repository-suite, live provider, external-alarm, measured suspension or
fleet-savings claim. Fresh final-head CI/security, exact-head Greptile
5/5 and actionable Ponytail findings remain separate merge gates.

## Risks

- Live publication is the existing server-owned executor in one process.
Independent filesystem writers need an explicit signal contract. Startup
reconstructs restart state.
- Busy sessions and concrete inspection/removal failures retain retries.
Unknown cleanup-source archives remain protected.
- The existing scan enumerates directory names before applying its
entry/removal budget. This PR preserves that behavior and does not claim
a bounded total directory-listing allocation.
- An in-process deadline cannot wake stopped compute. Cloud sleep-safety
and external-alarm coverage stay unchanged. Physical evidence
reclamation can wait for a supported host wake.
- Backup expiration still has an hourly database eligibility scan. This
slice removes only the archived-evidence duty from that loop and does
not claim full instance suspension.

## Model Used

OpenAI Codex (GPT-6-based coding agent), using repository inspection,
normal Git merges, composition proofs and complete
PostgreSQL/filesystem/startup suites. No subagents were used in this
reconciliation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
authored and GitHub committed 2026-10-29 11:12:16 -05:00
1 parent 81dcc4f6bf
commit 07c6df260b
9 files changed
+596 -26

No files matched your search

@@ -3525,3 +3525,15 @@ writer inventory, module contracts, local acceptance, and limitations. Remote
review and checks remain pending. The user's October 27 instruction now authorizes
merging qualifying sleep PRs in dependency order after Greptile 5/5 and any
present Ponytail review passes, with all required CI checks passing.
## Slice 65 — archived runner evidence retention (2026-10-27)
Split evidence retention from native-harness backup eligibility in their shared
legacy service. Evidence now reconstructs once, wakes from the existing archive/
quarantine publication signal, and uses saved retention timestamps. Busy or failed
items retain recovery and bounded continuation. The backup timer selects only its
own duty and remains for the next slice. See the
[evidence audit](2026-10-27-runner-evidence-retention-sleep-audit.md) for producer,
clock-boundary, interrupted-cursor, writer-model, and verification evidence.
Local acceptance passes 158 cases plus server typechecking/build. Remote review
and CI are pending; this is not a deployed sleep claim.
@@ -0,0 +1,290 @@
# Runner evidence retention scheduling
Slice 65, parent slice 64 (#17923) at
`d2ccbcdfc9118b8841299da65e94e487807e73ab` before its review repair.
This slice owns archived runner evidence. Native-harness backup expiration is a
separate remaining duty, although the old implementation inspects both in one loop.
The default branch was observed at `50b932a142216331b8d476042b454bc123739c40`.
The stack's pinned integration baseline remains `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`;
the later default-branch changes require final stack integration and verification.
## Obligation and existing persistence
Continuity-break archives and identity quarantine directories preserve evidence
from a retired native session. Their names contain the creation timestamp. The
existing retention service falls back to mtime for a timestamp in the future,
preserves cleanup-source quarantine archives, refuses symlinks, checks session
activity twice before removal, and moves eligible evidence to a unique expired
name before recursive removal. Interrupted expired directories are recovered.
These filesystem facts already persist the retention obligation. There is no
reason to add a database table. No database work is needed for evidence inspection.
The existing hourly tick still invokes the backup-expiration path, whose eligibility
also depends on issues, runs, finalizations, and leases. Moving those two duties
independently keeps each PR's trigger and correctness claim reviewable.
## Triggers and module ownership
- Startup reconstructs evidence once using the existing bounded session/archive/
quarantine continuation state.
- `quarantineRunnerdStateRoot` publishes `noteNativeRunnerEvidence` after the
quarantine rename. `archiveContinuityState` publishes the same signal when it
creates the continuity archive, before it moves the retired state. The saved
retention deadline is days later; the signal does not authorize immediate
deletion or claim that the archive's write is complete.
- A publication adds its timestamp to the existing earliest evidence deadline and
wakes the registered worker. Earlier deadlines continue to win. The next due
inspection reconstructs all retained evidence, including multiple writes.
- A recent item contributes its saved creation time plus the configured retention
window and one millisecond, because the existing protection comparison is
inclusive at the window boundary.
- Busy sessions, incomplete scans, and failed inspection/removal retain actual
recovery obligations. Bounded continuations run after one second. Concrete
failed or busy evidence retries use a one-hour worker retry; an earlier saved
expiry or new publication can still wake the worker sooner.
- Empty evidence, ignored cleanup-source archives, and fully processed history
leave no evidence deadline or recurring filesystem/SQL calls.
The existing retention owner exposes `nextEvidenceAt()` and a per-instance duty
selection. Legacy callers that omit the selection keep combined behavior. Startup
selects `backups` for the existing backup timer and registers an `evidence` instance
with the delivery coordinator. The shared scheduler/coordinator owns overlap,
admission, drain, retries, deadlines, and shutdown. No feature logic moves into
`index.ts`. The worker's first promise stays tracked without delaying readiness;
shutdown unsubscribes the local signal before stopping/joining delivery work.
The backup interval remains deliberately visible for the next independent slice.
This PR removes evidence from that interval; it does not claim that the instance
as a whole is idle while backup and other recurring duties remain.
Supported publication is the existing server-owned session executor in one
process. Unrelated filesystem writers need an explicit signal/ingress contract.
Startup reconstructs restart state; it is not cross-process live notification.
Cloud sleep gates and external alarm coverage are unchanged. Expiration can wait
for a supported host wake when compute is stopped.
## Failure record and local evidence
Two regressions failed before repair: the saved deadline was one millisecond
before the first removable instant, and an EACCES reading an archive directory
was treated as an empty directory and disarmed retention. Missing paths remain
quiet; other inspection failures now retain recovery.
A further controlled EACCES at the session root reproduced a lost cursor: retry
continued after the failed scope and left its evidence behind. The owner now
records that concrete failed inspection for a new full evidence pass. Healthy
scopes continue, and the failed scope is revisited after the bounded retry.
An intermediate full run exposed a compatibility regression in the existing
manual dry-run notification case. The publication timestamp's original due
semantics were restored; the inclusive boundary adjustment applies to the
observed recent item's next inspection. The old test and deletion protections
were preserved rather than weakened.
Final local acceptance: 158 cases pass across native-harness/evidence retention,
retention configuration, and startup. The retention suite uses a guarded isolated
PostgreSQL database; all selected cases execute. Tests measure quiet filesystem
and SQL behavior, saved deadlines, local publication, restart, busy/failure retry,
first work during drain, one-entry continuation, absence of backup SQL in the
evidence worker, disabled registration, and tracked startup after HTTP readiness.
Existing backup, manifest, lease, issue, dry-run, symlink, and cleanup-archive
cases still pass. Server TypeScript checking and the full server build also pass.
Remote review/CI and whole-stack acceptance remain pending.
## Ponytail repair: quiet handled retries, summaries, and publication boundary
Controlled tests reproduced a busy archive causing whole-tree inspection every
thirty seconds, that overdue retry masking another archive's ten-second saved
expiry, missing deletion/byte summaries, and a publication deadline one millisecond
before the first removable instant. All four fail before this repair.
The evidence-only owner now records an hourly retry for handled busy/inspection/
removal failures. Genuine earlier saved expiries remain the minimum deadline.
One-second truncated continuations remain independent. Unexpected state-base
inspection failure is also retained as an hourly worker retry; a new publication
can reactivate it immediately. Admission/drain reconciliation still uses the
coordinator's existing short memory-only retry. Legacy combined/manual callers
retain their original explicit-tick behavior.
The worker again logs candidate, removal, reclaimed-byte, failure, and dry-run
summary fields when it processes candidates or failures. Publication now schedules
the first removable instant in evidence-only mode; the old manual notification
case is unchanged. An added base-directory fault test proves five minutes of quiet
before a publication resumes processing. Final owning acceptance passes 162 cases
across retention, configuration, and startup, plus server typechecking. Earlier
receipts remain tied to their recorded heads; integrated verification follows the
parent skill-retention repair.
## October 29 master reconciliation and current acceptance
This existing archived-evidence slice was reconciled with the qualified retention
parent through normal merges. The expiration service now exposes both the new
saved evidence deadline and the parent's separate startup checkpoint methods.
The finite checkpoint failure budget remains intact. The native retention fixture
keeps its explicit external-database override and migrations while using the
parent's fresh recovery-database helper by default. Identical logger/coordinator
imports introduced by automatic composition were deduplicated. Every other path
matches the automatic merge, and packages remain byte-identical to the parent.
Current local acceptance is **285 passing unique cases across 11 complete files**,
without skips. The original combined run passed 283 cases but failed two new
startup fixtures because their mocked service omitted the preserved
`registerCheckpointExpiration` method. Production startup correctly called that
method. The fixture repair supplies the actual API and adds explicit assertions
that enabled and disabled evidence configurations retain checkpoint registration,
including while evidence reconstruction is held after HTTP readiness. The full
100-case startup file then passed in 7.55 seconds, without weakening any prior
assertion or timeout. The other ten complete files passed 185 cases and their
source remained unchanged. Accepted source after that fixture-only repair is
`da44dea2f3cd86cc72a25f2a419c63acafda1f7a`.
Coverage includes real archived-evidence expiry, filesystem and path safety,
inclusive retention boundaries, retry and paged continuation, preserved finite
checkpoint startup work, backup cleanup/configuration, server wiring, shared
deadline/coordinator and idle-work behavior, plus retained skill-cache behavior
and its own-company expiry regression. Database build, production server
typechecking/build, module boundaries, target migration order and Runner profile
identity passed before the fixture-only correction; runtime source did not change
afterward. Two fresh PostgreSQL 17 schemas plus per-file fixtures applied all 412
journal entries through index 413. All temporary local fixture imports were
restored exactly, and every selected test hash is bound in the acceptance record.
The archive worker uses the evidence duty and performs no backup SQL. The backup
duty and startup checkpoint owner remain distinct. Saved publication/retention
boundaries and one-hour busy/failure retries retain nearer expiries and bounded
one-second continuation pages. Unsubscription still precedes coordinator stop.
Publication must adopt the actual merged skill-retention prerequisite and obtain
fresh exact-head CI/security and reviews. No full local repository-suite, live
provider, external wake, measured instance/database suspension or fleet-cost claim
is made. Historical results and limitations above remain tied to their revisions.
## Requalification with later upstream changes
Normal reconciliation preserved the final skill-retention repair and all 95
unrelated master paths from the actual timer squash, task activity index and task
read changes. On source `2fa458833fa43b47eddc728a5e57f5ffcf5fa3e4`, all **285
cases across 11 complete files passed together**, with no skips, in 47.26 seconds.
This fresh full selection includes the preserved checkpoint APIs and their
repaired startup fixtures. Database build, production server typechecking/build,
module boundaries, target migration order and generated TypeScript/Rust release
profile consistency passed. Fresh PostgreSQL 17 schemas applied all 413 journal
entries through index 414. Every selected source hash and temporary fixture import
restored exactly. The final parent audit was then adopted through a doc-only
normal merge without changing any accepted runtime/test/package tree.
The inherited upstream Copilot source-identity failure is recorded in the
[parent retention audit](2026-10-27-skill-version-retention-sleep-audit.md#final-current-master-qualification-before-publication).
This slice leaves its release attestation and all identity inputs unchanged.
Generated admission still matches the active release manifest; the separate
source-identity computation is an explicit inherited exception, not a pass or a
claim of new live-provider qualification. Required fresh exact-head CI/security
and reviews remain merge gates. Actual parent squash adoption is still required
before publication; any new upstream behavior must remain preserved and receive
appropriate combined-context checks.
## Workspace-seed upstream composition
The next upstream merge (`27fbdb2a74e13141065555b6a2fa5acdd8da7964`)
optimizes verified workspace seed reuse. It is preserved by a normal merge,
including its adapter implementation, tests and native workspace history. On
combined source `a288a58287047193774eda54601ac9eda732d2e8`, **321 cases
across 14 complete files passed together**, without skips, in 95.73 seconds.
This includes the previous complete 11-file retention selection and the full
workspace-seed reuse, GitHub launcher and native workspace-history files. These
results replace, rather than add to, the earlier 285-case acceptance total for
this combined source. Adapter/database builds, server typechecking/build, module
boundaries, target migration order and generated release profiles pass. Every
selected source hash and temporary fixture import restores exactly. Provider
identity inputs remain unchanged; the inherited exception above still applies.
Actual skill-retention squash adoption remains a publication prerequisite. If
its tree carries only these already accepted upstream changes, tree identity
will prove that adoption changes no accepted runtime. Any additional behavior
will receive scoped combined-context validation before publication.
## Runner-authority upstream qualification
Master `27be8590442fc02d29a6c257a4e9dbe5bbbd66d2` then introduced Runner
authority for runtime activity and updated provider manifests. Normal composition
through `b80eee32f344ae0218d8b791c14f2ea174da09cc` preserves its complete
package tree byte for byte, the accepted workspace changes and the parent browser
readiness repair. The server retention implementation is unchanged.
On that combined source, **652 TypeScript cases across 22 complete files passed**.
These include the complete 14-file evidence-retention/workspace selection (321
cases) and eight affected Runner files (331 cases). One actual closed Pi snapshot
launch is intentionally skipped because its external distribution root is not
provided; this is not a live-distribution qualification. **29 Rust cases passed**
in the full native-backend target (26), full runtime-notification target (one),
and both configured-environment unit cases. Adapter/Runner/database builds, Runner
and server typechecking, Rust formatting/workspace checking, server build, module
boundaries, migration order and generated TypeScript/Rust profiles all passed.
Fresh PostgreSQL 17 schemas apply all 413 entries through index 414. All selected
source hashes and temporary imports restore exactly. These fresh counts replace
rather than add to the prior selections.
The new upstream manifest uses Copilot profile 81, but its separate source-identity
check still fails: the active fixture's `configuredEnvironmentRustSourceSha256`
differs from the actual merged Rust file. Every bound source and the entire
package tree equal master. This preserves the earlier inherited OpenCode-key
identity drift through the upstream profile update; no retention edit causes it.
Grok's separate identity check passes. The Copilot exception remains explicit,
and no release attestation is regenerated or new live-provider qualification
claimed. Required fresh exact-head CI/security and reviews remain merge gates.
Actual parent squash adoption remains necessary before final publication. The
new parent browser-repair commit and its original assertions are carried exactly;
the browser gate itself must still be green on the final published head.
## Actual prerequisite adoption for final publication
The skill-retention prerequisite [#17923](https://github.com/paperclipai/paperclip/pull/17923)
merged at `b06ea111a254c62e0a55199b010d82e0adb33bf0`, after all 58 current
checks were clean and the exact reviewed head `269935f6ccd7685866fd94bea66f7f803c48058d`
had Greptile and Ponytail 5/5. Its separate, evidence-supported failed-job retry
passed; no failing gate or required review finding was waived.
This last slice adopts that actual squash through normal merge
`3c40eaad954f7705144e6f378d93be2adff0c54a`. Duplicate ancestry conflicts were
resolved to the proven composed tree. The complete pre-adoption held tree is
unchanged, including the 652-case TypeScript and 29-case Rust accepted sources,
all selected tests and the parent's browser-readiness repair. Package, UI, E2E
and map trees equal the actual prerequisite. The final diff has eight owned
paths. This adoption changes ancestry rather than runtime behavior, so accepted
checks are not repeated; this paragraph and the PR body only record provenance.
The original remote head remains an ancestor, and no history is force-pushed.
Fresh required CI/security, exact-head Greptile 5/5 and any actionable Ponytail
findings are still gates on this slice's final publication. The inherited source
identity exception and all bounded-acceptance limitations above remain explicit.
## Final outage-journey readiness repair
The first final publication, `762d48605f93351e47f9fd1249d9293625834242`,
received exact-head Greptile 5/5 and Ponytail 5/5 with no actionable findings.
Its required checks completed cleanly except the second browser shard and its
aggregate. The title-outage journey reached the 60-second global test timeout;
its failure artifact still shows the original title, and no issue write was
observed. The same file then passed that case locally but reproduced a timeout
in the failed-session startup case (four passing cases and one failure). These
failures and the original assertions remain recorded; no failing gate is waived.
A visible initial task heading alone does not establish that the conversation
layout has settled after the upstream loading changes. The test now requires
visible conversation history before the ordinary title-edit flows, including
the cached offline task switch. Four additive readiness assertions are the only
spec changes. Removing those four lines reproduces the exact prior file bytes:
all original recovery assertions, outage controls, and timeout budgets remain.
The deliberately held first-session-lookup case retains its original pre-auth
readiness, so it still queues a write before the owner is known. An initial
broader readiness attempt blocked that intentional startup state; that extra
condition was removed rather than changing the scenario.
All **five complete outage-recovery browser journeys pass** afterward on a new
isolated PostgreSQL 17 database. They still verify failed-save retention, actual
retry after recovery, separation of two offline drafts, delayed owner binding,
comment replay idempotency and failed-session retry. The accepted 652-case
TypeScript and 29-case Rust runtime sources, all prior sleep regression files,
packages and UI are unchanged. The final nine-path diff adds this spec-readiness
repair to the eight-path feature. Fresh exact-head CI/security and reviews are
required after publication; older-head review passes are not reused.
@@ -1,3 +1,8 @@
import { applyPendingMigrations } from "@paperclipai/db";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { registerNativeRunnerEvidenceRetentionWork } from "../services/native-runtime/native-runner-evidence-retention-work.js";
import { noteNativeRunnerEvidence } from "../services/native-runtime/native-runner-evidence-signal.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
import { createHash, randomUUID } from "node:crypto";
import os from "node:os";
import path from "node:path";
@@ -20,9 +25,10 @@ import { nativeHarnessBackupExpirationService } from "../services/native-runtime
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { logger } from "../middleware/logger.js";
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_RETENTION_TEST_DATABASE_URL");
const describeEmbeddedPostgres = external || embeddedPostgresSupport.supported ? describe : describe.skip;
if (!embeddedPostgresSupport.supported) {
if (!external && !embeddedPostgresSupport.supported) {
console.warn(
`Skipping embedded Postgres native harness backup expiration tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
);
@@ -41,11 +47,17 @@ describeEmbeddedPostgres("native harness backup expiration", () => {
const now = new Date();
beforeAll(async () => {
tempDb = await startRecoveryPostgres("paperclip-native-backup-expiration-");
db = createDb(tempDb.connectionString);
if (!external) tempDb = await startRecoveryPostgres("paperclip-native-backup-expiration-");
if (external) await applyPendingMigrations(external);
db = createDb(external ?? tempDb!.connectionString);
}, 20_000);
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
let worker: ReturnType<typeof registerNativeRunnerEvidenceRetentionWork> | undefined;
let admitted = true;
afterEach(async () => {
worker?.unsubscribe(); await coordinator?.stop(); coordinator = undefined; worker = undefined; admitted = true;
vi.useRealTimers(); vi.restoreAllMocks();
await db.delete(agentTaskSessions);
await db.delete(nativeRunFinalizations);
await db.delete(environmentLeases);
@@ -58,6 +70,7 @@ describeEmbeddedPostgres("native harness backup expiration", () => {
});
afterAll(async () => {
await db?.$client.end({ timeout: 1 });
await tempDb?.cleanup();
});
@@ -522,6 +535,146 @@ describeEmbeddedPostgres("native harness backup expiration", () => {
return createHash("sha256").update(randomUUID()).digest("hex");
}
async function startEvidence(options: Parameters<typeof nativeHarnessBackupExpirationService>[1] = {}) {
if (!vi.isFakeTimers()) {
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
vi.setSystemTime(now);
}
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => true,
onError: error => { throw error; } });
worker = registerNativeRunnerEvidenceRetentionWork(coordinator, db, { stateBase, isSessionBusy: () => false, ...options }, () => admitted);
await worker.ready;
}
async function settled(deadline: number | null = null) {
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(1000);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBe(deadline);
}, { timeout: 5000 });
}
async function quiet(ms = 3 * 60 * 60_000) {
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"), directories = vi.spyOn(fs, "readdir");
await vi.advanceTimersByTimeAsync(ms);
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); expect(directories).not.toHaveBeenCalled();
reads.mockRestore(); writes.mockRestore(); directories.mockRestore();
}
it("leaves empty evidence and retained backups quiet without invoking backup SQL", async () => {
const session = await seedSession();
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction");
await startEvidence(); await settled(); expect(await exists(session.current)).toBe(true);
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled();
reads.mockRestore(); writes.mockRestore(); await quiet();
});
it("reconstructs and replaces the saved evidence deadline after a publication signal", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const first = await seedArchive(session.root, { stampedDaysAgo: 2 });
const firstDeadline = now.getTime() + 5 * DAY_MS + 1;
await startEvidence(); await settled(firstDeadline); await quiet();
const writtenAt = Date.now();
const next = await seedArchive(session.root, { name: `${writtenAt}-${randomUUID()}` });
noteNativeRunnerEvidence(writtenAt);
await settled(firstDeadline);
await vi.advanceTimersByTimeAsync(firstDeadline - Date.now());
await settled(writtenAt + 7 * DAY_MS + 1); expect(await exists(first)).toBe(false);
await vi.advanceTimersByTimeAsync(writtenAt + 7 * DAY_MS + 1 - Date.now()); await settled();
expect(await exists(next)).toBe(false); expect(await exists(session.current)).toBe(true); await quiet();
});
it.each(["session root", "archive directory"])("preserves an evidence inspection failure at %s until its concrete retry succeeds", async location => {
const session = await seedSession({ issueStatus: "in_progress" });
const archive = await seedArchive(session.root);
const original = fs.lstat;
const denied = vi.spyOn(fs, "lstat").mockImplementation(async (...args) => {
if (String(args[0]) === (location === "session root" ? session.root : path.join(session.root, "continuity-breaks"))) throw Object.assign(new Error("unavailable"), { code: "EACCES" });
return original(...args);
});
await startEvidence(); expect(coordinator!.nextWakeAt()).not.toBeNull();
const retryAt = coordinator!.nextWakeAt()!; expect(await exists(archive)).toBe(true);
denied.mockRestore(); await quiet(5000);
await vi.advanceTimersByTimeAsync(retryAt - Date.now()); await settled();
expect(await exists(archive)).toBe(false); await quiet();
});
it("retries a busy evidence obligation and disarms after the session releases it", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const archive = await seedArchive(session.root); let busy = true;
await startEvidence({ isSessionBusy: () => busy });
const retryAt = coordinator!.nextWakeAt()!; expect(retryAt - Date.now()).toBeGreaterThanOrEqual(60 * 60_000);
await quiet(5000); busy = false;
await vi.advanceTimersByTimeAsync(retryAt - Date.now()); await settled();
expect(await exists(archive)).toBe(false); await quiet();
});
it("retains the first publication during drain and cleans up on resume", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
await startEvidence(); await settled(); admitted = false;
const archive = await seedArchive(session.root);
noteNativeRunnerEvidence(now.getTime() - 10 * DAY_MS);
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(1000); expect(idleWorkSnapshot().active).toBe(0); });
await quiet(60_000); expect(await exists(archive)).toBe(true);
admitted = true; await settled(); expect(await exists(archive)).toBe(false); await quiet();
});
it("reconstructs the same retained deadline after restart", async () => {
const scope = await emptyStateBase(); await seedQuarantine(scope, { stampedDaysAgo: 6 });
const deadline = now.getTime() + DAY_MS + 1;
await startEvidence(); await settled(deadline);
worker!.unsubscribe(); await coordinator!.stop();
await startEvidence(); await settled(deadline); await quiet(60_000);
expect(coordinator!.nextWakeAt()).toBe(deadline);
});
it("backs off a failed state-base inspection and lets publication reactivate it", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const archive = await seedArchive(session.root), read = fs.readdir;
const denied = vi.spyOn(fs, "readdir").mockImplementation(async (...args) => {
if (args[0] === stateBase) throw Object.assign(new Error("state base unavailable"), { code: "EACCES" });
return read(...args);
});
await startEvidence(); expect(coordinator!.nextWakeAt()! - Date.now()).toBeGreaterThanOrEqual(60 * 60_000);
denied.mockRestore(); await quiet(5 * 60_000); expect(await exists(archive)).toBe(true);
noteNativeRunnerEvidence(now.getTime() - 10 * DAY_MS);
await settled(); expect(await exists(archive)).toBe(false); await quiet();
});
it("lets healthy saved expiry preempt a long-busy evidence retry", async () => {
const busy = await seedSession({ issueStatus: "in_progress" }), healthy = await seedSession({ issueStatus: "in_progress" });
const held = await seedArchive(busy.root);
const dueAt = now.getTime() + 10_001;
const available = await seedArchive(healthy.root, { name: `${dueAt - 7 * DAY_MS - 1}-${randomUUID()}` });
await startEvidence({ isSessionBusy: scope => scope === busy.scope }); await settled(dueAt);
await vi.advanceTimersByTimeAsync(dueAt - Date.now());
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(100); expect(await exists(available)).toBe(false); });
expect(await exists(held)).toBe(true); expect(coordinator!.nextWakeAt()! - Date.now()).toBeGreaterThan(59 * 60_000);
await quiet(60_000);
});
it("reports reclaimed evidence bytes and dry-run mode", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const archive = await seedArchive(session.root); const reported = vi.spyOn(logger, "info");
await startEvidence(); await settled();
expect(reported.mock.calls).toEqual(expect.arrayContaining([[expect.objectContaining({ evidenceCandidates: 1,
evidenceRemoved: 1, evidenceReclaimedBytes: expect.any(Number), dryRun: false }), "native runner evidence retention"]]));
expect(await exists(archive)).toBe(false); await quiet();
});
it("schedules publication at the first removable instant after its retention boundary", async () => {
const scope = await emptyStateBase(); await startEvidence(); await settled(); await quiet();
const writtenAt = Date.now();
await seedQuarantine(scope, { name: `${scope}.identity_indeterminate.${writtenAt}.4242.${randomUUID()}` });
noteNativeRunnerEvidence(writtenAt); await settled(writtenAt + 7 * DAY_MS + 1);
});
it("uses bounded continuations without recounting an evidence session forever", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const archives = [await seedArchive(session.root), await seedArchive(session.root)];
const quarantine = await seedQuarantine(session.scope);
await startEvidence({ maxEntries: 1 }); await settled();
for (const archive of [...archives, quarantine]) expect(await exists(archive)).toBe(false);
await quiet();
});
it("removes continuity archives older than the retention window and keeps recent ones", async () => {
const session = await seedSession({ issueStatus: "in_progress" });
const old = await seedArchive(session.root, { stampedDaysAgo: 9 });
@@ -1,3 +1,4 @@
import { nativeHarnessBackupExpirationService } from "../services/native-runtime/native-harness-backup-expiration.js";
import { skillVersionRetentionService } from "../services/skill-version-retention.js";
import { approvalQueueSeedingService } from "../services/approval-queue-seeding-work.js";
import { joinRequestRetentionService } from "../services/join-request-retention.js";
@@ -539,6 +540,18 @@ vi.mock("../board-claim.js", () => ({
initializeBoardClaimChallenge: vi.fn(async () => undefined),
}));
vi.mock("../services/native-runtime/native-harness-backup-expiration.js", () => ({
NATIVE_HARNESS_BACKUP_EXPIRATION_INTERVAL_MS: 60 * 60_000,
NATIVE_HARNESS_BACKUP_EXPIRATION_STARTUP_DELAY_MS: 2 * 60_000,
NATIVE_HARNESS_BACKUP_EXPIRATION_DEFAULT_DAYS: 7,
NATIVE_RUNNER_EVIDENCE_RETENTION_DEFAULT_DAYS: 7,
nativeHarnessBackupExpirationService: vi.fn(() => ({
tick: vi.fn(async () => ({ checked: 0, candidates: 0, evidenceCandidates: 0, failed: 0, truncated: false })),
noteEvidence: vi.fn(), nextEvidenceAt: vi.fn(() => null),
tickCheckpoints: vi.fn(), registerCheckpointExpiration: vi.fn(),
})),
}));
vi.mock("../services/skill-version-retention.js", () => ({
skillVersionRetentionService: vi.fn(() => ({
listCompanies: vi.fn(async () => []), pruneCompany: vi.fn(async () => ({ checked: 0, truncated: false, nextRunAt: null })),
@@ -582,6 +595,31 @@ describe("startServer feedback export wiring", () => {
process.env.BETTER_AUTH_SECRET = "test-secret";
});
it.each([false, true])("registers runner evidence independently from the backup scan (%s)", async enabled => {
loadConfigMock.mockReturnValue(buildTestConfig({ nativeHarnessBackupExpirationEnabled: enabled }));
await startServer(); await settleStartupHousekeeping();
expect(deliveryWorkRegisterMock.mock.calls.filter(([queue]) => queue === DELIVERY_QUEUES.nativeRunnerEvidenceRetention)).toHaveLength(enabled ? 1 : 0);
expect(vi.mocked(nativeHarnessBackupExpirationService).mock.calls.map(([, options]) => options?.duty)).toEqual(enabled ? ["backups", "evidence"] : ["backups"]);
const backups = vi.mocked(nativeHarnessBackupExpirationService).mock.results[0]!.value;
expect(backups.registerCheckpointExpiration).toHaveBeenCalledTimes(enabled ? 1 : 0);
});
it("tracks held evidence reconstruction after HTTP readiness", async () => {
loadConfigMock.mockReturnValue(buildTestConfig({ nativeHarnessBackupExpirationEnabled: true }));
const pending = Promise.withResolvers<{ truncated: boolean }>();
vi.mocked(nativeHarnessBackupExpirationService).mockReturnValueOnce({ tick: vi.fn(), noteEvidence: vi.fn(), nextEvidenceAt: () => null,
tickCheckpoints: vi.fn(), registerCheckpointExpiration: vi.fn() } as never);
vi.mocked(nativeHarnessBackupExpirationService).mockReturnValueOnce({ tick: vi.fn(() => pending.promise), noteEvidence: vi.fn(), nextEvidenceAt: () => null,
tickCheckpoints: vi.fn(), registerCheckpointExpiration: vi.fn() } as never);
try {
await startServer(); await settleStartupHousekeeping();
expect(getStartupRecoveryState().phase).toBe("ready"); expect(idleWorkSnapshot().active).toBe(1);
expect(vi.mocked(nativeHarnessBackupExpirationService).mock.results[0]!.value.registerCheckpointExpiration).toHaveBeenCalledOnce();
} finally { pending.resolve({ truncated: false }); await settleStartupHousekeeping(); }
expect(idleWorkSnapshot().active).toBe(0);
expect(deliveryWorkErrors.filter(error => error.queue === DELIVERY_QUEUES.nativeRunnerEvidenceRetention)).toEqual([]);
});
it.each([false, true])("registers skill cache retention only when enabled (%s)", async enabled => {
loadConfigMock.mockReturnValue(buildTestConfig({ skillVersionRetentionEnabled: enabled }));
await startServer(); await settleStartupHousekeeping();
+10 -3
View File
@@ -32,7 +32,7 @@ import {
NATIVE_HARNESS_BACKUP_EXPIRATION_STARTUP_DELAY_MS,
nativeHarnessBackupExpirationService,
} from "./services/native-runtime/native-harness-backup-expiration.js";
import { onNativeRunnerEvidence } from "./services/native-runtime/native-runner-evidence-signal.js";
import { registerNativeRunnerEvidenceRetentionWork } from "./services/native-runtime/native-runner-evidence-retention-work.js";
import { gitRestorePointRetentionService, registerGitRestorePointWork } from "./services/git-restore-point-ledger.js";
import {
nativeWorkspaceSyncRetentionService,
@@ -1259,12 +1259,12 @@ async function startServerWithDatabaseTeardown(
};
const skillSourceSync = skillSourceSyncService(db as any);
const nativeHarnessBackupExpiration = nativeHarnessBackupExpirationService(db as any, {
duty: "backups",
dryRun: config.nativeHarnessBackupExpirationDryRun,
closedForDays: config.nativeHarnessBackupExpirationDays,
evidenceRetentionDays: config.nativeRunnerEvidenceRetentionDays,
shouldContinue: () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby(),
});
const unsubscribeNativeRunnerEvidence = onNativeRunnerEvidence(nativeHarnessBackupExpiration.noteEvidence);
let nativeHarnessBackupExpirationInFlight = false;
let nativeHarnessBackupExpirationContinuation: ReturnType<typeof setTimeout> | undefined;
const sweepNativeHarnessBackups = () => {
@@ -1344,6 +1344,13 @@ async function startServerWithDatabaseTeardown(
server.on("close", () => { void taskDrainExpiryWork.stop(); });
const executionSchedule = executionControlSchedule(db);
const workspaceRecoveryRuntime = environmentRuntimeService(db, { pluginWorkerManager });
const nativeRunnerEvidenceRetentionWorker = config.nativeHarnessBackupExpirationEnabled
? registerNativeRunnerEvidenceRetentionWork(deliveryWork, db as any, {
dryRun: config.nativeHarnessBackupExpirationDryRun,
evidenceRetentionDays: config.nativeRunnerEvidenceRetentionDays,
}, () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby())
: null;
if (nativeRunnerEvidenceRetentionWorker) trackHeartbeatSchedulerWork(nativeRunnerEvidenceRetentionWorker.ready);
const skillVersionRetentionWorker = config.skillVersionRetentionEnabled
? registerSkillVersionRetentionWork(deliveryWork, skillVersionRetentionService(db as any, {
dryRun: config.skillVersionRetentionDryRun,
@@ -2191,7 +2198,7 @@ async function startServerWithDatabaseTeardown(
clearInterval(nativeHarnessBackupExpirationInterval);
clearTimeout(nativeHarnessBackupExpirationStartup);
clearTimeout(nativeHarnessBackupExpirationContinuation);
unsubscribeNativeRunnerEvidence();
nativeRunnerEvidenceRetentionWorker?.unsubscribe();
await skillSourceSync.stop();
unsubscribeChatCompletions();
unsubscribeNativeResponsePresentation?.();
@@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
/** Topics for existing durable queues; register intent before their writes. */
export const DELIVERY_QUEUES = {
nativeRunnerEvidenceRetention: "native-runner-evidence-retention",
skillVersionRetention: "skill-version-retention",
databaseBackups: "database-backups",
importSpoolCleanup: "import-spool-cleanup",
@@ -40,6 +40,7 @@ const MAX_CONTROL_PLANE_STATE_BYTES = 4 * 1024 * 1024;
const CHECKPOINT_EXPIRATION_MAX_FAILED_PASSES = 3;
export type NativeHarnessBackupExpirationOptions = {
duty?: "backups" | "evidence";
closedForDays?: number;
evidenceRetentionDays?: number;
dryRun?: boolean;
@@ -77,6 +78,7 @@ type EvidenceContext = {
retainedBefore: number;
canInspect: () => boolean;
dueAt: (at: number) => void;
retry: () => void;
};
type SessionIdentity = {
@@ -105,8 +107,13 @@ function emptyResult(): NativeHarnessBackupExpirationResult {
};
}
function missingPath(error: NodeJS.ErrnoException) {
if (error.code === "ENOENT") return null;
throw error;
}
async function isRealDirectory(target: string): Promise<boolean> {
const stat = await fs.lstat(target).catch(() => null);
const stat = await fs.lstat(target).catch(missingPath);
return Boolean(stat?.isDirectory() && !stat.isSymbolicLink());
}
@@ -262,6 +269,8 @@ export function nativeHarnessBackupExpirationService(
db: Db,
defaults: NativeHarnessBackupExpirationOptions = {},
) {
const includeBackups = defaults.duty !== "evidence";
const includeEvidence = defaults.duty !== "backups";
let cursor: string | null = null;
let resume: { scope: string; archive: string } | null = null;
let quarantineCursor: string | null = null;
@@ -421,7 +430,10 @@ export function nativeHarnessBackupExpirationService(
}
async function removeExpiredLeftovers(parent: string, context: EvidenceContext, result: NativeHarnessBackupExpirationResult) {
for (const name of await fs.readdir(parent).catch(() => [] as string[])) {
for (const name of await fs.readdir(parent).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return [] as string[];
throw error;
})) {
if (!name.startsWith(EXPIRED_PREFIX) || !UUID_PATTERN.test(name.slice(EXPIRED_PREFIX.length))) continue;
if (!context.canInspect()) {
context.dueAt(context.now);
@@ -444,17 +456,17 @@ export function nativeHarnessBackupExpirationService(
result.skipped[reason] = (result.skipped[reason] ?? 0) + 1;
};
const target = path.join(parent, name);
const stat = await fs.lstat(target).catch(() => null);
const stat = await fs.lstat(target).catch(missingPath);
if (path.dirname(target) !== parent || !stat?.isDirectory()) {
return skip("unsafe_path");
}
const createdAt = stampedAt > 0 && stampedAt <= context.now ? stampedAt : stat.mtimeMs;
if (createdAt >= context.retainedBefore) {
context.dueAt(createdAt + context.retentionMs);
context.dueAt(createdAt + context.retentionMs + 1);
return skip("recent_evidence");
}
if (context.isSessionBusy(scopeDigest)) {
context.dueAt(context.now);
context.retry();
return skip("session_active");
}
const bytes = await allocatedBytes(target);
@@ -462,7 +474,7 @@ export function nativeHarnessBackupExpirationService(
result.evidenceCandidateBytes += bytes;
if (context.dryRun) return;
if (context.isSessionBusy(scopeDigest)) {
context.dueAt(context.now);
context.retry();
return skip("session_active");
}
const trash = path.join(parent, `${EXPIRED_PREFIX}${randomUUID()}`);
@@ -480,7 +492,7 @@ export function nativeHarnessBackupExpirationService(
result: NativeHarnessBackupExpirationResult,
): Promise<string | null> {
const parent = path.join(root, "continuity-breaks");
const stat = await fs.lstat(parent).catch(() => null);
const stat = await fs.lstat(parent).catch(missingPath);
if (!stat) return null;
if (!stat.isDirectory()) {
result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1;
@@ -499,7 +511,7 @@ export function nativeHarnessBackupExpirationService(
await retireEvidence(parent, name, scopeDigest, Number(match[1]), context, result);
} catch (err) {
result.failed++;
context.dueAt(context.now);
context.retry();
logger.warn({ err, scope: scopeDigest }, "native runner evidence retention skipped a continuity archive");
}
}
@@ -513,7 +525,7 @@ export function nativeHarnessBackupExpirationService(
result: NativeHarnessBackupExpirationResult,
): Promise<boolean> {
const parent = path.join(base, "quarantine");
const stat = await fs.lstat(parent).catch(() => null);
const stat = await fs.lstat(parent).catch(missingPath);
if (!stat) return true;
if (!stat.isDirectory()) {
result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1;
@@ -537,7 +549,7 @@ export function nativeHarnessBackupExpirationService(
await retireEvidence(parent, name, match[1]!, Number(match[2]), context, result);
} catch (err) {
result.failed++;
context.dueAt(context.now);
context.retry();
logger.warn({ err, scope: match[1] }, "native runner evidence retention skipped a quarantine entry");
}
}
@@ -546,7 +558,7 @@ export function nativeHarnessBackupExpirationService(
}
function noteEvidence(writtenAt: number) {
evidenceDueAt = Math.min(evidenceDueAt, writtenAt + evidenceRetentionMs());
evidenceDueAt = Math.min(evidenceDueAt, writtenAt + evidenceRetentionMs() + (defaults.duty === "evidence" ? 1 : 0));
}
function evidenceRetentionMs(options: NativeHarnessBackupExpirationOptions = {}) {
@@ -572,9 +584,10 @@ export function nativeHarnessBackupExpirationService(
const closedBefore = new Date(now - closedForDays * DAY_MS);
const result = emptyResult();
const hasBudget = () => shouldContinue() && result.candidates + result.evidenceCandidates < maxRemovals;
if (!passActive && !includeBackups && evidenceDueAt > now) return result;
if (!passActive) {
passActive = true;
evidencePass = evidenceDueAt <= now;
evidencePass = includeEvidence && evidenceDueAt <= now;
if (evidencePass) {
evidenceDueAt = Number.POSITIVE_INFINITY;
passDueAt = Number.POSITIVE_INFINITY;
@@ -590,10 +603,16 @@ export function nativeHarnessBackupExpirationService(
dueAt: (at) => {
passDueAt = Math.min(passDueAt, at);
},
retry: () => {
passDueAt = Math.min(passDueAt, now + (defaults.duty === "evidence" ? 60 * 60_000 : 0));
},
};
if (phase === "sessions") {
const names = (await fs.readdir(base).catch(() => [] as string[]))
const names = (await fs.readdir(base).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return [] as string[];
throw error;
}))
.filter((name) => SCOPE_PATTERN.test(name))
.sort();
const pending = cursor ? names.filter((name) => name > cursor!) : names;
@@ -608,11 +627,18 @@ export function nativeHarnessBackupExpirationService(
const previous = cursor;
cursor = name;
const root = path.join(base, name);
if (path.dirname(root) !== base || !(await isRealDirectory(root))) {
result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1;
try {
if (path.dirname(root) !== base || !(await isRealDirectory(root))) {
result.skipped.unsafe_path = (result.skipped.unsafe_path ?? 0) + 1;
continue;
}
} catch (err) {
result.failed++;
context.retry();
logger.warn({ err, scope: name }, "native runner retention could not inspect a session root");
continue;
}
if (resumeAfter === null) {
if (resumeAfter === null && includeBackups) {
try {
await expireSession(root, name, { dryRun, isSessionBusy, closedBefore }, result);
} catch (err) {
@@ -626,7 +652,7 @@ export function nativeHarnessBackupExpirationService(
stoppedAt = await retireContinuityBreaks(root, name, resumeAfter, context, result);
} catch (err) {
result.failed++;
context.dueAt(now);
context.retry();
logger.warn({ err, scope: name }, "native runner evidence retention skipped a session");
}
if (stoppedAt !== null) {
@@ -651,7 +677,7 @@ export function nativeHarnessBackupExpirationService(
}
} catch (err) {
result.failed++;
context.dueAt(now);
context.retry();
quarantineCursor = null;
logger.warn({ err }, "native runner evidence retention skipped quarantine");
}
@@ -662,6 +688,11 @@ export function nativeHarnessBackupExpirationService(
return result;
}
function nextEvidenceAt() {
if (passActive) return Date.now();
return Number.isFinite(evidenceDueAt) ? evidenceDueAt : null;
}
async function tickCheckpoints(options: NativeHarnessBackupExpirationOptions = {}) {
const closedForDays = Math.max(1, options.closedForDays ?? defaults.closedForDays ?? NATIVE_HARNESS_BACKUP_EXPIRATION_DEFAULT_DAYS);
const result = emptyResult();
@@ -714,5 +745,5 @@ export function nativeHarnessBackupExpirationService(
});
}
return { tick, noteEvidence, tickCheckpoints, registerCheckpointExpiration };
return { tick, noteEvidence, nextEvidenceAt, tickCheckpoints, registerCheckpointExpiration };
}
@@ -0,0 +1,34 @@
import { logger } from "../../middleware/logger.js";
import { registerDeadlineWork } from "../deadline-work.js";
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
import { nativeHarnessBackupExpirationService, type NativeHarnessBackupExpirationOptions } from "./native-harness-backup-expiration.js";
import { onNativeRunnerEvidence } from "./native-runner-evidence-signal.js";
export function registerNativeRunnerEvidenceRetentionWork(
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
db: Parameters<typeof nativeHarnessBackupExpirationService>[0],
options: NativeHarnessBackupExpirationOptions,
canRun: () => boolean,
) {
const service = nativeHarnessBackupExpirationService(db, { ...options, duty: "evidence" });
let continuation: number | null = null;
const worker = registerDeadlineWork(coordinator, DELIVERY_QUEUES.nativeRunnerEvidenceRetention, {
retryMs: 30_000, canRun,
run: async signal => {
try {
const result = await service.tick({ shouldContinue: () => !signal.aborted && canRun() });
if (result.evidenceCandidates > 0 || result.failed > 0) {
logger.info({ ...result, dryRun: options.dryRun ?? false }, "native runner evidence retention");
}
continuation = result.truncated ? Date.now() + 1000 : null;
} catch (err) {
continuation = Date.now() + 60 * 60_000;
logger.warn({ err }, "native runner evidence inspection failed");
}
},
next: async () => continuation ?? service.nextEvidenceAt(),
});
const unsubscribe = onNativeRunnerEvidence(at => { service.noteEvidence(at); worker.wake(); });
return { ...worker, unsubscribe };
}
@@ -16,6 +16,7 @@ test("a task title survives an outage and saves after server recovery", async ({
data: { title: "Another task stays unchanged", status: "backlog" },
}));
await page.goto(`/${company.issuePrefix}/issues/${issue.identifier}`);
await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 });
const header = page.getByTestId("issue-detail-header");
await expect(header.getByRole("heading", { name: issue.title, exact: true })).toBeVisible();
@@ -60,6 +61,7 @@ test("two cached tasks keep separate title saves while the browser is offline",
await expect(page.getByRole("heading", { name: second.title, exact: true })).toBeVisible();
await page.getByRole("link", { name: "Tasks", exact: true }).first().click();
await page.getByRole("link").filter({ hasText: first.title }).first().click();
await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 });
const header = page.getByTestId("issue-detail-header");
await expect(header.getByRole("heading", { name: first.title, exact: true })).toBeVisible();
const dismissAnnouncement = page.getByRole("button", { name: "Dismiss announcement" });
@@ -74,6 +76,7 @@ test("two cached tasks keep separate title saves while the browser is offline",
await secondLink.focus();
await secondLink.press("Enter", { timeout: 5_000 });
await expect(page).toHaveURL(new RegExp(`${second.id}$`));
await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 });
await expect(header.getByRole("heading", { name: second.title, exact: true })).toBeVisible();
await header.getByRole("heading", { name: second.title, exact: true }).click();
await expect(header.locator("textarea")).toHaveValue(second.title);
@@ -189,6 +192,7 @@ test("startup work retries a failed session lookup while other APIs stay healthy
await route.fulfill({ status: 500, json: { error: "session_lookup_failed" } });
});
await page.goto(`/${company.issuePrefix}/issues/${issue.id}`);
await expect(page.getByTestId("task-chat-history-content")).toBeVisible({ timeout: 30_000 });
const header = page.getByTestId("issue-detail-header");
await header.getByRole("heading", { name: issue.title, exact: true }).click();
await header.locator("textarea").fill("Saved after session recovery");