## Thinking Path > - Paperclip manages agent work and its execution workspaces. > - Workspace recovery scans candidates and retains deadlines for unfinished work. > - Rounded database timestamps can prevent a scan from advancing. > - Separate clock reads can give one retry obligation different deadlines. > - Repeated inspection must preserve both scan progress and the saved retry time. > - This PR keeps full cursor precision and computes each retry deadline once. > - Recovery can finish its scan and wait until work is actually due. ## Linked Issues or Issue Description **What happened?** Terminal workspace recovery can repeatedly inspect the oldest skipped candidate when its timestamp contains microseconds. Eligible workspaces behind it never reach cleanup. Separately, nested-clone migration and isolated-repository rehome recovery use separate clock reads for their retry gate and advertised deadline. A later inspection can move the advertised deadline backward. **Expected behavior** Each bounded scan advances using the complete timestamp and ID. The retry gate and advertised deadline represent the same saved time, including when the clock advances between calls. **Steps to reproduce** For cursor starvation, place two skipped workspaces with microsecond timestamps before an eligible terminal workspace and use a one-row page. For deadline drift, inject a clock that advances one millisecond per read, run disabled migration or rehome recovery, then inspect it again before the retry time. Stacked on #17497. Related cursor fixes #17479 and #17482 have separate owners. This PR owns workspace recovery cursor and retry precision. ## What Changed - Preserve the database timestamp as text in the private timestamp/ID cursor. - Compute one retry time per migration or rehome outcome and reuse it in the existing maps. - Test the cursor with real PostgreSQL microsecond timestamps and the retry boundary with a deterministic advancing clock. - Support validated disposable external test databases in the affected fixtures, retaining embedded fallback. - Document the evidence in `doc/idle-terminal-workspace-pagination.md`. ## Verification Current test revision: `9d846c84e76c90893ab62a09548c11bbdb627771`, followed by docs-only head `a34dc50de3`, which received both 5/5 reviews and passing CI. This candidate normally integrates landed approval retention `553a004926`. Every runtime and acceptance-test source matches that tested/reviewed tree; only four independent Runner E2E accounting files and the ledger differ. Migration metadata is unchanged. Existing local checks are reused for their unchanged source; fresh final-head CI and reviews are required against master. This remains the terminal-precision duty. - **136 cases pass** in four complete suites: execution workspace service, terminal worker, isolated repository rehome and nested-clone migration. Server typecheck and production build pass with the current Runner vendor. - The preceding `f0f0de1f6f` run also passed all 136 on three newly created, unmigrated PostgreSQL databases. The suites themselves applied migrations. The current run reuses those clean fixtures after integrating the parent's approval wake repair; it is not presented as a fresh-database run. - The regressions retain real PostgreSQL microsecond timestamps and an advancing clock. Before repair, skipped rows starved eligible cleanup and early inspection moved a saved retry by one millisecond. Current assertions preserve exact progress/deadline behavior; no timeout or business policy is weakened. - Earlier external-fixture setup tests failed with missing company/restore-point schema. Their setup repair and red/green record remain in `doc/idle-terminal-workspace-pagination.md`. This PR adds no migration or new queue. - Earlier stack integration receipts and failures remain in the stack ledger and `doc/idle-approved-tool-recovery.md`; they are historical parent evidence, not this head's acceptance. No full local repository suite, deployed suspension, fleet cost or live provider qualification is claimed. Commands: server Vitest for `execution-workspaces-service`, `terminal-workspace-work`, `isolated-repository-rehome` and `nested-clone-migration`; server typecheck/build. Fresh exact-head CI and Greptile/Ponytail 5/5 remain required before merge in dependency order. ## Risks The cursor remains private query state. Retry intervals, eligibility, cleanup authority, and shutdown behavior are unchanged. No schema, queue, external alarm, or additional polling loop is introduced. The tests establish workspace recovery behavior, not host suspension or cost savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
6.0 KiB
Terminal workspace scan and retry deadline precision
Terminal workspace maintenance already uses the shared delivery coordinator and retains deadlines only for outstanding work. Its bounded scan must progress past ineligible rows so that later eligible work can finish and the worker can park.
The source query orders candidates by (updated_at, id). PostgreSQL timestamps
can contain microseconds. The former cursor used a JavaScript Date, which loses
that precision. A skipped row could then compare greater than its own rounded
cursor. Repeated full pages kept recovery active and prevented later cleanup.
The query now carries the database timestamp as text alongside each workspace. Only the private scan cursor uses that value. Workspace data, business deadlines, and the frozen rotation boundary keep their existing types and behavior. The cursor comparison uses the same timestamp/ID tuple as the ordering. The source service continues to own cleanup eligibility, authorization, concurrency fences, page limits, abort handling, failed-page recovery, and rotation under churn. No scheduler API, queue, migration, producer signal, or external alarm changes.
Verification
The existing full-page starvation regression now gives its skipped rows actual PostgreSQL microsecond timestamps. Before the repair, the eligible workspace was never archived. After the repair, the full workspace service and terminal worker selection passes all 87 tests, including bounded rotation under continuous churn, changed rows behind the cursor, failed-page recovery, quiet workers, and shutdown. The service suite accepts the repository's validated disposable external PostgreSQL URL. Setup applies pending migrations before opening the test client, then clears fixture tables before the first test as well as after each test. This supports a fresh database and reuse after an interrupted test run. The database connection closes during teardown. Server typechecking passes.
This fixes recovery progress. It does not establish provider suspension or change which active or unresolved resources prevent instance sleep.
Workspace-wide typechecking and the production build also pass. Full sharded CI and fresh review remain required before this PR is ready.
Review follow-up reproduced the missing-schema failure against a newly created, unmigrated disposable database. The setup repair migrates that same database inside the suite; the full workspace-service selection then runs without manual schema preparation. The production cursor implementation is unchanged.
Preserve one retry deadline through early inspection
Nested-clone migration and isolated-source rehoming kept a retry guard and a separate advertised scheduler deadline. Each value called the clock separately. When the clock advanced between those calls, an early inspection copied the earlier guard back into the advertised deadline. CI observed a one-millisecond backward change in an otherwise unchanged retry.
Each path now computes one retry timestamp and stores that value in all of its retry and scheduling state. The cadence, cleanup eligibility, quiet conditions, rotation, and wake contract do not change. No new timer, queue, or table is added.
The existing nested-clone regression and the isolated-rehome dry-run case now use a clock that advances one millisecond per read. Both failed before the fix with the same one-millisecond drift. Both exact-deadline assertions remain. The complete migration suites passed all 49 cases afterward, using real PostgreSQL and Git, including preserved files/history and interrupted migration recovery. The complete workspace-service and terminal-worker suites passed another 87 cases; server typechecking passed. The two focused pre-fix failures used a name filter intentionally; the reported final 136 cases are four complete suites without skips. Validated external test database hooks keep each migration suite on its own disposable database and retain the existing embedded fixture fallback.
These are local correctness and scheduling results, not deployed host-sleep or external-wake proof. Remote acceptance must be checked at the published head.
External fixture migrations (October 26 review repair)
Review found that the two migration suites opened external test databases without
applying migrations. The previous verification used prepared databases, which
hid this setup gap. Both selected retry cases failed against new empty disposable
databases with missing companies and restore-point tables. Setup now applies
pending migrations before creating its test client, matching the workspace
service fixture. The embedded path is unchanged.
All 49 cases in the two complete suites then passed against those same databases,
including Git/filesystem migration, recovery, and exact retry-deadline checks.
The fixture itself prepared the databases. This repair changes no production
recovery code. Evidence: /tmp/sleep-workspace-fresh-red.log and
/tmp/sleep-workspace-fresh-green.log.
Current parent verification
Approval-retention parent 3efda33812 is integrated with normal merges. All
workspace precision, retry and fixture implementations remain unchanged from the
preceding candidate. All 136 cases in the complete workspace-service, terminal
worker, isolated-rehome and nested-clone suites pass at 9d846c84e7; server types
and production build pass with the current Runner vendor.
The preceding f0f0de1f6f run also passed all 136 cases with three newly created,
unmigrated external databases: the suites themselves applied the current
migrations. The 9d846c84e7 run reused those clean fixtures after the parent's
approval wake repair. This retains actual proof of fixture self-preparation
without mislabeling the reused run as a fresh-database test. No new migration,
queue, wake contract or workspace policy change is added by this PR.
Earlier microsecond starvation, one-millisecond retry drift and missing-schema failures remain documented above. Fresh exact-head CI and both 5/5 reviews remain required in stack order; the parent is not claimed merged.