Files
PaperClipAI/doc/idle-terminal-workspace-pagination.md
T
DottaandPaperclip d23dd19750 fix(server): preserve terminal workspace cursors and retry deadlines (#17505)
## Thinking Path

> - Paperclip manages agent work and its execution workspaces.
> - Workspace recovery scans candidates and retains deadlines for
unfinished work.
> - Rounded database timestamps can prevent a scan from advancing.
> - Separate clock reads can give one retry obligation different
deadlines.
> - Repeated inspection must preserve both scan progress and the saved
retry time.
> - This PR keeps full cursor precision and computes each retry deadline
once.
> - Recovery can finish its scan and wait until work is actually due.

## Linked Issues or Issue Description

**What happened?**

Terminal workspace recovery can repeatedly inspect the oldest skipped
candidate when its timestamp contains microseconds. Eligible workspaces
behind it never reach cleanup. Separately, nested-clone migration and
isolated-repository rehome recovery use separate clock reads for their
retry gate and advertised deadline. A later inspection can move the
advertised deadline backward.

**Expected behavior**

Each bounded scan advances using the complete timestamp and ID. The
retry gate and advertised deadline represent the same saved time,
including when the clock advances between calls.

**Steps to reproduce**

For cursor starvation, place two skipped workspaces with microsecond
timestamps before an eligible terminal workspace and use a one-row page.
For deadline drift, inject a clock that advances one millisecond per
read, run disabled migration or rehome recovery, then inspect it again
before the retry time.

Stacked on #17497. Related cursor fixes #17479 and #17482 have separate
owners. This PR owns workspace recovery cursor and retry precision.

## What Changed

- Preserve the database timestamp as text in the private timestamp/ID
cursor.
- Compute one retry time per migration or rehome outcome and reuse it in
the existing maps.
- Test the cursor with real PostgreSQL microsecond timestamps and the
retry boundary with a deterministic advancing clock.
- Support validated disposable external test databases in the affected
fixtures, retaining embedded fallback.
- Document the evidence in `doc/idle-terminal-workspace-pagination.md`.

## Verification

Current test revision: `9d846c84e76c90893ab62a09548c11bbdb627771`,
followed by docs-only head `a34dc50de3`, which received both 5/5 reviews
and passing CI. This candidate normally integrates landed approval
retention `553a004926`. Every runtime and acceptance-test source matches
that tested/reviewed tree; only four independent Runner E2E accounting
files and the ledger differ. Migration metadata is unchanged. Existing
local checks are reused for their unchanged source; fresh final-head CI
and reviews are required against master. This remains the
terminal-precision duty.

- **136 cases pass** in four complete suites: execution workspace
service, terminal worker, isolated repository rehome and nested-clone
migration. Server typecheck and production build pass with the current
Runner vendor.
- The preceding `f0f0de1f6f` run also passed all 136 on three newly
created, unmigrated PostgreSQL databases. The suites themselves applied
migrations. The current run reuses those clean fixtures after
integrating the parent's approval wake repair; it is not presented as a
fresh-database run.
- The regressions retain real PostgreSQL microsecond timestamps and an
advancing clock. Before repair, skipped rows starved eligible cleanup
and early inspection moved a saved retry by one millisecond. Current
assertions preserve exact progress/deadline behavior; no timeout or
business policy is weakened.
- Earlier external-fixture setup tests failed with missing
company/restore-point schema. Their setup repair and red/green record
remain in `doc/idle-terminal-workspace-pagination.md`. This PR adds no
migration or new queue.
- Earlier stack integration receipts and failures remain in the stack
ledger and `doc/idle-approved-tool-recovery.md`; they are historical
parent evidence, not this head's acceptance. No full local repository
suite, deployed suspension, fleet cost or live provider qualification is
claimed.

Commands: server Vitest for `execution-workspaces-service`,
`terminal-workspace-work`, `isolated-repository-rehome` and
`nested-clone-migration`; server typecheck/build. Fresh exact-head CI
and Greptile/Ponytail 5/5 remain required before merge in dependency
order.

## Risks

The cursor remains private query state. Retry intervals, eligibility,
cleanup authority, and shutdown behavior are unchanged. No schema,
queue, external alarm, or additional polling loop is introduced. The
tests establish workspace recovery behavior, not host suspension or cost
savings.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 19:42:58 -05:00

6.0 KiB

Terminal workspace scan and retry deadline precision

Terminal workspace maintenance already uses the shared delivery coordinator and retains deadlines only for outstanding work. Its bounded scan must progress past ineligible rows so that later eligible work can finish and the worker can park.

The source query orders candidates by (updated_at, id). PostgreSQL timestamps can contain microseconds. The former cursor used a JavaScript Date, which loses that precision. A skipped row could then compare greater than its own rounded cursor. Repeated full pages kept recovery active and prevented later cleanup.

The query now carries the database timestamp as text alongside each workspace. Only the private scan cursor uses that value. Workspace data, business deadlines, and the frozen rotation boundary keep their existing types and behavior. The cursor comparison uses the same timestamp/ID tuple as the ordering. The source service continues to own cleanup eligibility, authorization, concurrency fences, page limits, abort handling, failed-page recovery, and rotation under churn. No scheduler API, queue, migration, producer signal, or external alarm changes.

Verification

The existing full-page starvation regression now gives its skipped rows actual PostgreSQL microsecond timestamps. Before the repair, the eligible workspace was never archived. After the repair, the full workspace service and terminal worker selection passes all 87 tests, including bounded rotation under continuous churn, changed rows behind the cursor, failed-page recovery, quiet workers, and shutdown. The service suite accepts the repository's validated disposable external PostgreSQL URL. Setup applies pending migrations before opening the test client, then clears fixture tables before the first test as well as after each test. This supports a fresh database and reuse after an interrupted test run. The database connection closes during teardown. Server typechecking passes.

This fixes recovery progress. It does not establish provider suspension or change which active or unresolved resources prevent instance sleep.

Workspace-wide typechecking and the production build also pass. Full sharded CI and fresh review remain required before this PR is ready.

Review follow-up reproduced the missing-schema failure against a newly created, unmigrated disposable database. The setup repair migrates that same database inside the suite; the full workspace-service selection then runs without manual schema preparation. The production cursor implementation is unchanged.

Preserve one retry deadline through early inspection

Nested-clone migration and isolated-source rehoming kept a retry guard and a separate advertised scheduler deadline. Each value called the clock separately. When the clock advanced between those calls, an early inspection copied the earlier guard back into the advertised deadline. CI observed a one-millisecond backward change in an otherwise unchanged retry.

Each path now computes one retry timestamp and stores that value in all of its retry and scheduling state. The cadence, cleanup eligibility, quiet conditions, rotation, and wake contract do not change. No new timer, queue, or table is added.

The existing nested-clone regression and the isolated-rehome dry-run case now use a clock that advances one millisecond per read. Both failed before the fix with the same one-millisecond drift. Both exact-deadline assertions remain. The complete migration suites passed all 49 cases afterward, using real PostgreSQL and Git, including preserved files/history and interrupted migration recovery. The complete workspace-service and terminal-worker suites passed another 87 cases; server typechecking passed. The two focused pre-fix failures used a name filter intentionally; the reported final 136 cases are four complete suites without skips. Validated external test database hooks keep each migration suite on its own disposable database and retain the existing embedded fixture fallback.

These are local correctness and scheduling results, not deployed host-sleep or external-wake proof. Remote acceptance must be checked at the published head.

External fixture migrations (October 26 review repair)

Review found that the two migration suites opened external test databases without applying migrations. The previous verification used prepared databases, which hid this setup gap. Both selected retry cases failed against new empty disposable databases with missing companies and restore-point tables. Setup now applies pending migrations before creating its test client, matching the workspace service fixture. The embedded path is unchanged.

All 49 cases in the two complete suites then passed against those same databases, including Git/filesystem migration, recovery, and exact retry-deadline checks. The fixture itself prepared the databases. This repair changes no production recovery code. Evidence: /tmp/sleep-workspace-fresh-red.log and /tmp/sleep-workspace-fresh-green.log.

Current parent verification

Approval-retention parent 3efda33812 is integrated with normal merges. All workspace precision, retry and fixture implementations remain unchanged from the preceding candidate. All 136 cases in the complete workspace-service, terminal worker, isolated-rehome and nested-clone suites pass at 9d846c84e7; server types and production build pass with the current Runner vendor.

The preceding f0f0de1f6f run also passed all 136 cases with three newly created, unmigrated external databases: the suites themselves applied the current migrations. The 9d846c84e7 run reused those clean fixtures after the parent's approval wake repair. This retains actual proof of fixture self-preparation without mislabeling the reused run as a fresh-database test. No new migration, queue, wake contract or workspace policy change is added by this PR.

Earlier microsecond starvation, one-millisecond retry drift and missing-schema failures remain documented above. Fresh exact-head CI and both 5/5 reviews remain required in stack order; the parent is not claimed merged.