Commit Graph
2 Commits
Author SHA1 Message Date
DottaandPaperclip 5982246771 fix(server): schedule custom-image setup expiry from committed changes (#17443)
## Thinking Path

> - Paperclip manages AI agent work and its execution environments.
> - Interactive image setup sessions must expire even when users leave.
> - The server currently scans these sessions on every maintenance tick.
> - Empty scans keep the instance and database active.
> - Existing session rows already record status and expiry.
> - This pull request uses transaction signals and the shared scheduler
for those rows.
> - Empty and completed setup history no longer create recurring cleanup
queries.

## Linked Issues or Issue Description

**What happened?**

The server checks custom-image setup expiry every heartbeat maintenance
tick, even when no active session exists.

**What did you expect?**

Session writes should update the next cleanup deadline. Existing
provider cancellation and cleanup ownership must remain intact. Empty
state should stay quiet.

**Steps to reproduce**

Run an instance with heartbeat scheduling enabled and no custom-image
setup sessions. Observe repeated setup-expiry reads.

Refs #17145. This is the first PR in the recurring server sleep stack.
Its base is master. The full requested scope and verification contract
are recorded in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`.
Follow-up PRs target their predecessor and each own a separate reason
for recurring work.

## What Changed

- Carry the historical cleanup retry-observation fixture repair and its
existing external test-database hook at the stack foundation. Runtime
scheduling is unchanged.

- Carry the previously verified login-cleanup fixture settlement
correction at the stack foundation so every intermediate CI run receives
it. This is test-only maintenance; custom-image expiry behavior is
unchanged.

- Add a custom-image expiry worker to the existing delivery coordinator.
Remove its maintenance-tick scan.
- Signal initial creation, provider refresh, cancellation/failure, and
promotion inside their owning transaction.
- Keep the active-status predicate, earliest saved expiry, and provider
behavior in the custom-image service. Shared code gains only a queue
topic.
- Preserve startup recovery, bounded retries, concurrent write
generations, and drain deferral. No tables or external alarm changes.
- Add PostgreSQL lifecycle tests and repair this suite's database
cleanup callback. A dedicated external test URL permits disposable
PostgreSQL when the host cannot start another embedded cluster.

## Verification

Current head: `3507c6be1e054fdd93353d5dcf77d4899b44518f`. This remains a
single-duty PR stacked on master.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/3507c6be1e054fdd93353d5dcf77d4899b44518f/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md),
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/3507c6be1e054fdd93353d5dcf77d4899b44518f/doc/plans/2026-10-24-recurring-server-sleep-stack.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.

## Risks

- Signals cover supported service producers sharing the DB owner.
Independent SQL and another process require the existing
restart/reconciliation contract.
- Failed provider cancellation retains the existing failed-session and
historical-cleanup behavior. This PR changes scheduling, not teardown
authority.
- Active rows without an expiry preserve the existing behavior; no
deadline is invented. Cloud sleep eligibility remains conservative.
- The worker remains in the existing heartbeat-enabled startup path. It
does not change disabled-scheduler behavior or prove deployed
suspension.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Review follow-up: Added a real producer commit during a paused deadline
read. Updated startup assertions for coordinator ownership and removed
added code comments. The three affected suites pass (102 tests). The
separate login-cleanup suite also passes locally (16 tests); its prior
CI consume test timed out, and the fresh complete CI run passed.
External PostgreSQL tests now use the repository guard for disposable
databases ending in `_test`.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 14:17:01 -05:00
DottaandPaperclip 71c04c0bfb fix: stop idle environment cleanup polling (#17145)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent environments can leave pending teardown or unverified provider
resources.
> - Recovery must preserve those resources until ownership and workspace
safety allow cleanup.
> - Empty maintenance scans keep an idle instance and its database
active.
> - The existing rows, transaction signals, and scheduler can represent
the actual recovery obligation.
> - This pull request schedules pending cleanup and historical
verification only while work remains.
> - An empty or fully verified instance stops these scans without losing
cleanup recovery.

## Linked Issues or Issue Description

**What happened?**

The server scanned pending environment cleanup on its maintenance tick.
The run reaper also scanned it even when it recovered no runs or active
leases. Historical resource verification scanned every five minutes,
including when all history already had valid cleanup proof.

**What did you expect?**

Writes should wake cleanup. Pending rows, deferred orphan buffers, and
unverified history should retain recovery deadlines. When none remain,
these workers should make no periodic database or provider calls.

**Steps to reproduce**

Start a configured instance with no pending leases and fully verified
history. Leave it idle. Observe repeated pending-cleanup and
historical-verification reads from the maintenance paths.

Related: #17106 supplies the preceding login-cleanup migration. #13523
improves lease-reaper query indexes. #16809 addresses reusable-sandbox
recovery authority. This PR keeps those authority checks in their
existing services.

## What Changed

- Register two workers with the existing delivery coordinator and shared
scheduler. Remove their unconditional maintenance-tick calls. Run the
reaper's inline pending cleanup only after it recovers actual work.
- Add commit-aware signals to the audited lease and custom-image
writers. Existing transactions notify directly; other writers use one
small transaction helper. No tables or migrations are added.
- Derive pending cleanup deadlines from the existing row backoff and
retry or claim metadata. Retain a historical retry only while an
unverified candidate exists.
- Register unwritten orphan buffers across runtimes that share the
database handle. Buffer registration wakes recovery without a database
connection. Load the existing spool at startup and retry failed or
capacity-limited loads while needed.
- Isolate spool failures per runtime. Attempt all buffers and persisted
pending rows, then retain a retry for any failed spool.
- Document producer coverage, idle behavior, and remaining blockers.
Update the sleep inventory.
- Repair the inherited CSS-recovery E2E test: the current build no
longer emits task-page CSS. Exercise the actual lazy environment
stylesheet, preserving real network failure, recovery, and bounded
reload assertions.

## Verification

October 24 integration update: merged current master and resolved the
runtime/spool conflicts. Preserve serialized ingestion of records
appended after startup. Plugin replacement failures now register
deferred recovery with an idle hold; concurrent callers share one
persistence attempt. The producer audit includes this new writer.

- Focused cleanup selection: 270 tests now pass across five suites. The
initial run lacked a built plugin SDK and executed no tests; building
the SDK fixed setup. The new concurrent-flush assertion then reproduced
duplicate rows; a single-flight guard fixed it and all 18 plugin
integration tests passed on rerun.
- Server typecheck passed. After review fixes, all 211 tests across the
cleanup worker, real orphan spool, environment runtime, and plugin
integration suites passed. These overlap the earlier focused selection.
The rate-limit test initially expected a write after twenty idle minutes
to wait; corrected it to assert that subsequent writes cannot accelerate
the five-minute cadence.
- Review fixes preserve retries for unreadable spool
directories/records, keep historical provider checks off the startup
wait while joining them at shutdown, and restore their independent
five-minute minimum spacing. The complete [CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/40108905033)
passed on final commit `79ae1589c3c7b924db4bedb337d0c7ed3d95020b`,
including all E2E shards, server/workspace tests, runner checks,
typecheck, build/public-install verification, and security checks.

- CI found a stale E2E asset matcher: `issue-detail-*.css` is no longer
emitted. Both corrected browser tests passed against a fresh production
UI build and disposable local instance. Fresh Greptile review is 5/5 on
that same final commit, with all review threads resolved.

Previous implementation verification:

- `pnpm -r typecheck` and `pnpm build` passed.
- New PostgreSQL worker suite: 12 tests passed. Covers quiet configured
state, verified history, writes, outer commit and rollback, retry
deadlines, restart, drain, deferred orphan buffers, spool failure,
pagination, provider uncertainty, shutdown, and the idle run-reaper
path.
- Worker and environment runtime suites after review fix: 183 tests
passed. A new assertion reproduced buffer starvation before the fix.
Updated transaction fault injection and isolated simulated-instance
database handles. Server typecheck also passed after the fix.
- Run-recovery suites: 58 tests passed. Pending-cleanup suite: 38 tests
passed after updating the logging fixture to seed real orphan recovery;
an idle reaper no longer invokes cleanup.
- Stop and workspace-recovery integration selection: 268 tests passed;
two live-provider tests skipped by their environment gate. This overlaps
the worker suite and is not an additional unique-test count.
- Started the full local `pnpm test:run`, then stopped that older
duplicate run after CI found the fixture mismatch. The corrected 38-test
suite passed locally. The full sharded [CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/39678174479)
passed on final commit `fc0ff3c4554ca4190a8b0b81ec06ef7065b8ad2c`,
including build, typecheck, runner, server, workspace, and E2E checks.
This is not a claim that the local full run completed.

## Risks

- Missing a future producer signal could delay discovery until restart.
The producer audit is documented in `doc/idle-environment-cleanup.md`.
Independent SQL writers and another server process are outside the
existing process-local notification contract.
- Historical candidates blocked by ownership or provider uncertainty
keep a five-minute retry until they can be verified. Provider
destruction, claim fencing, Stop retention, and workspace safety remain
controlled by their existing services.
- This does not remove orphan-active discovery or legacy shutdown
settlement from run recovery. Active custom-image expiry is also still
periodic. External wake coverage does not change.
- Local tests prove these workers become quiet and recover. They do not
establish deployed instance suspension or fleet cost savings.

## Model Used

OpenAI Codex (GPT-6), with code execution and reasoning. The exact
serving model ID and context size were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-24 09:04:49 -07:00