diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index ff918b88dfc..4a7392f5a9e 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -3052,3 +3052,115 @@ The owner audit preserves the wrong initial ordinary-release completion assertio and its authority-preserving correction. Fresh-head gates remain pending; nothing merged. Earlier E2E failures on #17490/#17572 and voice findings on #17576/#17580 were revalidated and remain unresolved. All original remaining duties stay in scope. + + +Slice 50 repaired-head acceptance: #17709 at +`9a9e4262bb9beb39d67d26320c14e99651b6d6bd` now has all 53 checks passing, +Greptile 5/5 at that exact head and zero unresolved review threads. Its PR body +records the result and retains the local embedded-database limitation. It remains +draft and unmerged. `/tmp/sleep50-release-latest-gates.json` records fresh CI. +Older stack findings and master reconciliation remain outstanding. + +Slice 51 is now `codex/sleep-51-retained-native-session-cleanup`, based on that +repaired head. The existing joined discovery has a cohesive native cleanup owner; +scoped reconstruction preserves future lease deadlines. All 35 database discovery/ +finalization cases, 72 exact physical-cleanup cases and server types pass. See +[the audit](2026-10-26-retained-native-session-cleanup-sleep-audit.md) for producer, +physical-owner, shutdown and outcome gaps. No scheduled cleanup worker is wired +and periodic cleanup remains enabled. The full scope remains incomplete. + + +Slice 51 outcome checkpoint: pre-claim database/commit failures now propagate +instead of being discarded as ineligible history. Typed waiting outcomes preserve +process/session/environment/maintenance ownership and the actual coordinator lease +expiry. Four injected failures, a real PostgreSQL row-lock regression and six +waiting-state regressions reproduced the old behavior. The repaired source passes +798 executor cases (15 embedded-DB cases excluded), 36 database cases and server +types; strengthened physical reactivation tests pass all 80 selected cases again. +The owner audit preserves the reused-fixture failure and initial fixture type error. +Worker registration, producer/release notifications, quiet/lifecycle/cost proof, +periodic removal and a slice 51 PR remain outstanding. No polling duty is claimed +removed by this checkpoint; all original stack scope stays active and unmerged. + + +Slice 51 worker checkpoint: retained cleanup now has scoped shared-scheduler +reconstruction, exact lease deadlines, a concrete process-exit probe and event +waiting for other owners. Startup/manual/scheduled attempts join duplicates and +serialize different inspections. A real finalization-notification missed wake was +reproduced and repaired through the existing transaction fanout. All 19 worker +lifecycle cases, 36 existing cleanup/finalization cases, 37 reassessment consumer +cases and server types pass. The 103-item backlog becomes SQL-quiet after settlement. +Actual local/environment/process/maintenance producer coverage, physical callback +drain ownership, large-history costs, startup wiring, periodic removal, publication +and remote acceptance remain open. Periodic cleanup is still enabled; no new PR +or merge is claimed and the full original stack scope stays active. + + +Slice 51 durable release checkpoint: six real-producer missed wakes were +reproduced and repaired: single/run-wide lease release, settled process release, +stop-only cleanup, export-resume settlement and confirmed sandbox absence. +Notifications remain run/company-scoped and tied to outer transaction ownership. +All 208 selected worker/environment/workspace consumer cases and server types pass. +The owner audit retains the asynchronous rollback-test failure and correction, +and distinguishes controlled stop evidence from actual physical/provider proof. +Local session and maintenance releases, remaining finalization producers, callback +ownership, startup wiring, cost proof, periodic removal and PR gates remain open. +Periodic cleanup remains enabled. No PR or merge; all original scope stays open. + + +Slice 51 local owner checkpoint: session execution/initialization releases and +all warm-session retirement paths now emit local scoped hints. Warm removal checks +the expected owner; the existing close/retention decisions remain authoritative. +Three lifecycle regressions reproduced missing release notifications. Final +executor verification passes 798 cases (15 embedded-DB cases excluded), the real +PostgreSQL worker suite passes 26 cases, and server types pass. Retained maintenance +completion and physical callback ownership are next; the audit records the exact +boundary gap. Startup registration, periodic removal, cost proof, remaining +producers and PR acceptance remain incomplete. Nothing published or merged; +all original recurring-server duties and older stack gates remain open. + + +Slice 51 maintenance and terminal-producer checkpoint: cleanup now owns original +Runner maintenance attempts/callbacks through settlement, using Runner's existing +drain boundary. Finalizer projection and heartbeat terminal writes now notify +cleanup in their transactions. Three reproduced producer regressions include the +real separate-commit decision/projection race. All 64 worker/lifecycle cases, +146 finalizer corpus cases, 799 selected executor cases, 19 Runner settlement +cases and server types pass; 15 embedded-DB cases remain excluded for the recorded +host limitation. The audit records earlier failures and fixture corrections. +Startup wiring, query cost, periodic removal, full gates and publication remain. +No PR or merge; the complete recurring-server scope and older stack gates stay open. + + +Slice 51 startup integration: the enabled server now registers retained cleanup +and its shutdown subscription. The periodic orphan pass skips this duty; explicit +and startup recovery retain their bounded joined sweep. Heartbeat joins original +scheduled physical operations through shutdown. All 100 startup/worker cases and +server types pass. After the user's explicitly approved cleanup of 11 orphaned +semaphore sets, the full executor suite passes all 814 cases with no exclusions, +including strengthened initialization-release coverage. A 10,000-completed-session +probe measures broad discovery at 28–30 ms locally and run-scoped discovery at +0.030 ms; broad discovery remains linear in history and no deployed claim is made. +Full workspace acceptance, PR publication/review, older gates and every remaining +duty stay open. No merge has occurred. + + +Slice 51 publication: [#17717](https://github.com/paperclipai/paperclip/pull/17717) +is draft on #17709. Published head `8c62a6b0c0` passed workspace types/build and +received Greptile 5/5, but three CI gates failed: two tests retained the old +heartbeat-owned maintenance drain, and the Copilot transport identity was stale. +The repair follows the cleanup-owned lifetime and creates a new attestation while +preserving old profile history. Full affected recovery/process suites (34 + 408) +and profile tests (69) pass. The local whole-suite OAuth timeout is retained in +the audit alongside a passing complete 472-case isolated rerun. Latest-head CI +and review acceptance are still pending; no stack PR is merged. + +The next native finalization audit identifies three distinct remaining duties: +workspace copy-back/validation; accepted-result commit/projection; and resource +settlement after finalization. Current committed-history upkeep repeatedly calls +finalization and best-effort cleanup, including swallowed resource-release +failures. A completed result cannot be used as proof that those residual duties +are finished. Audit first-result publication in `paperclip-control-plane-port.ts`, +workspace operation writes, durable workspace-owner release, retry updates and +cleanup completion before disabling their existing recovery paths. No migration +of these next duties is claimed by slice 51. diff --git a/doc/plans/2026-10-26-retained-native-session-cleanup-sleep-audit.md b/doc/plans/2026-10-26-retained-native-session-cleanup-sleep-audit.md new file mode 100644 index 00000000000..e87ecb7d813 --- /dev/null +++ b/doc/plans/2026-10-26-retained-native-session-cleanup-sleep-audit.md @@ -0,0 +1,620 @@ +# Retained native-session cleanup: sleep migration audit + +Parent: slice 50, #17709 at `9a9e4262bb9beb39d67d26320c14e99651b6d6bd`. +Branch: `codex/sleep-51-retained-native-session-cleanup`. +Inspected master: `1fbbef45a8d399771add7a27a04b12f3a20e264c`. + +## Obligation and ownership + +A finished native run with an accepted result can retain an exact local checkpoint +following the specific durable-suspend close failure. Cleanup may settle that +checkpoint only after its existing artifact, journal, identity, lease and physical +owner proofs succeed. A terminal run alone grants no cleanup authority. Failed +maintenance with uncertain provider ownership stays quarantined for inspection. + +The old orphan pass calls a joined, per-database discovery sweep on every pass; +startup native recovery calls it too. Its cursor wraps after the last row, so an +ineligible retained history is inspected repeatedly. The sweep takes at most five +candidates (normally one). It keeps one pending promise per database and continues +past a failed candidate. Heartbeat tracks the attempt in its active execution +promises and joins retained runner maintenance callbacks before releasing shutdown +ownership. Preserve these lifecycle guarantees when moving to the shared scheduler. + +The discovery and scheduling owner is now +`server/src/services/native-runtime/retained-native-session-cleanup.ts`. +`native-finalization-reconciler.ts` retains its export for compatibility; heartbeat +imports the cohesive owner directly. Provider-specific exact-state cleanup remains +in `native-session-executor.ts`, which already owns the physical session and its +admission rules. No provider protocol belongs in index.ts or the generic scheduler. + +## Current implementation checkpoint + +- Extracted existing bounded, joined discovery without changing its behavior. +- Added company/run/task/agent-scoped, keyset-paged candidate discovery. It returns + existing lease metadata. Scheduled reconstruction can include future leases; + ordinary sweep callers still exclude an unexpired claim. +- Candidate discovery still excludes nonterminal runs, unaccepted results, + uncommitted assessments/decisions, saved finalization retry work, recorded runner + epochs, operator-required source archives and ambiguous maintenance history. +- The exact legacy two-entry pre-ownership maintenance exception remains unchanged. +- A separate shared-scheduler worker now consumes scoped candidates and typed + outcomes. Its lease deadline comes from durable state; a concrete unowned PID + gets a scoped 30-second absence probe. Session/environment/maintenance waits + need release notifications; ineligible history has no periodic fallback. +- The existing finalization notification fanout includes cleanup. There is also + a per-database local release subscription boundary. Durable owner releases, + local initialization/execution/warm-session releases and finalizer/heartbeat + terminal projection are covered. The worker is registered in server startup. +- No new table, migration, independent timer or provider operation is added. +- **The periodic orphan pass now skips this cleanup duty. Startup and explicit + manual recovery remain supported. Tests establish worker quiet under named + conditions; deployed app/database suspension has not been qualified.** + +## Reactivation and outcome audit still required + +| Condition or producer | Existing owner / evidence | Required handling before periodic removal | +| --- | --- | --- | +| Accepted result, committed decision, terminal projection or recovered close failure | Finalizer, status committer, executor; existing native-finalization notification fanout | Audit every writer, register cleanup intent in the owning transaction, and test real commit/rollback | +| Coordinator claim expires or clears | Native finalization row | Reconstruct its actual lease deadline; recheck claim ownership at execution; wake on early release | +| Active/pending-cleanup environment lease releases | Environment runtime, workspace settlement and cleanup writers | Audit release/deletion writers, not only pending-cleanup notifications; wake affected runs after commit | +| Process identity is cleared after verified stop | `native-local-process-stop.ts`, recovery and executor | Retain server-authored stop evidence and same-transaction notification; absence alone never authorizes replay | +| Same session is active, initializing or warm | Executor-owned maps and release/close paths | Use local completion/release signals with company/agent or narrower scope; a different run can own the same session | +| Prior retained callback is still running | Runner maintenance operation registry and drain function | Preserve original promise ownership and reactivate after settlement; a timeout is not proof the callback stopped | +| Relevant PID is still live without a controllable local owner | Exact cleanup's process check | Decide the explicit scoped probe contract; if needed, retain only a concrete unresolved obligation, with stop condition and documented sleep impact | +| Database failure or NOWAIT contention before claim | Executor now propagates operational failures; real lock regression passes | Shared worker must retain recovery and reconstruct durable state, especially after an unknown commit; it must not blindly replay physical effects | +| Invalid/foreign/changed journal, contract, provider state or artifacts | Existing exact-state verifier | Preserve refusal and original evidence; no automatic provider launch or broad retry of unknown outcomes | +| Cleanup already attempted with unknown outcome | Durable epoch/maintenance/archive history | Keep quarantine and existing operator-required policy, with no new automatic retry | + +The original `not_eligible` outcome conflated transient local ownership, database +errors and permanent physical refusal. The operation now returns a typed deferred +outcome identifying process exit, session release, environment release, retained +maintenance settlement or a real coordinator lease expiry. The lease outcome +carries the saved expiry. Operational database/commit/unexpected I/O errors propagate; +malformed stored input, malformed JSON and missing/unsafe/changed proof artifacts +remain ineligible. Post-claim failures still retain the existing operator-required +quarantine policy. No new physical retry is authorized by this outcome contract. + +The shared worker consumes this contract, but is not enabled in startup yet. +Before enabling it, complete the actual release producers for each waiting outcome. +It must not drop deferred work without the corresponding release notification or +concrete probe/deadline, and must not retry every ineligible retained history. A valid PID whose absence +cannot be proven remains deferred and cannot grant cleanup authority. The supported +process-exit observation and local session-release contracts remain to implement. + +The implementation uses the existing scoped deadline worker for reconstruction, +coalescing, fair bounded dispatch, retries and drain retention. Eligibility and +execution remain with the native cleanup owner. Duplicate startup/manual/scheduled +attempts join their original per-run promise; different cleanup inspections share +one per-database FIFO lane. Production wiring must provide a cleanup callback that +retains ownership through the original physical callbacks, including ones that +outlive a bounded maintenance timeout. Tests currently prove the promise-joining +boundary; heartbeat's retained-callback drain still needs integration. +Preserve startup/manual recovery compatibility while removing only this duty from +the periodic orphan pass after coverage is complete. + +Process-local transaction notifications cover supported in-process writers only. +No independent SQL-writer or multi-replica notification transport is being added. +Cloud alarm coverage and sleep eligibility remain unchanged and conservative. + +## Evidence + +All commands ran in the dedicated stack worktree; no source edits occurred while +local acceptance processes were running. External test databases are dedicated +scratch databases, avoiding the separately diagnosed embedded-PostgreSQL host +semaphore limit. No shared OS semaphore was removed. + +- Before extraction: all 32 existing `native-finalization-recovery.test.ts` cases + pass. `/tmp/sleep51-before.log` and `.status` (0). +- After extraction: the same 32 pass against a second fresh database. + `/tmp/sleep51-after.log` and `.status` (0). +- All 72 retained native cleanup activation cases pass, including exact original + evidence, foreign/live owners, archive/claim commit failures, stalled commits, + lease loss, changed artifacts, quarantine and stopped-identity proof. Other + executor tests were intentionally not selected (733 skipped). + `/tmp/sleep51-physical.log` and `.status` (0). +- Scoped discovery adds three database tests: future lease reconstruction without + early legacy execution; company/run/task/agent isolation with a real second + tenant; bounded keyset traversal excluding already-attempted maintenance. + All 35 finalization/discovery cases pass on a third fresh database. + `/tmp/sleep51-discovery.log` and `.status` (0). +- Server typecheck passes after extraction and after scoped discovery. + `/tmp/sleep51-types.log` and `/tmp/sleep51-discovery-types.log`, both status 0. + +These are 107 distinct selected passing cases on the final source (35 discovery/ +finalization and 72 physical-operation cases; the physical owner was unchanged +by the subsequent discovery extension). This is not full-suite acceptance, +worker lifecycle proof, deployed sleep proof or PR readiness. + +## Remaining acceptance + +- [x] Explicit transient/dormant/terminal cleanup outcomes and operational failures. +- [x] Producer audit and real commit/rollback/reactivation tests for the supported in-process writers. +- [x] Local session-release and retained-callback completion ownership. +- [x] Shared worker startup, first producer, retry, changes during sweep, drain, + resume and shutdown; prove SQL/network quiet with retained history. +- [x] Discovery/backlog cost and fairness under the measured 10,000-history fixture. +- [x] Startup/periodic wiring and removal of this duty from the orphan pass. +- [ ] Applicable broader verification, complete PR template, CI and 5/5 review. + +All other duties in the recurring-server stack remain in scope. No merge is +requested or authorized by this migration checkpoint. + + +## Outcome contract checkpoint + +The retained cleanup owner now distinguishes waiting, refusal and operational +failure. This is a prerequisite to scheduling; periodic discovery is still enabled. +The legacy sweep continues to isolate and log a failed candidate, advance its cursor, +and release its joined owner. Shared scheduling and producer coverage remain pending. + +Regression evidence: + +- Three injected transaction failures (disconnection, nested NOWAIT, missing commit + acknowledgement) and the existing rolled-back source-claim commit failure all + incorrectly returned `not_eligible` before repair. Four expected failures in + `/tmp/sleep51-outcome-before.log` / `.status` (1); 73 other selected cases passed. +- The first real lock invocation reused a database with company prefix `NCS` and + failed fixture setup with unique violation 23505. All 36 tests were skipped; + this was not product evidence. `/tmp/sleep51-lock-before.log` / `.status` (1). +- On a fresh dedicated database the real competing run-row lock reproduced the + incorrect ineligible outcome. `/tmp/sleep51-lock-before-fresh.log` / `.status` + (1), one failed regression and 35 deliberately unselected cases. +- After preserving operational errors, all 77 cleanup-operation cases and all + 36 database discovery/finalization cases passed. Logs: `/tmp/sleep51-outcome-after` + and `/tmp/sleep51-lock-after`, each `.log` and `.status` (0). +- Six further regressions reproduced incorrectly discarded waiting states: live + run/process ownership, live provider process, coordinator lease, active environment + lease and busy legacy maintenance. `/tmp/sleep51-deferred-before.log` / `.status` + (1), six failed and 74 passing selected cases. +- The deferred outcome repair passes all 80 cleanup-operation cases. A typecheck + caught the new malformed-input fixture casting its V1 field to the broader input + union; correcting only that fixture's cast restored server typecheck. Preserve + `/tmp/sleep51-outcomes-types.log` / `.status` (1) and the corrected + `/tmp/sleep51-outcomes-types-final.log` / `.status` (0). +- Broader executor verification passes 798 cases, with the 15 embedded-database + remote warm-ownership cases deliberately excluded for the existing host resource + limit. `/tmp/sleep51-executor-consumers.log` / `.status` (0). The full 36-case + database suite also passes on a fresh database after the outcome changes: + `/tmp/sleep51-discovery-final.log` / `.status` (0). Total: 834 distinct cases. +- The final strengthened physical tests verify a deferred run can settle after + its run process, coordinator lease, environment lease or retained maintenance + blocker clears, including release of temporary local scope reservations. + All 80 selected cases pass again: `/tmp/sleep51-reactivation.log` / `.status` (0). + Provider-PID deferral is covered, but its exit-driven reactivation and local + session-owner notification still require integration tests with the worker. +- Final server typecheck passes after the strengthened reactivation assertions: + `/tmp/sleep51-reactivation-types.log` / `.status` (0). + +No source edits occurred during the cited local acceptance processes. No Cloud +alarm coverage, sleep eligibility, quarantine policy or provider protocol changed. +All other stack duties and older acceptance gaps remain open. + + +## Shared worker checkpoint + +New owner: `native-runtime/retained-native-session-cleanup-work.ts`. +Notification boundary: `retained-native-session-cleanup-notifications.ts`. +The public finalization notification helper now wakes this queue in the same +transaction as its other finalization duties. Provider mechanics stay in the +executor. The scheduler receives only candidate identity, normalized outcome and +next deadline; no provider selection was added to shared scheduling. + +- Discovery reconstructs scoped candidates in 100-row keyset pages and includes + future coordinator leases. Execution checks the current row before dispatch. +- Worker batches contain one physical cleanup. Duplicate attempts join one promise; + different runs serialize through the same database lane. This preserves bounded + inspection and avoids mistaking another cleanup's brief session reservation for + a long-lived owner requiring a future release event. +- A coordinator lease uses its saved expiry. `process_exit` retains one 30-second + probe for that concrete candidate. This exception is for a retained process + lacking a local exit callback, not idle-history polling; every attempt rechecks + all authority and physical evidence. It stops when the candidate settles, + becomes ineligible/operator-required, is deleted, or changes to an event wait. +- Other deferred outcomes drop their scheduled entry until an owner release or + committed mutation reactivates discovery. **Those production release producers + must be installed and tested before periodic discovery is disabled.** +- Local release subscriptions are scoped to the database owner and are explicitly + removed on shutdown. Coordinator shutdown waits for its joined operation. +- No startup registration or periodic skip flag has been added yet. Cloud alarm + coverage and conservative sleep blockers remain unchanged. + +Evidence on this source checkpoint: + +- Initial worker tests: 12 passed, `/tmp/sleep51-worker-first.log` / `.status` (0). +- Switching the first post-idle writer to the existing finalization notification + helper reproduced a missed cleanup wake: `/tmp/sleep51-finalization-signal-before.log` + / `.status` (1), one failed regression with 16 cases deliberately unselected. + Adding cleanup to that existing fanout repaired it. No new transaction wrapper + was introduced. +- The expanded selection passes 18 cases, then 19 after adding a 103-item backlog. + `/tmp/sleep51-worker-second.log` and `/tmp/sleep51-worker-final.log`, status 0. + The final suite covers empty-to-active-to-quiet, real physical refusal with + retained history, named release signals, restart reconstruction, exact lease + expiry, early release, cancellation, concrete process probing, outer rollback, + first write during drain, failed work across drain, paused startup, discovery + failure, a committed wake during cleanup, duplicate joining, serialization, + shutdown, and completed history across discovery pages. +- The release-signal tests exercise the notification boundary, not all real session, + environment and maintenance producers. The main first-writer test exercises + the existing finalization notification helper inside a real transaction, but + the complete domain-writer audit is still pending. +- Existing manual/startup cleanup and finalization: all 36 cases pass on a fresh + database, `/tmp/sleep51-worker-legacy.log` / `.status` (0). +- Existing reassessment worker: all 37 lifecycle/producer cases pass on a separate + fresh database after extending finalization fanout, + `/tmp/sleep51-worker-reassessment.log` / `.status` (0). +- Server typecheck passes: `/tmp/sleep51-worker-types-final.log` / `.status` (0). + Total current selection: 92 passing cases. Earlier physical-operation and broader + executor results remain recorded above; they are not presented as rerun here. + +The 103-item test proves paged reconstruction, actual bounded draining and SQL +quiet. It is not the outstanding large-history/candidate cost qualification. +Remaining work includes every actual producer/release path, operational ownership +through retained callbacks, startup integration, periodic removal, full PR checks +and the rest of the recurring-server stack. No new PR or merge is claimed. + + +## Durable owner release checkpoint + +The environment service owns lease release and its existing transaction wrapper. +It now signals the affected run before single/run-wide release commits. Metadata +updates and still-pending cleanup do not activate retained-session cleanup. The +run-wide API resolves company scope from its authoritative run rather than an +unscoped notification. It has no production call sites in the inspected checkout; +its public service behavior remains covered. + +Three direct settlement owners also bypass ordinary lease release: stop-only +cleanup, workspace-export-resume settlement, and confirmed sandbox absence. Each +now registers the same scoped intent before the status write. The first two reuse +their transaction; absence settlement uses a short transaction that composes with +a caller-owned transaction. No provider I/O enters those transactions. Existing +receipt validation, ownership and compare-and-set predicates are unchanged. +Verified settled-process release signals before clearing its mutable identity. + +The environment writer audit found no production raw-SQL lease updates/deletes +or direct lease deletes in server/src. Other direct writers either retain/add +active or pending-cleanup ownership, change metadata/lease policy, or retire an +already nonblocking lease. In particular, environment-task-runtime completion +only changes retained -> released; retained is already excluded by this cleanup's +active/pending-cleanup predicate. Reusable replacement also starts from released +or retained. These transitions do not unblock this worker. This inventory does +not finish the separate run/finalization writer or local-owner audit. + +Evidence: + +- `/tmp/sleep51-env-before.log` / `.status` (1): both real single and run-wide + release fail to reactivate cleanup. `/tmp/sleep51-env-after.log` / `.status` + (0): all 21 lifecycle cases pass after repair. +- `/tmp/sleep51-process-before.log` / `.status` (1): verified process release + fails to reactivate before the existing process probe. The process stop check + is controlled in this integration test; database, receipt writer and service + transactions are real. +- `/tmp/sleep51-release-after.log` / `.status` (1): the initial process rollback + test failed to await the coordinator's rollback reconciliation before opening + the next transaction, then contaminated the following deadline assertion. + The corrected barrier waits for that reconciliation and verifies the persisted + original PID. `/tmp/sleep51-release-final.log` / `.status` (0): 23 cases pass. +- `/tmp/sleep51-settlement-before.log` / `.status` (1): all three direct settlement + producers reproduce the missing wake after commit. The absence test supplies + already verified typed proof at the settlement boundary; it does not claim + Cloud receipt signature/provider qualification. +- `/tmp/sleep51-settlement-after.log` / `.status` (0): all 26 lifecycle cases pass. + These prove outer rollback, no delivery before outer commit, release activation, + successful settlement and subsequent SQL quiet. Failed cleanup, metadata-only + updates and stale attempt IDs do not prematurely release the wait. +- `/tmp/sleep51-env-consumer.log` / `.status` (0): all 51 existing environment + service cases pass on a fresh isolated database. +- `/tmp/sleep51-settlement-consumers.log` / `.status` (0): all 131 existing legacy + restore/native failed-workspace recovery cases pass. +- `/tmp/sleep51-settlement-types.log` / `.status` (0): server typecheck passes. + +This checkpoint covers 208 selected cases. Actual process exit and physical +cleanup safety tests from previous checkpoints remain distinct evidence. Session +owner release, maintenance completion, full run/finalization producer coverage, +physical callback ownership, startup wiring, large-history cost, periodic removal, +publication and full PR acceptance are still open. No polling removal is claimed. + + +## Local session-owner release checkpoint + +The executor now signals company/agent scope after its execution scope and +startup ownership settle, and after backend initialization relinquishes its claim. +An older retained run can share the physical session with a newer run, so a +run-only hint would miss supported waiters. Durable candidate discovery and all +physical authority checks still determine whether cleanup can proceed. + +Warm-session removal is centralized in `forgetWarmNativeSession`. It checks the +expected entry identity, removes that entry, then invokes its captured release +callback. All existing retirement paths use this owner boundary, including idle +expiry, explicit shutdown, configuration/instruction changes and quarantine. +Existing failed-close retention rules remain; a late close cannot delete a newer +entry. The callback holds the database scope without adding database fields to +the warm-session record or teaching the scheduler provider behavior. Cleanup's +own temporary reservation release deliberately does not emit this hint, avoiding +self-reactivation on every inspection of retained history. + +Three real executor lifecycle regressions first failed with zero release hints: +both startup cancellation variants and warm idle retirement. The strengthened +warm test holds its original close promise unresolved and proves no release hint +until that promise settles. These tests use the existing controlled executor/DB +fixtures. They test actual owner paths; the separate PostgreSQL worker suite +proves the notification-to-reconciliation lifecycle. They are not a single live +provider-to-database end-to-end qualification. + +- `/tmp/sleep51-local-release-before.log` / `.status` (1): 3 expected failures, + 810 deliberately unselected cases. +- `/tmp/sleep51-local-release-after.log` / `.status` (0): 798 executor cases pass, + 15 embedded-PostgreSQL remote warm ownership cases excluded because of the + previously diagnosed host semaphore limitation. +- `/tmp/sleep51-local-release-types.log` / `.status` (0): server types pass. +- `/tmp/sleep51-local-release-worker.log` / `.status` (0): 26 real-database worker + lifecycle cases pass after local producer wiring. +- `/tmp/sleep51-local-release-final.log` / `.status` (0): final 798-case executor + rerun passes after removing duplicate entry guards from callers. The same 15 + embedded-database cases remain excluded, not reported as passing. + +Next: retained maintenance still has a separate lifetime. Runner tracks active +roots plus original bounded callbacks, but exposes only idle inspection and an +all-roots drain. A timed-out callback can still own its root. The scheduler needs +an owned completion signal for that concrete wait; heartbeat's existing outer +sweep drain must move inside the common physical-operation lifetime so scheduled, +manual and shutdown callers join the original work. Do not clear that obligation +on timeout or add a recurring maintenance-state poll. Runner must expose its own +completion contract; it must not import server notification policy. + +Remaining finalization writer audit, maintenance ownership, initialization-path +reactivation proof, startup integration, large-history cost, periodic removal, +publication, and complete PR gates still stand. This slice remains unpublished; +periodic cleanup remains enabled and the full stack remains unmerged. + + +## Original maintenance ownership and terminal projection checkpoint + +Runner's existing drain now joins active maintenance attempts as well as their +original callbacks. A root-specific promise resolves only when neither remains; +the global drain repeats discovery if another operation was registered meanwhile. +This is local ownership settlement, not physical retirement or launch permission. +The root waiter stays inside the Runner transport module; the server consumes the +existing public drain contract. No server policy is imported into Runner. + +The physical cleanup entry point now joins that drain in its own finally block, +including on error. Manual, startup and future scheduled callers therefore own +the complete operation. A maintenance-deferred result emits its scoped release +hint only after the original callbacks settle. Heartbeat's sweep no longer owns +a separate trailing drain. Existing timeout/quarantine and physical proof rules +remain unchanged; an indefinitely unresolved original callback retains ownership. + +Two more producer boundaries were missing cleanup intent: finalizer run +projection (both committed-result repair and ordinary post-decision projection), +and heartbeat's shared terminal run-status writer. They now signal within their +existing transactions, including terminal metadata repairs. Running-progress +writes do not wake this duty. The ordinary finalizer signal is unconditional in +its projection transaction because metadata may make a previously terminal run +eligible even when projectRunStatus is false. + +Real PostgreSQL regressions cover outer commit, rollback, and a deterministic +separate-commit race: delay the finalizer after its actual decision commit, allow +the worker to consume that notification while the run is still running, then +permit terminal projection. Before the new signal cleanup is stranded; afterward +it runs once. No finalizer/provider decision logic was mocked for this race. + +Evidence (all /tmp paths have matching .status files): + +- sleep51-maintenance-before: invalid root Vitest selection, no matching Runner + tests; not regression evidence. Correct package invocation + sleep51-maintenance-before-runner failed the missing completion API test. +- sleep51-maintenance-after: all 19 Runner settlement cases pass, including + timeout/abort ownership, unrelated-root idleness and unresolved original callbacks. +- sleep51-owned-maintenance-before: both new physical lifetime tests failed; + sleep51-owned-maintenance-after passes 799 executor cases. The 15 embedded-DB + remote warm cases remain excluded for the diagnosed host semaphore limit. +- sleep51-maintenance-types initially failed the test mock's Promise + inference; an explicit Promise fixes it. maintenance-types-final passes. +- sleep51-maintenance-worker: all 26 existing real-DB worker cases pass. +- sleep51-projection-before: committed-finalizer producer regression failed. +- sleep51-lifecycle-before: both terminal writer variants failed; running progress + remained quiet. The repaired sleep51-projection-after passes all 64 worker and + heartbeat lifecycle cases, including outer rollback and terminal metadata repair. +- sleep51-corpus-projection-before: the separate-commit race failed with zero + cleanup calls after the terminal projection. sleep51-corpus-projection-after + passes the entire 146-case finalizer corpus, including that regression. +- sleep51-projection-types: server typecheck passes. + +Fresh origin/master remains 1fbbef45a8d399771add7a27a04b12f3a20e264c. +Remaining: final producer/initialization audit, startup integration, history/query +cost, periodic removal, broader acceptance and PR gates. Nothing is published or +merged for slice 51; periodic cleanup is still enabled at this checkpoint. + + +## Startup integration and full executor verification + +The enabled server path now registers the scoped worker with the shared delivery +coordinator. Its admission predicate includes warm standby, task drain and server +shutdown. Server shutdown removes local release subscriptions before stopping the +coordinator. Heartbeat also tracks each original physical operation, so its direct +execution drain joins scheduled cleanup independently of coordinator shutdown. +The periodic orphan pass skips only retained-session cleanup. Startup recovery +and explicit/manual recovery still call the bounded legacy sweep; all paths share +the per-database lane and join identical run operations. + +The startup and assembled-heartbeat regressions first failed because registration +was absent. After wiring, all 100 selected startup/worker tests and server types +pass (/tmp/sleep51-startup-{before,after,types}.{log,status}). The assembled test +holds the original cleanup promise and proves heartbeat's drain cannot finish +before it settles. Existing worker tests cover drain/resume, error/retry, rollback, +concurrent hints and saved lease deadlines. + +The user explicitly approved removal of 11 named orphaned PostgreSQL semaphore +sets. Each exact ID/key and its recorded dead PIDs was rechecked; the corresponding +shared-memory segment was absent. Only those 11 approved sets were removed. +The complete executor suite then passed all 814 tests, with no exclusions +(/tmp/sleep51-executor-full.log and .status). This supersedes the earlier 15-case +host limitation for this suite; it does not erase the earlier failed setup evidence. +The four overlapping initialization cases now also prove no cleanup release hint +until both original preparation operations settle, on success and failure. + +The query-cost probe cloned 10,000 settled retained sessions, with real result, +assessment, decision and finalization foreign keys, in an isolated database. +EXPLAIN ANALYZE measured startup at 29.764 ms/62,266 shared-hit blocks, company +scope at 27.731 ms/63,190 blocks, agent-release scope at 27.672 ms/63,190 blocks, +and run scope at 0.030 ms/6 blocks. All return zero candidates. Broad discovery +still grows with retained history; these local warm-cache measurements are not a +fleet latency guarantee. It runs on startup or a relevant scope change, not on a +recurring idle scan. The separate 103-candidate test covers paging and settlement. +No new index is claimed necessary or added from this bounded measurement. + +Probe source: /tmp/sleep51-cost-probe.mts. Plans: /tmp/sleep51-cost-final.log +(status 0). Initial root invocation failed because tsx is package-local; the +corrected invocation then exposed a copied supersedes-assessment FK from the +fixture. The seed transaction rolled back. Clearing that copied predecessor link +produced the successful realistic-history fixture; no constraint was disabled. + +Broader workspace verification, publication and all remote CI/review gates remain. +The four older stack gates were refreshed and still need repair: #17490/#17572 +E2E shard 3, #17576 security/review findings, and #17580 review findings. +Nothing in this stack is merged or authorized for merge. + + +## Published PR and CI repair checkpoint + +Draft [#17717](https://github.com/paperclipai/paperclip/pull/17717) was published +at `8c62a6b0c0c381326ab94fd5c75c78e0dd426e1a`, based on #17709. Workspace +typecheck and build passed there. Greptile reviewed that exact head at 5/5 with +no actionable findings. CI run `40323191795` found three failures: + +- The heartbeat recovery module test still mocked cleanup through the old barrel + import and expected heartbeat to drain Runner maintenance itself. It now mocks + the extracted cleanup owner and holds its operation promise. The complete + 34-case module suite passes after correcting the mock target. +- The assembled heartbeat process-recovery test also expected the old outer + drain call. Its controlled physical cleanup now retains its original callback + inside the cleanup operation, matching the executor contract. It still proves + startup/reaping stay responsive, duplicate cleanup joins, shutdown stays + non-quiescent, no provider turn starts, and persisted results remain unchanged. + The complete 408-case process-recovery file passes. Actual Runner callback + settlement remains covered in the executor and Runner suites, rather than + asserted through this controlled boundary alone. +- The Copilot source attestation still bound the old transport bytes. Preserve + the v58 fixture and add v59 using `copilot-profile-identity.mjs`; regenerate + TypeScript/Rust admission from the shared manifest. Historical v58 remains + decodable, but exact launch admission uses the new identity. The identity also + catches inherited source drift in native runtime instructions, native-session + runtime and transport contracts. No source changes to those three files are + included here. All 69 installation/compatibility cases pass, including rejection + of the old launch identity and preservation of its history. This is source + identity verification, not a claim of new live Copilot qualification. + +Local repair evidence preserves unsuccessful attempts: the first module repair +still targeted the old import; the initial identity update retained an old digest +literal; an overbroad replacement changed an unrelated historical expectation and +was restored before acceptance. The first isolated Runner settlement attempt had +no built `paperclip-runnerd` and failed all 19 cases at setup; the new checkout +must build that binary before those tests count as executed. + +The full default local suite at the published head remains separate. It reported +an OAuth scope/catalog-refresh test timing out at 15 seconds. A separate complete +472-case tool-access suite passed with a disposable PostgreSQL database under the +same host load. This rerun does not erase the original full-suite failure or +establish that its whole command passed. The original checkout remains unchanged +while that command runs. Latest-head remote gates, complete suite disposition, +and repaired-head review remain required before acceptance. + + +## Master integration + +Master advanced the source-bound Copilot profile to v63 while this branch +published a different v62 identity for retained maintenance settlement. The +combined runtime preserves master's uncertain-process and controller-retirement +behavior plus this worker's event-driven idle waiters. Its generated profile is +v64. Master's v62/v63 fixtures stay byte-for-byte unchanged; the stack's former +v62 is retained separately as `copilot-profile-v62-sleep-maintenance-historical-identity.json`. +Tests verify their checksums and exact changed source fields, historical decoding, +and refusal to launch any historical identity through current installation +admission. TypeScript and Rust declarations are generated together. This is source +identity validation, not a claim of live provider qualification. + +### Master integration follow-up (2026-10-27) + +Pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` advances Copilot to v65. +The combined transport retains its exhausted-reconnect cancellation behavior and +this duty's per-root maintenance-idle completion notifications. The generated +combined identity is v66; only the version, compatibility source, and runtime +request bridge differ from master's v65 declaration. Master's v64/v65 fixtures +remain unchanged. The stack's different former v64 is preserved as +`copilot-profile-v64-sleep-maintenance-historical-identity.json` alongside its +former v62 evidence. Installation tests retain both histories and prove historical +profiles remain decodable but cannot launch as the current pinned profile. +The profile generator's check, TypeScript/Rust agreement, and runner typecheck +pass. This remains source identity verification, not live provider qualification. + +## Merge acceptance on current master (2026-10-29) + +This section supersedes earlier provisional enablement and readiness notes. The +reviewed parent #17709 is merged. Its exact reviewed tree is unchanged in the +landed squash. A normal merge adopts that ancestry without altering this child’s +complete tree. This slice is now prepared directly for master. + +Routine company-only finalization hints do not reconstruct retained cleanup. +Ordinary fanout requires an issue or run identity; pending-effects serialization +retains its existing company/run contract. Cleanup producers allocate their run +identity before registering transaction intent and before writing work. The +local release path can still wake a dormant candidate. A controlled regression +removed only the scope guard and failed; restoring that guard passed the same +assertions. The complete 29-case worker file also passes, including durable +reconstruction, commit/rollback, drain/resume, changes during execution, physical +owner settlement and quiet retained history. + +Current parent integration preserved 12,693 independent parent entries, 14 +owned-only entries and 15 automatic shared entries. Ten manually reviewed paths +retain both runtime histories and both sets of installation assertions. All 92 +parent historical fixtures remain byte-exact. The sleep branch’s different +profile 66 is preserved under a separate historical filename, alongside its +prior 62 and 64 variants. Forty added code comments were removed with identical +nontrivia TypeScript token streams before generating source identities. + +Local server acceptance covers **2,342 distinct passing cases in 18 complete +files**. This includes 977 native executor cases, 146 arbiter corpus cases, 410 +process-recovery cases, 273 external-chat response cases, 76 startup cases and +29 cleanup worker cases. Production server typechecking, build, module boundaries, +migration order and profile generator checks pass on the stated source. Existing +SQL and the database package tree are unchanged from parent migration 413. +Disposable PostgreSQL 17 fixtures were prepared independently. Temporary imports +use the existing fresh-per-call recovery fixture helper for embedded-only paths; +every import was restored byte-for-byte. No test assertion or timeout was relaxed. + +The first 15-file invocation reported 1,607 passes and one startup shutdown failure, +plus its associated unhandled rejection. The shared coordinator test double lacked +the real worker’s `stop()` method, which task-drain shutdown now calls. Only that +test double changed. All 76 unchanged startup cases then passed in 6.17 seconds. +The other 14 complete files account for 1,532 passes. The remaining three complete +files passed separately: process recovery 410, environment service 51 and external +chat response 273. The failed log remains part of the acceptance record; the first +whole invocation is not reported as a successful command. + +Master then advanced independently with Claude ACP reconnect replay and a lockfile +refresh. Its server, UI and database trees are unchanged. Its published Copilot +profile 79 differs from this branch’s prepared 79. The merge preserves master’s +canonical fixture and saves the sleep variant as +`copilot-profile-v79-sleep-maintenance-historical-identity.json`. Combined profile +80 binds current compatibility decoding and retained-maintenance completion; +other providers retain master’s declarations. Historical identities remain +readable and fail current launch admission. This is source identity verification, +not live Copilot qualification. + +Current integration validation and exact review/CI results are recorded in the +PR. Local evidence proves named worker behavior. It does not establish full +repository acceptance, Cloud alarm coverage, deployed host/database suspension +or measured cost savings. Concrete live-PID absence probes and durable lease +expiries remain real obligations; this change does not discard them to claim quiet. + +The current profile 80 runtime selection passes 130 cases across four files in +125.90 seconds. One existing opt-in closed Pi snapshot case is skipped because +`PAPERCLIP_TEST_PI_DISTRIBUTION_ROOT` is not set. This skip is not counted as +acceptance of a live Pi installation. The Copilot installation/history and +transport-settlement files have no skipped cases. + +Current integration at `d08f764e3efe75e96929173afe7ee8739121ad9f` passes +all 12 validation commands: frozen install, three source/profile checks, Runner +and server typechecking, server build, the four-file runtime selection, installed +ACPX/Grok contract tests (22 passes), native-provider Rust tests (26 passes), +module boundaries and migration order. The live-provider limits above remain. diff --git a/packages/paperclip-runner/acpx-profiles.json b/packages/paperclip-runner/acpx-profiles.json index d9fb690f000..2d9ab64d1cb 100644 --- a/packages/paperclip-runner/acpx-profiles.json +++ b/packages/paperclip-runner/acpx-profiles.json @@ -32,12 +32,12 @@ "requiresProviderPolicy": true }, "copilot": { - "agentProfileVersion": 79, + "agentProfileVersion": 80, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:a0e9f339935f6ee616fee644c45cb09353e9aed1d1f22215739f9504153ceca1", + "commandDigest": "sha256:7b8a6a31925256d225ea3d72fdcfd81b70a45c4b67d3eb0ee5073efff9ac8f8b", "requiresProviderPolicy": true }, "claude": { diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs index f61400ce917..46218f0548b 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs @@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: None, agent_runtime_version: None, command_digest: - "sha256:a0e9f339935f6ee616fee644c45cb09353e9aed1d1f22215739f9504153ceca1", + "sha256:7b8a6a31925256d225ea3d72fdcfd81b70a45c4b67d3eb0ee5073efff9ac8f8b", requires_provider_policy: true, }, "claude" => AcpxReleaseProfile { diff --git a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs index c92ee2ad192..98a99aaf120 100644 --- a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs +++ b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs @@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url"; import { resolve, dirname } from "node:path"; const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); -const fixture = resolve(root, "test/fixtures/copilot-profile-v79-identity.json"); +const fixture = resolve(root, "test/fixtures/copilot-profile-v80-identity.json"); const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8")); -const declaration = { ...historical.declaration, agentProfileVersion: 79 }; +const declaration = { ...historical.declaration, agentProfileVersion: 80 }; delete declaration.permissionIdentitySourceSha256; const sources = { displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts", diff --git a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts index 82812032921..92b05070d37 100644 --- a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts @@ -87,9 +87,71 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi describe("Copilot build-owned installation", () => { it("admits the current profile with environment launch preparation", () => { - expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(79); + expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(80); expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined(); - expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:a0e9f339935f6ee616fee644c45cb09353e9aed1d1f22215739f9504153ceca1"); + expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:7b8a6a31925256d225ea3d72fdcfd81b70a45c4b67d3eb0ee5073efff9ac8f8b"); + }); + +it("preserves published v65 while binding retained-maintenance waits in v66", async () => { + const priorBytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v65-identity.json", import.meta.url)); + expect(createHash("sha256").update(priorBytes).digest("hex")).toBe("ca1d95c21b3f49c99b6d7c6bd3c2bf149715db3263927ad653cc0c6f33fb4462"); + const prior = JSON.parse(priorBytes.toString("utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v66-sleep-maintenance-historical-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256", + ]); + expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort()); + expect(isSupportedAcpxProfileVersion("copilot", 65)).toBe(true); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: 65, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); + }); + +it("preserves the historical sleep-maintenance v64 identity", async () => { + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v64-sleep-maintenance-historical-identity.json", import.meta.url), "utf8")); + for (const [name, checksum, changed] of [ + ["copilot-profile-v63-identity.json", "4adb05bd642f0222a3bcc85cfefbeb2988de439f58125012e1c575caedd90798", ["agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256"]], + ["copilot-profile-v62-sleep-maintenance-historical-identity.json", "e0c93dd7210fbb0115e09b70887ec4ae40621fc5314bd1625e586caf2163c94e", ["agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256", "semanticCompletionContractSourceSha256", "semanticContinuationSourceSha256"]], + ] as const) { + const bytes = readFileSync(new URL(`../../../test/fixtures/${name}`, import.meta.url)); + expect(createHash("sha256").update(bytes).digest("hex")).toBe(checksum); + const prior = JSON.parse(bytes.toString("utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual(changed); + expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort()); + const sorted = Object.fromEntries(Object.entries(prior.declaration).sort(([a], [b]) => a.localeCompare(b))); + expect(`sha256:${createHash("sha256").update(JSON.stringify(sorted)).digest("hex")}`).toBe(prior.commandDigest); + expect(isSupportedAcpxProfileVersion("copilot", prior.declaration.agentProfileVersion)).toBe(true); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: prior.declaration.agentProfileVersion, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); + } + }); + + it("preserves profile 78 while binding retained-session waits in profile 79", async () => { + const bytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v78-identity.json", import.meta.url)); + expect(createHash("sha256").update(bytes).digest("hex")).toBe("da2d7d33dbcdb900f3afb0ad0d0d1502c31f7638772a58009a3300f7e442fd1c"); + const prior = JSON.parse(bytes.toString("utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v79-sleep-maintenance-historical-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256", + ]); + expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort()); + expect(isSupportedAcpxProfileVersion("copilot", 78)).toBe(true); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: 78, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); + }); + + + it("preserves published profile 79 while binding retained cleanup in profile 80", async () => { + const bytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v79-identity.json", import.meta.url)); + expect(createHash("sha256").update(bytes).digest("hex")).toBe("92fb6b5f1a312933f78d650fdc037ded4876d112c472fcfd5c1116cdbb0dae44"); + const prior = JSON.parse(bytes.toString("utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v80-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256", + ]); + expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort()); + expect(isSupportedAcpxProfileVersion("copilot", 79)).toBe(true); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: 79, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); }); it("retains the published profile 75 fixture while advancing the event contract", async () => { diff --git a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts index 4512b8cfb92..60a667c3f5a 100644 --- a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts +++ b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts @@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "copilot", - "agentProfileVersion": 79, + "agentProfileVersion": 80, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:a0e9f339935f6ee616fee644c45cb09353e9aed1d1f22215739f9504153ceca1", + "commandDigest": "sha256:7b8a6a31925256d225ea3d72fdcfd81b70a45c4b67d3eb0ee5073efff9ac8f8b", "permissionPolicy": "interactive" }, "claude": { diff --git a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts index 15f892b66c5..21eaeedd6d6 100644 --- a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts +++ b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts @@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js"; /** Decodable historical profile revisions, not permission to launch them. * Exact current profile/package/digest admission is checked separately. */ -type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78; +type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79; type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"]; export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion; const historicalVersions: Readonly> = { pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], - grok: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78], + grok: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79], cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], }; diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport-settlement.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport-settlement.test.ts index 826d5c466fc..e776729200f 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport-settlement.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport-settlement.test.ts @@ -46,6 +46,7 @@ import { settleRetainedRunnerdSession, retainedRunnerdCleanupProofIsCurrent, retainedRunnerdMaintenanceIsIdle, + waitForRetainedRunnerdMaintenanceIdle, } from "./runnerd-codex-transport.js"; // Explicit private-artifact test lane; production/default dist is never changed. @@ -605,6 +606,11 @@ it.each([ authorize: () => stuckAuthorization, }); const observed = blocked.catch((error: unknown) => error); + let idle = false; + const idleWait = waitForRetainedRunnerdMaintenanceIdle(copy).then(() => { idle = true; }); + + await waitForRetainedRunnerdMaintenanceIdle(join(copy, "unrelated")); + expect(idle).toBe(false); if (interruption === "abort") abort.abort(); else await vi.advanceTimersByTimeAsync(30_000); expect(await observed).toMatchObject({ @@ -623,9 +629,11 @@ it.each([ // The bounded wrapper has already failed, but its original callback // remains owned until it actually settles. No retry/proof is granted. expect(drained).toBe(false); + expect(idle).toBe(false); releaseAuthorization(); - await drain; + await Promise.all([drain, idleWait]); expect(drained).toBe(true); + expect(idle).toBe(true); expect(retainedRunnerdMaintenanceIsIdle(copy)).toBe(true); } finally { releaseAuthorization(); diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index 912fb0de809..beeb607a8f0 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -2095,6 +2095,7 @@ export type RetainedRunnerdMaintenanceEpochReceipt = { const retainedRunnerdCleanupProofs = new WeakMap(); const retainedMaintenanceOperations = new Map>>(); const activeMaintenanceRoots = new Set(); +const maintenanceIdleWaiters = new Map; resolve: () => void }>(); /** No absence claim about an old child; only this controller's joined work. */ export function retainedRunnerdMaintenanceIsIdle(directory: string): boolean { @@ -2102,16 +2103,33 @@ export function retainedRunnerdMaintenanceIsIdle(directory: string): boolean { return !activeMaintenanceRoots.has(root) && !retainedMaintenanceOperations.has(root); } + + +export function waitForRetainedRunnerdMaintenanceIdle(directory: string): Promise { + const root = resolve(directory); + if (retainedRunnerdMaintenanceIsIdle(root)) return Promise.resolve(); + const existing = maintenanceIdleWaiters.get(root); + if (existing) return existing.promise; + let release!: () => void; + const promise = new Promise(done => { release = done; }); + maintenanceIdleWaiters.set(root, { promise, resolve: release }); + return promise; +} + +function releaseMaintenanceIdleWaiters(root: string) { + if (!retainedRunnerdMaintenanceIsIdle(root)) return; + const waiting = maintenanceIdleWaiters.get(root); + maintenanceIdleWaiters.delete(root); + waiting?.resolve(); +} + /** A timeout revokes cleanup authority, not ownership of an already-started * callback. Shutdown must join the original operations, including any that * another retained callback registers while the current snapshot settles. */ export async function drainRetainedRunnerdMaintenanceOperations(): Promise { - while (retainedMaintenanceOperations.size > 0) { - await Promise.allSettled( - [...retainedMaintenanceOperations.values()].flatMap((operations) => [ - ...operations, - ]), - ); + while (activeMaintenanceRoots.size > 0 || retainedMaintenanceOperations.size > 0) { + const roots = new Set([...activeMaintenanceRoots, ...retainedMaintenanceOperations.keys()]); + await Promise.all([...roots].map(waitForRetainedRunnerdMaintenanceIdle)); } } @@ -2358,6 +2376,7 @@ export async function settleRetainedRunnerdSession( return await settleRetainedRunnerdSessionOwned(input); } finally { activeMaintenanceRoots.delete(root); + releaseMaintenanceIdleWaiters(root); } } @@ -2458,6 +2477,7 @@ async function settleRetainedRunnerdSessionOwned( retainedMaintenanceOperations.get(root) === retained ) retainedMaintenanceOperations.delete(root); + releaseMaintenanceIdleWaiters(root); }; void operation.then(released, released); let timer: ReturnType | undefined; diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v62-sleep-maintenance-historical-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v62-sleep-maintenance-historical-identity.json new file mode 100644 index 00000000000..504c1c450b4 --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v62-sleep-maintenance-historical-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:4c5b7cd84ddbb5375ae044012a87391156633e309f60aba3a99acc80edaee2e0", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 62, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "3343df6362252d44d12b079ff78c8f6539d1544f3ce5ec36ff36db70ab0fac31", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "cd23577bfee9ab95bb8a177d1948917e96a7bf6245f320263fb6526ebf5a76cf", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918", + "profileCompatibilitySourceSha256": "be60656e66d7f61a290f8d1c9a3c9ec511fb13b6541825a087b470e971ccc020", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "d248e47a0fe7f2536d1e64cea0b0b410f34104800cd643d995b4473104ecc8e0", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v64-sleep-maintenance-historical-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v64-sleep-maintenance-historical-identity.json new file mode 100644 index 00000000000..c765291d15f --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v64-sleep-maintenance-historical-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:83f8a5bfc7a619c72a2b7bee9e2ce15a0f779f4b483253186b590386186ac76d", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 64, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "ccb98a1696366c67518a6be33d69c9e5e00324a74dee294193a6ba5d536ed53f", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918", + "profileCompatibilitySourceSha256": "2f875a2f39f4e99a3d03ffce9f325ce2461c51c6b8cb70218ce89ec008eaa5b8", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v66-sleep-maintenance-historical-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v66-sleep-maintenance-historical-identity.json new file mode 100644 index 00000000000..423589c1057 --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v66-sleep-maintenance-historical-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:ee0e229a3db5849b5f5528489a44e4450939d9854d6ef98d29a48611acf250f5", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 66, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "cda99b35c428c4189c116c62b34349fc92e6f2de951d58847197b2d17bdb57de", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918", + "profileCompatibilitySourceSha256": "d96c3b809d083f9d43936af05e6bc14b1755c55f255c8e5967b8c16661f34ffa", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v79-sleep-maintenance-historical-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v79-sleep-maintenance-historical-identity.json new file mode 100644 index 00000000000..8c23ab768f7 --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v79-sleep-maintenance-historical-identity.json @@ -0,0 +1,88 @@ +{ + "commandDigest": "sha256:9934065aaee959a19f05e102d9cf6fc41288c15be18f2769f62af6d7cc8a25b7", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 79, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "7d6e948b7cff4df52a9afc916db46ace0f42b4b2d3134ba0c21ee0a928857e62", + "semanticDirectDriverSourceSha256": "d683429427dd8a1a5efe50952018ea486f808bdbf0a92fc5946ee267fe2c72ee", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "b20ec8ee6e7165117233ce4421a975bb9d3f276220ce31655c0b9a717cb2d9cb", + "semanticSchemaBundleSourceSha256": "81ef98ca7b34896a0f76390ac7af3dd8ef7609fe6f38d54bd6a69373d50fe40b", + "semanticSidecarProtocolSourceSha256": "26dd0cfbef8b2b401de6b7862a4af7f243fb703e23678a1a5ed8fa68d4784d42", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "baba1a210b863518f01e0c875606ed37f9467c433222251438578bd999bc43b5", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "7495d9898d22dca1ebf6336361d332bafefc4498f6e49f1135025fad846c8586", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "82ac6e5f7275b4ef79fb166c7330efc504e80cab3d7e9081e4e40a8b570bc50b", + "providerLaunchSourceSha256": "c744235b3067ea75973523f78f1e13399337bc155f1775af694e34782d5c9bef", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "2d167741defe23d6b2b0647a54405331ae4457d298936f0796733628ad79d7bc", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "d60b8ea63bad975c0144f0702b2749d7db9a7da50680a6147f20d4d11f2d5297", + "profileCompatibilitySourceSha256": "9000cf5a7f4191cf28699d4deb23e17cd02507a3b2f12c12a7051a0fea782496", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "3a93267146bfc3c489f43262386d6dcb05ec4e3a8b1765414cd43d80d0c64d4a", + "admissionDiagnosticsSourceSha256": "df8f1aa4f2d1bc63b2635bdd09cb8165c8b7681ddcc2618435c7c08ca90475af", + "grokAdmissionSourceSha256": "f1e46cf2e1e36b7704bd797b678e3db8db2003f97bdb9d827cfa3d7662698777", + "sidecarErrorSourceSha256": "17948ad74350dc487ab20d24a0fb89daf0b426b98396454fff0e3856e922d82c", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "preparedHarnessHomeSourceSha256": "bb286f7ddc6e9597c56d46bf7ff6e3b556c5b680d7e147fd46518724432b083e", + "runtimeHostSourceSha256": "13f9ac382a279a67adb8974c36bbed38fe43d9b93b87bd7a05e879ead7f03d63", + "runtimeAdapterSourceSha256": "7c47c05306ebe7d1ca6b5d046a96f5726a48adec4d27e7c16788f505f25b5db2", + "runtimeSandboxSourceSha256": "04a49a24e3828ce8fca908de15e31ae932b8b1b4f7ecec0308627f6a8da9ba20", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v80-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v80-identity.json new file mode 100644 index 00000000000..60861aa2a95 --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v80-identity.json @@ -0,0 +1,88 @@ +{ + "commandDigest": "sha256:7b8a6a31925256d225ea3d72fdcfd81b70a45c4b67d3eb0ee5073efff9ac8f8b", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 80, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "1e7dae8c83bf8620f53f2bea13c175eeff95e7e955783f6f529bb1dcbdcdb6d2", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "7d6e948b7cff4df52a9afc916db46ace0f42b4b2d3134ba0c21ee0a928857e62", + "semanticDirectDriverSourceSha256": "d683429427dd8a1a5efe50952018ea486f808bdbf0a92fc5946ee267fe2c72ee", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "b20ec8ee6e7165117233ce4421a975bb9d3f276220ce31655c0b9a717cb2d9cb", + "semanticSchemaBundleSourceSha256": "81ef98ca7b34896a0f76390ac7af3dd8ef7609fe6f38d54bd6a69373d50fe40b", + "semanticSidecarProtocolSourceSha256": "26dd0cfbef8b2b401de6b7862a4af7f243fb703e23678a1a5ed8fa68d4784d42", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "baba1a210b863518f01e0c875606ed37f9467c433222251438578bd999bc43b5", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "7495d9898d22dca1ebf6336361d332bafefc4498f6e49f1135025fad846c8586", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "82ac6e5f7275b4ef79fb166c7330efc504e80cab3d7e9081e4e40a8b570bc50b", + "providerLaunchSourceSha256": "c744235b3067ea75973523f78f1e13399337bc155f1775af694e34782d5c9bef", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "2d167741defe23d6b2b0647a54405331ae4457d298936f0796733628ad79d7bc", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "d60b8ea63bad975c0144f0702b2749d7db9a7da50680a6147f20d4d11f2d5297", + "profileCompatibilitySourceSha256": "c3bde2939555d6826b983928735d5a2e3d2949e4235d5910de16a1abf80ad7e9", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "3a93267146bfc3c489f43262386d6dcb05ec4e3a8b1765414cd43d80d0c64d4a", + "admissionDiagnosticsSourceSha256": "df8f1aa4f2d1bc63b2635bdd09cb8165c8b7681ddcc2618435c7c08ca90475af", + "grokAdmissionSourceSha256": "f1e46cf2e1e36b7704bd797b678e3db8db2003f97bdb9d827cfa3d7662698777", + "sidecarErrorSourceSha256": "17948ad74350dc487ab20d24a0fb89daf0b426b98396454fff0e3856e922d82c", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "preparedHarnessHomeSourceSha256": "bb286f7ddc6e9597c56d46bf7ff6e3b556c5b680d7e147fd46518724432b083e", + "runtimeHostSourceSha256": "13f9ac382a279a67adb8974c36bbed38fe43d9b93b87bd7a05e879ead7f03d63", + "runtimeAdapterSourceSha256": "7c47c05306ebe7d1ca6b5d046a96f5726a48adec4d27e7c16788f505f25b5db2", + "runtimeSandboxSourceSha256": "04a49a24e3828ce8fca908de15e31ae932b8b1b4f7ecec0308627f6a8da9ba20", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/server/src/__tests__/attention-service.test.ts b/server/src/__tests__/attention-service.test.ts index c059516def2..8cb9bf90788 100644 --- a/server/src/__tests__/attention-service.test.ts +++ b/server/src/__tests__/attention-service.test.ts @@ -7,7 +7,6 @@ import { eq, sql } from "drizzle-orm"; import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture"; import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; import { - activityLog, agents, approvals, assets, @@ -17,11 +16,8 @@ import { createDb, decisionQueueItems, decisionQueues, - decisionArchiveNotificationOutbox, - decisionRetention, decisions, decisionTriage, - decisionTriageEvents, documents, heartbeatRunEvents, heartbeatRuns, @@ -70,35 +66,9 @@ describeEmbeddedPostgres("attention service", () => { }, 30_000); afterEach(async () => { - await db.delete(inboxDismissals); - await db.delete(decisionArchiveNotificationOutbox); - await db.delete(decisionRetention); - await db.delete(decisionTriageEvents); - await db.delete(decisionTriage); - await db.delete(decisionQueueItems); - await db.delete(decisionQueues); - await db.delete(decisions); - await db.delete(issueThreadInteractions); - await db.delete(issueApprovals); - await db.delete(issueAttachments); - await db.delete(issueDocuments); - await db.delete(heartbeatRunEvents); - await db.delete(heartbeatRuns); - await db.delete(budgetIncidents); - await db.delete(budgetPolicies); - await db.delete(joinRequests); - await db.delete(invites); - await db.delete(issueRecoveryActions); - await db.delete(issueRelations); - await db.delete(activityLog); - await db.delete(approvals); - await db.delete(issues); - await db.delete(assets); - await db.delete(documents); - await db.delete(projectWorkspaces); - await db.delete(projects); - await db.delete(agents); - await db.delete(companies); + + + await db.execute(sql`TRUNCATE TABLE companies CASCADE`); }); afterAll(async () => { diff --git a/server/src/__tests__/heartbeat-process-recovery.test.ts b/server/src/__tests__/heartbeat-process-recovery.test.ts index ca2576a5b59..fd13bc0282e 100644 --- a/server/src/__tests__/heartbeat-process-recovery.test.ts +++ b/server/src/__tests__/heartbeat-process-recovery.test.ts @@ -2793,15 +2793,6 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { const lateCallback = new Promise((resolve) => { releaseLateCallback = resolve; }); - const pendingOperationDrain = - outcome === "late_callback" - ? vi - .spyOn( - paperclipRunner, - "drainRetainedRunnerdMaintenanceOperations", - ) - .mockImplementation(() => lateCallback) - : null; let physicalCleanupFinished = false; mockRetainedNativeCleanup.mockImplementationOnce( async (cleanupDb, input) => { @@ -2809,6 +2800,9 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { expect(input).toEqual({ companyId, runId }); await held; physicalCleanupFinished = true; + + + if (outcome === "late_callback") await lateCallback; if (outcome === "rejected") throw new Error("maintenance-test-closed-failure"); return { @@ -2873,10 +2867,8 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { expect(mockAdapterExecute).not.toHaveBeenCalled(); expect(mockExecutePaperclipNativeSession).not.toHaveBeenCalled(); release(); - if (pendingOperationDrain) { - await vi.waitFor(() => - expect(pendingOperationDrain).toHaveBeenCalled(), - ); + if (outcome === "late_callback") { + await vi.waitFor(() => expect(physicalCleanupFinished).toBe(true)); expect(physicalCleanupFinished).toBe(true); expect(drainFinished).toBe(false); expect(heartbeat.getTaskDrainStatus().quiescent).toBe(false); @@ -2916,7 +2908,6 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => { await startup; await drain; await heartbeat.drainActiveRunExecutions(); - pendingOperationDrain?.mockRestore(); } }); }, diff --git a/server/src/__tests__/native-finalization-recovery.test.ts b/server/src/__tests__/native-finalization-recovery.test.ts index f54811a36d1..2b8f36166b2 100644 --- a/server/src/__tests__/native-finalization-recovery.test.ts +++ b/server/src/__tests__/native-finalization-recovery.test.ts @@ -1,3 +1,4 @@ +import { findRetainedNativeSessionCleanupCandidates } from "../services/native-runtime/retained-native-session-cleanup.js"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { randomUUID } from "node:crypto"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; @@ -27,7 +28,7 @@ import { reconcileRetainedNativeSessionCleanups, } from "../services/native-runtime/native-finalization-reconciler.js"; import { PaperclipControlPlanePort } from "../services/native-runtime/paperclip-control-plane-port.js"; -import { assertRetainedNativeSourceArchiveSettled } from "../services/native-runtime/native-session-executor.js"; +import { assertRetainedNativeSourceArchiveSettled, reconcileRetainedNativeSessionCleanup } from "../services/native-runtime/native-session-executor.js"; const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_FINALIZATION_TEST_DATABASE_URL"); @@ -392,6 +393,7 @@ describe("retained native cleanup discovery", () => { async function candidate( options: { + owner?: { companyId: string; agentId: string }; run?: Partial; coordinator?: Partial; schemaStatus?: string; @@ -399,6 +401,7 @@ describe("retained native cleanup discovery", () => { revision?: number; } = {}, ) { + const owner = options.owner ?? { companyId, agentId }; const runId = randomUUID(); const issueId = options.issueId ?? randomUUID(); const contractId = randomUUID(); @@ -406,13 +409,13 @@ describe("retained native cleanup discovery", () => { .insert(issues) .values({ id: issueId, - companyId, + companyId: owner.companyId, title: "Retained session", - assigneeAgentId: agentId, + assigneeAgentId: owner.agentId, }); await db.insert(completionContracts).values({ id: contractId, - companyId, + companyId: owner.companyId, issueId, revision: options.revision ?? 1, schemaVersion: "paperclip.completion-contract.v1", @@ -427,8 +430,8 @@ describe("retained native cleanup discovery", () => { }); await db.insert(heartbeatRuns).values({ id: runId, - companyId, - agentId, + companyId: owner.companyId, + agentId: owner.agentId, runtimeMode: "native", status: "succeeded", nativeIssueId: issueId, @@ -444,7 +447,7 @@ describe("retained native cleanup discovery", () => { const [result] = await db .insert(nativeRunResults) .values({ - companyId, + companyId: owner.companyId, issueId, runId, completionContractId: contractId, @@ -457,13 +460,13 @@ describe("retained native cleanup discovery", () => { const [assessment] = await db .insert(workAssessments) .values({ - companyId, + companyId: owner.companyId, issueId, runId, contractId, resultId: result!.id, triggerKind: "native_result", - triggerActorCompanyId: companyId, + triggerActorCompanyId: owner.companyId, priorIssueStatus: "in_progress", priorStatusVersion: 0, policyVersion: "phase6-v3", @@ -474,7 +477,7 @@ describe("retained native cleanup discovery", () => { const [decision] = await db .insert(statusDecisions) .values({ - companyId, + companyId: owner.companyId, issueId, runId, assessmentId: assessment!.id, @@ -488,7 +491,7 @@ describe("retained native cleanup discovery", () => { }) .returning(); await db.insert(nativeRunFinalizations).values({ - companyId, + companyId: owner.companyId, issueId, runId, phase: "committed", @@ -500,6 +503,73 @@ describe("retained native cleanup discovery", () => { return runId; } + it("does not misclassify a real competing row lock as ineligible retained history", async () => { + const runId = await candidate(); + await db.transaction(async tx => { + await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update"); + await expect(reconcileRetainedNativeSessionCleanup(db, { companyId, runId })) + .rejects.toMatchObject({ cause: { code: "55P03" } }); + }); + }); + + it("reconstructs future lease deadlines without authorizing early legacy cleanup", async () => { + const leaseExpiresAt = new Date(Date.now() + 60_000); + const runId = await candidate({ coordinator: { leaseOwner: "other-owner", leaseExpiresAt } }); + const scope = { companyId, runId }; + expect(await findRetainedNativeSessionCleanupCandidates(db, { scope, limit: 100 })).toEqual([]); + expect(await findRetainedNativeSessionCleanupCandidates(db, { scope, limit: 100, includeLeased: true })) + .toEqual([expect.objectContaining({ runId, leaseOwner: "other-owner", leaseExpiresAt })]); + const cleanup = vi.fn(); + await reconcileRetainedNativeSessionCleanups(db, { cleanup }); + expect(cleanup).not.toHaveBeenCalled(); + await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1000) }) + .where(eq(nativeRunFinalizations.runId, runId)); + expect(await findRetainedNativeSessionCleanupCandidates(db, { scope, limit: 100 })) + .toEqual([expect.objectContaining({ runId })]); + }); + + it("scopes reconstruction to a company, run, task, or agent without leaking another tenant", async () => { + const other = { companyId: randomUUID(), agentId: randomUUID() }; + await db.insert(companies).values({ id: other.companyId, name: "Other cleanup tenant", issuePrefix: "NCO" }); + await db.insert(agents).values({ id: other.agentId, companyId: other.companyId, name: "Other cleanup", adapterType: "codex_local" }); + try { + const first = await candidate(); + const second = await candidate(); + const foreign = await candidate({ owner: other }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, first)); + for (const scope of [{ companyId, runId: first }, { companyId, issueId: run.nativeIssueId! }]) { + expect((await findRetainedNativeSessionCleanupCandidates(db, { scope, limit: 100, includeLeased: true })) + .map(row => row.runId)).toEqual([first]); + } + expect((await findRetainedNativeSessionCleanupCandidates(db, { scope: { companyId, agentId }, limit: 100 })) + .map(row => row.runId).sort()).toEqual([first, second].sort()); + expect(await findRetainedNativeSessionCleanupCandidates(db, { + scope: { companyId, runId: foreign }, limit: 100, + })).toEqual([]); + expect((await findRetainedNativeSessionCleanupCandidates(db, { scope: other, limit: 100 })) + .map(row => row.runId)).toEqual([foreign]); + } finally { + await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, other.companyId)); + } + }); + + it("pages reconstruction past retained history without wrapping or admitting previously attempted cleanup", async () => { + const ids = [await candidate(), await candidate(), await candidate()].sort(); + await candidate({ coordinator: { recoveryHistory: [{ kind: "native_cleanup_runner_epoch", phase: "spawned" }] } }); + let after: string | undefined; + const found: string[] = []; + for (;;) { + const rows = await findRetainedNativeSessionCleanupCandidates(db, { + scope: { companyId }, after, limit: 1, includeLeased: true, + }); + if (!rows.length) break; + found.push(rows[0].runId); + after = rows[0].runId; + expect(found.length).toBeLessThanOrEqual(ids.length); + } + expect(found).toEqual(ids); + }); + it("discovers exact committed results even with stale nativePhase or privately recovered errors", async () => { const stale = await candidate(); const recovered = await candidate({ diff --git a/server/src/__tests__/native-status-arbiter-corpus.test.ts b/server/src/__tests__/native-status-arbiter-corpus.test.ts index 6a0d69db056..225a591427b 100644 --- a/server/src/__tests__/native-status-arbiter-corpus.test.ts +++ b/server/src/__tests__/native-status-arbiter-corpus.test.ts @@ -1,3 +1,4 @@ +import { retainedNativeSessionCleanupWork } from "../services/native-runtime/retained-native-session-cleanup-work.js"; import { pendingNativeGovernance } from "../services/native-runtime/native-governance.js"; import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js"; import { ensureNativeCompletionContract } from "../services/native-runtime/completion-contracts.js"; @@ -3072,6 +3073,55 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => { expect((await issueService(db).getById(seeded.issueId))!.status).toBe(delivery === "queued" ? "in_progress" : "done"); }, 30_000); + it("wakes retained cleanup after a decision signal was consumed before terminal projection", async () => { + const seeded = await seedFixture({ ...corpus.fixtures[0]!, id: `cleanup-projection-${randomUUID()}` }); + await db.update(completionContracts).set({ risk: "low", completionAuthority: "agent_claim_policy" }) + .where(eq(completionContracts.id, seeded.contractId!)); + await db.update(heartbeatRuns).set({ status: "running", finishedAt: null, errorCode: "adapter_failed", + error: "provider_transport_failed: runner did not durably suspend before checkpoint" }) + .where(eq(heartbeatRuns.id, seeded.runId)); + const cleanup = vi.fn(async (input: { companyId: string; runId: string }) => { + await db.update(nativeRunFinalizations).set({ recoveryHistory: [{ kind: "native_cleanup_maintenance", phase: "settled" }] }) + .where(eq(nativeRunFinalizations.runId, input.runId)); + return { runId: input.runId, status: "settled" as const }; + }); + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + const worker = retainedNativeSessionCleanupWork(db, cleanup).register(coordinator, () => true); + const flush = () => vi.waitFor(() => { + expect(idleWorkSnapshot().active).toBe(0); + expect(coordinator.nextWakeAt()).toBeNull(); + }, { timeout: 10_000 }); + let consumedBeforeProjection = false; + const transaction = db.transaction.bind(db); + const boundary = vi.spyOn(db, "transaction").mockImplementation(async (...args) => { + const result = await transaction(...args); + if (!consumedBeforeProjection) { + const [row] = await db.select({ phase: nativeRunFinalizations.phase, status: heartbeatRuns.status }) + .from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId)) + .where(eq(heartbeatRuns.id, seeded.runId)); + if (row?.phase === "committed" && row.status === "running") { + consumedBeforeProjection = true; + + + await flush(); + expect(cleanup).not.toHaveBeenCalled(); + } + } + return result; + }); + try { + await worker.ready; + await finalizeNativeRun({ db, runId: seeded.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true, + skipNativeResponsePresentation: true, skipNativeReviewResponse: true }); + expect(consumedBeforeProjection).toBe(true); + await flush(); + expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId: seeded.runId }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)))[0].status).toBe("succeeded"); + expect(errors).toEqual([]); + } finally { boundary.mockRestore(); worker.stop(); await coordinator.stop(); } + }, 30_000); + it("completes a new merge run after an old CI review and bounds repeated incomplete results", async () => { const seeded = await seedAutomaticReview(); const [sourceRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)); diff --git a/server/src/__tests__/retained-native-session-cleanup-work.test.ts b/server/src/__tests__/retained-native-session-cleanup-work.test.ts new file mode 100644 index 00000000000..1f698c7b210 --- /dev/null +++ b/server/src/__tests__/retained-native-session-cleanup-work.test.ts @@ -0,0 +1,631 @@ +import { heartbeatService } from "../services/heartbeat.js"; +import * as nativeExecutor from "../services/native-runtime/native-session-executor.js"; +import { finalizeNativeRun } from "../services/native-runtime/native-run-finalizer.js"; +import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js"; +import { randomUUID } from "node:crypto"; +import os from "node:os"; +import { PROCESS_IDENTITY_RECORDED, releaseSettledNativeRunProcess } from "../services/native-local-process-stop.js"; +import { setTimeout as delay } from "node:timers/promises"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { eq } from "drizzle-orm"; +import { agents, companies, completionContracts, createDb, environmentLeases, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, + nativeRunResults, statusDecisions, workAssessments, type Db } from "@paperclipai/db"; +import { settleAbsentSandbox, settleStopOnlyCleanup } from "../services/sandbox-stop-and-retain.js"; +import { settleNativeWorkspaceExportResume } from "../services/native-runtime/native-workspace-export-resume.js"; +import type { SandboxAbsenceProof } from "../services/sandbox-absence-proof.js"; +import { environmentService } from "../services/environments.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; +import { notifyRetainedNativeSessionCleanup, releaseRetainedNativeSessionCleanup } from "../services/retained-native-session-cleanup-notifications.js"; +import { retainedNativeSessionCleanupWork } from "../services/native-runtime/retained-native-session-cleanup-work.js"; +import { reconcileRetainedNativeSessionCleanups, runRetainedNativeSessionCleanup, type NativeCleanupOutcome } from "../services/native-runtime/retained-native-session-cleanup.js"; +import { reconcileRetainedNativeSessionCleanup } from "../services/native-runtime/native-session-executor.js"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; + +function deferred() { + let resolve!: () => void; + const promise = new Promise(done => { resolve = done; }); + return { promise, resolve }; +} + +describe("retained native cleanup scheduled lifecycle", () => { + let temporary: Awaited> | undefined; + let db: ReturnType; + let coordinator: ReturnType | undefined; + let worker: ReturnType["register"]> | undefined; + const companyId = randomUUID(), agentId = randomUUID(); + const closeError = "provider_transport_failed: runner did not durably suspend before checkpoint"; + let admitted = true; + const errors: unknown[] = []; + async function settle(input: { runId: string; companyId: string }): Promise { + await db.update(nativeRunFinalizations).set({ leaseOwner: null, leaseExpiresAt: null, + recoveryHistory: [{ kind: "native_cleanup_maintenance", phase: "settled" }] }) + .where(eq(nativeRunFinalizations.runId, input.runId)); + return { runId: input.runId, status: "settled" }; + } + const cleanup = vi.fn(settle); + beforeAll(async () => { + const external = resolveExternalTestDatabaseUrl("PAPERCLIP_RETAINED_CLEANUP_TEST_DATABASE_URL"); + if (!external) temporary = await startEmbeddedPostgresTestDatabase("paperclip-retained-cleanup-"); + db = createDb(external ?? temporary!.connectionString); + await db.insert(companies).values({ id: companyId, name: "Retained cleanup lifecycle", issuePrefix: `RC${randomUUID().slice(0, 8)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Cleanup agent", adapterType: "codex_local" }); + }, 30_000); + beforeEach(() => { + admitted = true; errors.length = 0; cleanup.mockReset().mockImplementation(settle); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + worker?.stop(); await coordinator?.stop(); worker = undefined; coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); + await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, companyId)); + expect(errors).toEqual([]); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => true, + onError: error => errors.push(error) }); + worker = retainedNativeSessionCleanupWork(db, cleanup).register(coordinator, () => admitted); + await worker.ready; + } + async function flush() { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(idleWorkSnapshot().active).toBe(0); + expect(coordinator!.nextWakeAt()).toBeNull(); + }, { timeout: 15_000 }); + } + async function quiet(ms = 120_000) { + const previous = db.$client.options.debug; + const queries: string[] = []; + db.$client.options.debug = (_id, query) => { queries.push(query); }; + try { await vi.advanceTimersByTimeAsync(ms); await delay(10); expect(queries).toEqual([]); } + finally { db.$client.options.debug = previous; } + } + async function commitCandidate() { + const runId = randomUUID(); + return db.transaction(async tx => { + await notifyNativeFinalizationWork(tx, { companyId, runId }); + return candidate({ runId }, tx as unknown as Db); + }); + } + async function candidate( + options: { + runId?: string; + owner?: { companyId: string; agentId: string }; + run?: Partial; + coordinator?: Partial; + schemaStatus?: string; + issueId?: string; + revision?: number; + } = {}, + writer: Db = db, + ) { + const owner = options.owner ?? { companyId, agentId }; + const runId = options.runId ?? randomUUID(); + const issueId = options.issueId ?? randomUUID(); + const contractId = randomUUID(); + if (!options.issueId) await writer + .insert(issues) + .values({ + id: issueId, + companyId: owner.companyId, + title: "Retained session", + assigneeAgentId: owner.agentId, + }); + await writer.insert(completionContracts).values({ + id: contractId, + companyId: owner.companyId, + issueId, + revision: options.revision ?? 1, + schemaVersion: "paperclip.completion-contract.v1", + policyVersion: "phase6-v3", + risk: "standard", + completionAuthority: "server_arbiter", + incompleteCriteriaPolicy: "preserve_non_terminal", + contractJson: {}, + canonicalSha256: runId, + createdByActorType: "system", + createdByActorId: "test", + }); + await writer.insert(heartbeatRuns).values({ + id: runId, + companyId: owner.companyId, + agentId: owner.agentId, + runtimeMode: "native", + status: "succeeded", + nativeIssueId: issueId, + nativeSessionId: runId, + nativePhase: "terminal_failure", + completionContractId: contractId, + completionContractSha256: runId, + finishedAt: new Date(), + errorCode: "adapter_failed", + error: closeError, + ...options.run, + }); + const [result] = await writer + .insert(nativeRunResults) + .values({ + companyId: owner.companyId, + issueId, + runId, + completionContractId: contractId, + serverFingerprint: runId, + canonicalSha256: runId, + schemaStatus: options.schemaStatus ?? "accepted", + resultJson: {}, + }) + .returning(); + const [assessment] = await writer + .insert(workAssessments) + .values({ + companyId: owner.companyId, + issueId, + runId, + contractId, + resultId: result!.id, + triggerKind: "native_result", + triggerActorCompanyId: owner.companyId, + priorIssueStatus: "in_progress", + priorStatusVersion: 0, + policyVersion: "phase6-v3", + assessmentJson: {}, + inputDigest: runId, + }) + .returning(); + const [decision] = await writer + .insert(statusDecisions) + .values({ + companyId: owner.companyId, + issueId, + runId, + assessmentId: assessment!.id, + decisionVersion: options.revision ?? 1, + policyVersion: "phase6-v3", + fromStatus: "in_progress", + toStatus: "in_review", + reasonCode: "prior_status_terminal_preserved", + decisionJson: {}, + decisionDigest: runId, + }) + .returning(); + await writer.insert(nativeRunFinalizations).values({ + companyId: owner.companyId, + issueId, + runId, + phase: "committed", + resultId: result!.id, + assessmentId: assessment!.id, + decisionId: decision!.id, + ...options.coordinator, + }); + return runId; + } + + + it("becomes SQL-quiet with no work, then processes the first committed candidate", async () => { + await start(); await flush(); await quiet(); + const runId = await commitCandidate(); + await flush(); + expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId }); + await quiet(); + }); + + it("leaves retained unprovable history quiet through the real physical cleanup boundary", async () => { + await candidate(); + cleanup.mockImplementation(input => reconcileRetainedNativeSessionCleanup(db, input)); + await start(); await flush(); + expect(cleanup).toHaveBeenCalledTimes(1); + await quiet(); + }); + + it("keeps dormant retained cleanup quiet on unscoped finalization hints and resumes from local release", async () => { + const runId = await candidate(); + cleanup.mockResolvedValue({ runId, status: "deferred", waitFor: "session_release" }); + await start(); + await flush(); + await quiet(); + expect(cleanup).toHaveBeenCalledTimes(1); + expect(await retainedNativeSessionCleanupWork(db, cleanup).load({ companyId, runId })).toHaveLength(1); + const debug = db.$client.options.debug; + const queries: string[] = []; + db.$client.options.debug = (_connection, query) => { + if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query); + }; + try { + for (let i = 0; i < 3; i++) { + await db.transaction(tx => notifyNativeFinalizationWork(tx, { companyId })); + await flush(); + } + expect(queries).toEqual([]); + expect(cleanup).toHaveBeenCalledTimes(1); + } finally { db.$client.options.debug = debug; } + cleanup.mockImplementation(settle); + releaseRetainedNativeSessionCleanup(db, { companyId, runId }); + await flush(); + expect(cleanup).toHaveBeenCalledTimes(2); + await quiet(); + }); + + it.each(["session_release", "environment_release", "maintenance_settlement"] as const)( + "waits quietly for %s and then reactivates only the affected run", async waitFor => { + const runId = await candidate(); + cleanup.mockResolvedValueOnce({ runId, status: "deferred", waitFor }); + await start(); await flush(); await quiet(); + releaseRetainedNativeSessionCleanup(db, { companyId, runId }); + await flush(); + expect(cleanup).toHaveBeenCalledTimes(2); + await quiet(); + }, + ); + + it.each(["single", "run"] as const)("reactivates after the real %s lease release commits, but not rollback", async method => { + const runId = await candidate(); + const [lease] = await db.insert(environmentLeases).values({ companyId, heartbeatRunId: runId, + status: "active", provider: "local" }).returning(); + cleanup.mockImplementation(async input => { + const [current] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease!.id)); + return current!.status === "active" + ? { runId: input.runId, status: "deferred", waitFor: "environment_release" } + : settle(input); + }); + const release = async (writer: Db) => { + const service = environmentService(writer); + if (method === "single") await service.releaseLease(lease!.id); + else await service.releaseLeasesForRun(runId); + }; + await start(); await flush(); await quiet(); + expect(cleanup).toHaveBeenCalledTimes(1); + await expect(db.transaction(async tx => { + await release(tx as unknown as Db); + await vi.advanceTimersByTimeAsync(1000); + expect(cleanup).toHaveBeenCalledTimes(1); + throw new Error("rollback lease release"); + })).rejects.toThrow("rollback lease release"); + await flush(); await quiet(); + const afterRollback = cleanup.mock.calls.length; + await db.transaction(async tx => { + await release(tx as unknown as Db); + await vi.advanceTimersByTimeAsync(1000); + expect(cleanup).toHaveBeenCalledTimes(afterRollback); + }); + await flush(); + expect(cleanup).toHaveBeenCalledTimes(afterRollback + 1); + expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0]!.recoveryHistory) + .toEqual([{ kind: "native_cleanup_maintenance", phase: "settled" }]); + await quiet(); + }); + + it("keeps pending cleanup quiet through metadata updates and stale release attempts", async () => { + const runId = await candidate(), attemptId = randomUUID(); + const [lease] = await db.insert(environmentLeases).values({ companyId, heartbeatRunId: runId, + status: "pending_cleanup", provider: "local", metadata: { pendingCleanupAttemptId: attemptId } }).returning(); + cleanup.mockResolvedValueOnce({ runId, status: "deferred", waitFor: "environment_release" }); + await start(); await flush(); + const service = environmentService(db); + await service.updateLeaseMetadata(lease!.id, { pendingCleanupAttemptId: attemptId, pendingCleanupInFlight: true }); + await service.releaseLease(lease!.id, "pending_cleanup", { expectedPendingCleanupAttemptId: attemptId, cleanupStatus: "failed" }); + expect(await service.releaseLease(lease!.id, "released", { expectedPendingCleanupAttemptId: randomUUID(), cleanupStatus: "success" })).toBeNull(); + await flush(); await quiet(); + expect(cleanup).toHaveBeenCalledTimes(1); + expect(await service.releaseLease(lease!.id, "released", { expectedPendingCleanupAttemptId: attemptId, cleanupStatus: "success" })).not.toBeNull(); + await flush(); + expect(cleanup).toHaveBeenCalledTimes(2); + await quiet(); + }); + + it.each(["stop-only", "export-resume", "absent"] as const)("reactivates after real %s settlement with outer transaction ownership", async method => { + const ownerToken = randomUUID(); + const runId = await candidate({ run: { runnerProfileJson: { nativeWorkspaceFinalizationOwner: { token: ownerToken } } } }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const leaseId = randomUUID(), providerLeaseId = randomUUID(), attemptId = randomUUID(), pluginId = randomUUID(); + const binding = { companyId, runId, leaseId, provider: "daytona" as const, providerLeaseId }; + const marker = { ...binding, requestId: attemptId, pluginId, + schema: method === "export-resume" ? "paperclip.workspace-export-resume.v2" : "paperclip.sandbox-stop-and-retain.v1", + resultId: randomUUID() }; + const [lease] = await db.insert(environmentLeases).values({ id: leaseId, companyId, heartbeatRunId: runId, + issueId: run!.nativeIssueId, provider: "daytona", providerLeaseId, status: "pending_cleanup", cleanupStatus: "failed", + metadata: { pluginId, pendingCleanupAttemptId: attemptId, + [method === "export-resume" ? "nativeWorkspaceExportResume" : "sandboxStopAndRetain"]: marker } }).returning(); + const proof: SandboxAbsenceProof = { ...binding, issueId: run!.nativeIssueId!, ownerToken, v: 1, + iss: "paperclip-cloud", aud: "paperclip-sandbox-absence/v1", sub: "fixture", absent: true, + iat: Math.floor(Date.now() / 1000), exp: Math.floor(Date.now() / 1000) + 60 }; + cleanup.mockImplementation(async input => { + const [current] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId)); + return current!.status === "pending_cleanup" + ? { runId: input.runId, status: "deferred", waitFor: "environment_release" } + : settle(input); + }); + const release = async (writer: Db) => { + const options = { attemptId, receipt: { providerLeaseId, state: "stopped" } }; + if (method === "absent") expect(await settleAbsentSandbox(writer, lease!, { proof, receipt: "verified-upstream" })).toBe(true); + else if (method === "export-resume") expect(await settleNativeWorkspaceExportResume(writer, lease!, options)).not.toBeNull(); + else expect(await settleStopOnlyCleanup(writer, lease!, options)).not.toBeNull(); + }; + await start(); await flush(); await quiet(); + expect(cleanup).toHaveBeenCalledTimes(1); + await expect(db.transaction(async tx => { + await release(tx as unknown as Db); + await vi.advanceTimersByTimeAsync(1000); + expect(cleanup).toHaveBeenCalledTimes(1); + throw new Error("rollback settlement"); + })).rejects.toThrow("rollback settlement"); + await flush(); await quiet(); + const afterRollback = cleanup.mock.calls.length; + await db.transaction(async tx => { + await release(tx as unknown as Db); + await vi.advanceTimersByTimeAsync(1000); + expect(cleanup).toHaveBeenCalledTimes(afterRollback); + }); + await flush(); + expect(cleanup).toHaveBeenCalledTimes(afterRollback + 1); + await quiet(); + }); + + it("reactivates immediately after verified process release commits without waiting for its probe", async () => { + const pid = 2_000_000_000, startedAt = new Date(); + const runId = await candidate({ run: { status: "failed", processPid: pid, processGroupId: pid, + processStartedAt: startedAt, runnerProfileJson: { sessionCheckpoint: {} } } }); + await db.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: 0, + eventType: PROCESS_IDENTITY_RECORDED, payload: { targetKind: "local", hostname: os.hostname(), + processPid: pid, processGroupId: pid, processStartedAt: startedAt.toISOString() } }); + await db.update(heartbeatRuns).set({ nextEventSeq: 1 }).where(eq(heartbeatRuns.id, runId)); + const originalKill = process.kill.bind(process); + vi.spyOn(process, "kill").mockImplementation((id, signal) => { + if ((id === pid || id === -pid) && signal === 0) throw Object.assign(new Error("gone"), { code: "ESRCH" }); + return originalKill(id, signal); + }); + cleanup.mockImplementation(async input => { + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId)); + return run!.processPid !== null + ? { runId: input.runId, status: "deferred", waitFor: "process_exit" } + : settle(input); + }); + await start(); + expect(cleanup).toHaveBeenCalledTimes(1); + expect(coordinator!.nextWakeAt()).not.toBeNull(); + await expect(db.transaction(async tx => { + expect(await releaseSettledNativeRunProcess(tx as unknown as Db, runId)).toBe("released"); + await vi.advanceTimersByTimeAsync(1000); + expect(cleanup).toHaveBeenCalledTimes(1); + throw new Error("rollback process release"); + })).rejects.toThrow("rollback process release"); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(cleanup).toHaveBeenCalledTimes(2); + expect(idleWorkSnapshot().active).toBe(0); + }); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]!.processPid).toBe(pid); + const afterRollback = cleanup.mock.calls.length; + await db.transaction(async tx => { + expect(await releaseSettledNativeRunProcess(tx as unknown as Db, runId)).toBe("released"); + await vi.advanceTimersByTimeAsync(1000); + expect(cleanup).toHaveBeenCalledTimes(afterRollback); + }); + await vi.advanceTimersByTimeAsync(1000); + await vi.waitFor(() => expect(cleanup).toHaveBeenCalledTimes(afterRollback + 1)); + await flush(); await quiet(); + }); + + it("reactivates when the real finalizer projects an already committed result to a terminal run", async () => { + const runId = await candidate({ run: { status: "running", finishedAt: null } }); + await db.update(nativeRunResults).set({ resultJson: { terminal: { runTerminalState: "succeeded" } } }) + .where(eq(nativeRunResults.runId, runId)); + await start(); await flush(); await quiet(); + expect(cleanup).not.toHaveBeenCalled(); + const finalize = (writer: Db) => finalizeNativeRun({ db: writer, runId, workspaceFinalizeStatus: "succeeded", + projectRunStatus: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true }); + await expect(db.transaction(async tx => { + await finalize(tx as unknown as Db); + await vi.advanceTimersByTimeAsync(1000); + expect(cleanup).not.toHaveBeenCalled(); + throw new Error("rollback final projection"); + })).rejects.toThrow("rollback final projection"); + await flush(); await quiet(); + expect(cleanup).not.toHaveBeenCalled(); + await finalize(db); await flush(); + expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId }); + const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + expect(run!.status).toBe("succeeded"); + expect(run!.error).toBeNull(); + expect(run!.resultJson?.recoveredExecutionFailure).toMatchObject({ errorCode: "adapter_failed", error: closeError }); + await quiet(); + }); + + it("reconstructs a lease deadline across restart and does no SQL before it", async () => { + const leaseExpiresAt = new Date(Date.now() + 60_000); + const runId = await candidate({ coordinator: { leaseOwner: "other-owner", leaseExpiresAt } }); + await start(); + expect(cleanup).not.toHaveBeenCalled(); + expect(coordinator!.nextWakeAt()).toBe(leaseExpiresAt.getTime()); + await quiet(10_000); + worker!.stop(); await coordinator!.stop(); + await start(); + expect(coordinator!.nextWakeAt()).toBe(leaseExpiresAt.getTime()); + await quiet(10_000); + await vi.advanceTimersByTimeAsync(40_000); await flush(); + expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId }); + await quiet(); + }); + + it("retains only the concrete process-exit probe and disarms it after settlement", async () => { + const runId = await candidate(); + cleanup.mockResolvedValueOnce({ runId, status: "deferred", waitFor: "process_exit" }); + await start(); + expect(cleanup).toHaveBeenCalledTimes(1); + await quiet(20_000); + await vi.advanceTimersByTimeAsync(10_000); await flush(); + expect(cleanup).toHaveBeenCalledTimes(2); + await quiet(); + }); + + it("does not run before outer commit and clears rolled-back intent", async () => { + await start(); await flush(); + const entered = deferred(), release = deferred(); + const transaction = db.transaction(async outer => { + await outer.transaction(async tx => { + await notifyRetainedNativeSessionCleanup(tx, { companyId }); + await candidate({}, tx as unknown as Db); + }); + entered.resolve(); await release.promise; + throw new Error("rollback fixture"); + }); + const rolledBack = expect(transaction).rejects.toThrow("rollback fixture"); + await entered.promise; + await vi.advanceTimersByTimeAsync(10_000); + expect(cleanup).not.toHaveBeenCalled(); + release.resolve(); await rolledBack; await flush(); await quiet(); + const runId = await commitCandidate(); await flush(); + expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId }); + await quiet(); + }); + + it("retains the first write during drain without passive SQL until resume", async () => { + await start(); await flush(); admitted = false; + const runId = await commitCandidate(); + await vi.advanceTimersByTimeAsync(1000); await delay(10); + expect(cleanup).not.toHaveBeenCalled(); + await quiet(); + admitted = true; await vi.advanceTimersByTimeAsync(30_000); await flush(); + expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId }); + await quiet(); + }); + + it("retains an operational failure across drain and retries after resume", async () => { + await candidate(); cleanup.mockRejectedValueOnce(new Error("connection lost")); + await start(); expect(errors).toHaveLength(1); errors.length = 0; + admitted = false; await quiet(); + expect(cleanup).toHaveBeenCalledTimes(1); + admitted = true; await vi.advanceTimersByTimeAsync(30_000); await flush(); + expect(cleanup).toHaveBeenCalledTimes(2); + await quiet(); + }); + + it("keeps a committed wake arriving during cleanup and never overlaps worker dispatch", async () => { + await candidate(); + const entered = deferred(), release = deferred(); + let active = 0, maximum = 0; + cleanup.mockImplementation(async input => { + maximum = Math.max(maximum, ++active); + try { entered.resolve(); await release.promise; return await settle(input); } + finally { active--; } + }); + const starting = start(); await entered.promise; + await commitCandidate(); release.resolve(); await starting; await flush(); + expect(cleanup).toHaveBeenCalledTimes(2); expect(maximum).toBe(1); + await quiet(); + }); + + it("joins the original manual startup operation instead of performing cleanup twice", async () => { + await candidate(); + const entered = deferred(), release = deferred(); + cleanup.mockImplementation(async input => { entered.resolve(); await release.promise; return settle(input); }); + const manual = reconcileRetainedNativeSessionCleanups(db, { cleanup }); + await entered.promise; + const work = retainedNativeSessionCleanupWork(db, cleanup); + const [entry] = await work.load(); + const joined = work.check(entry); + try { expect(cleanup).toHaveBeenCalledTimes(1); } + finally { release.resolve(); } + await expect(joined).resolves.toBeNull(); await manual; + expect(cleanup).toHaveBeenCalledTimes(1); + }); + + it("serializes different cleanup inspections sharing a database lane", async () => { + const first = await candidate(), second = await candidate(); + const entered = deferred(), release = deferred(); + cleanup.mockImplementationOnce(async input => { entered.resolve(); await release.promise; return settle(input); }); + const active = runRetainedNativeSessionCleanup(db, { companyId, runId: first }, cleanup); + await entered.promise; + const queued = runRetainedNativeSessionCleanup(db, { companyId, runId: second }, cleanup); + try { await delay(10); expect(cleanup).toHaveBeenCalledTimes(1); } + finally { release.resolve(); } + await Promise.all([active, queued]); + expect(cleanup.mock.calls.map(([input]) => input.runId)).toEqual([first, second]); + }); + + it("joins scheduled physical cleanup through the assembled heartbeat shutdown owner", async () => { + const runId = await candidate(); + const entered = deferred(), release = deferred(); + const physical = vi.spyOn(nativeExecutor, "reconcileRetainedNativeSessionCleanup").mockImplementation(async (_db, input) => { + entered.resolve(); await release.promise; return settle(input); + }); + const heartbeat = heartbeatService(db); + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) }); + worker = heartbeat.registerRetainedNativeSessionCleanupWork(coordinator, () => true); + await entered.promise; + let drained = false; + const draining = heartbeat.drainActiveRunExecutions().then(() => { drained = true; }); + try { + await delay(10); + expect(drained).toBe(false); + expect(physical).toHaveBeenCalledExactlyOnceWith(db, { companyId, runId }); + } finally { release.resolve(); } + await worker.ready; await draining; await flush(); + expect(drained).toBe(true); await quiet(); + }); + + it("waits on the original in-flight cleanup when shutdown begins", async () => { + await candidate(); + const entered = deferred(), release = deferred(); + cleanup.mockImplementation(async input => { entered.resolve(); await release.promise; return settle(input); }); + const starting = start(); await entered.promise; + let stopped = false; + worker!.stop(); const stopping = coordinator!.stop().then(() => { stopped = true; }); + try { await delay(10); expect(stopped).toBe(false); } + finally { release.resolve(); } + await Promise.all([starting, stopping]); + expect(stopped).toBe(true); expect(cleanup).toHaveBeenCalledTimes(1); + releaseRetainedNativeSessionCleanup(db, { companyId }); + await quiet(); + }); + + it("replaces an old lease alarm when a committed change releases it early", async () => { + const runId = await candidate({ coordinator: { leaseOwner: "old-owner", leaseExpiresAt: new Date(Date.now() + 60_000) } }); + await start(); expect(cleanup).not.toHaveBeenCalled(); + await db.transaction(async tx => { + await notifyNativeFinalizationWork(tx, { companyId, runId }); + await tx.update(nativeRunFinalizations).set({ leaseOwner: null, leaseExpiresAt: null }).where(eq(nativeRunFinalizations.runId, runId)); + }); + await flush(); expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId }); + await quiet(); + }); + + it("cancels a saved alarm when a committed owner marks the candidate operator-required", async () => { + const runId = await candidate({ coordinator: { leaseOwner: "old-owner", leaseExpiresAt: new Date(Date.now() + 60_000) } }); + await start(); + await db.transaction(async tx => { + await notifyNativeFinalizationWork(tx, { companyId, runId }); + await tx.update(nativeRunFinalizations).set({ recoveryHistory: [{ kind: "native_cleanup_source_archive", phase: "operator_required" }] }) + .where(eq(nativeRunFinalizations.runId, runId)); + }); + await flush(); expect(cleanup).not.toHaveBeenCalled(); await quiet(); + }); + + it("retains startup discovery during drain and reconstructs only after admission resumes", async () => { + const runId = await candidate(); admitted = false; + await start(); await quiet(); expect(cleanup).not.toHaveBeenCalled(); + admitted = true; await vi.advanceTimersByTimeAsync(30_000); await flush(); + expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId }); await quiet(); + }); + + it("drains a backlog across discovery pages and becomes quiet with the completed history retained", async () => { + const runIds: string[] = []; + for (let index = 0; index < 103; index++) runIds.push(await candidate()); + await start(); await flush(); + expect(cleanup.mock.calls.map(([input]) => input.runId).sort()).toEqual(runIds.sort()); + expect(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, companyId))).toHaveLength(103); + await quiet(); + }); + + it("retries failed startup discovery without treating an unavailable database as empty", async () => { + const runId = await candidate(); + vi.spyOn(db, "select").mockImplementationOnce(() => { throw new Error("discovery connection lost"); }); + await start(); expect(errors).toHaveLength(1); errors.length = 0; + expect(cleanup).not.toHaveBeenCalled(); + await quiet(20_000); await vi.advanceTimersByTimeAsync(10_000); await flush(); + expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId }); await quiet(); + }); +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 1198f45960e..5b79c9d6dc5 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -64,7 +64,7 @@ const { retryMs?: number; run: (signal?: AbortSignal) => Promise; hasPending?: () => Promise; - }) => ({ ready: task.run(new AbortController().signal).catch(error => { deliveryWorkErrors.push({ queue, error }); }), wake: vi.fn() })); + }) => ({ ready: task.run(new AbortController().signal).catch(error => { deliveryWorkErrors.push({ queue, error }); }), wake: vi.fn(), stop: vi.fn(async () => {}) })); const createAppMock = vi.fn(async () => Object.assign((_: unknown, __: unknown) => {}, { locals: { fastResponses: { hasPending: vi.fn(async () => false), sweepPending: vi.fn(async () => undefined) }, @@ -103,6 +103,7 @@ const { reason: null, })); const heartbeatServiceMock = { + drainActiveRunExecutions: vi.fn(async () => undefined), reconcileCostAccounting: vi.fn(async () => ({ scanned: 0, accounted: 0 })), resolveSchedulingSuppression: resolveHeartbeatSchedulingSuppressionMock, recoverNativeRunsAfterRestart: vi.fn(async () => ({ @@ -116,6 +117,7 @@ const { reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })), registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), + registerRetainedNativeSessionCleanupWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })), registerNativeStatusReassessmentWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerNativeReviewResponseWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerNativeResponsePresentationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })), @@ -572,6 +574,17 @@ describe("startServer feedback export wiring", () => { } }); + it.each([false, true])("registers retained cleanup with its shutdown subscription only when maintenance is enabled (%s)", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); + await startServer(); await settleStartupHousekeeping(); + expect(heartbeatServiceMock.registerRetainedNativeSessionCleanupWork).toHaveBeenCalledTimes(enabled ? 1 : 0); + if (enabled) { + expect(heartbeatServiceMock.registerRetainedNativeSessionCleanupWork.mock.calls[0][1]()).toBe(true); + const worker = heartbeatServiceMock.registerRetainedNativeSessionCleanupWork.mock.results[0].value; + expect(fakeServer.on).toHaveBeenCalledWith("close", worker.stop); + } + }); + it.each([false, true])("registers native reassessment only when maintenance is enabled (%s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); await startServer(); await settleStartupHousekeeping(); @@ -1033,7 +1046,7 @@ describe("startServer feedback export wiring", () => { intervalCallback?.(); await new Promise((resolve) => setImmediate(resolve)); expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2); - expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true }); + expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true }); expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1); expect(retiredDetector).not.toHaveBeenCalled(); } finally { @@ -1189,6 +1202,37 @@ describe("startServer feedback export wiring", () => { } }); + it("becomes ready while retained cleanup is held but joins it at shutdown", async () => { + loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: true })); + let releaseCleanup!: () => void; + const ready = new Promise(resolve => { releaseCleanup = resolve; }); + const stop = vi.fn(); + heartbeatServiceMock.registerRetainedNativeSessionCleanupWork.mockReturnValueOnce({ ready, stop }); + const timer = vi.spyOn(globalThis, "setInterval").mockImplementation(() => 1 as unknown as ReturnType); + let started = false; + const starting = startServer().then(server => { started = true; return server; }); + let stopping: Promise | undefined; + let stopped = false; + try { + await vi.waitFor(() => expect(heartbeatServiceMock.registerRetainedNativeSessionCleanupWork).toHaveBeenCalledOnce()); + await settleStartupHousekeeping(); + expect(started).toBe(true); + expect(getStartupRecoveryState().phase).toBe("ready"); + expect(heartbeatServiceMock.registerNativeStatusReassessmentWork).toHaveBeenCalledOnce(); + const server = await starting; + stopping = server.shutdown().then(() => { stopped = true; }); + await settleStartupHousekeeping(); + expect(stop).toHaveBeenCalledOnce(); + expect(stopped).toBe(false); + } finally { + releaseCleanup(); + await starting; + await stopping; + timer.mockRestore(); + } + expect(stopped).toBe(true); + }); + it("reports ready before startup housekeeping finishes and starts the scheduler after it", async () => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: true, diff --git a/server/src/index.ts b/server/src/index.ts index 7202f5f7361..864edbeb808 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1404,6 +1404,7 @@ async function startServerWithDatabaseTeardown( }); let unregisterTerminalWorkspaceProbe: (() => void) | undefined; let unsubscribeNativeResponsePresentation: (() => void) | undefined; + let unsubscribeRetainedNativeSessionCleanup: (() => void) | undefined; let unsubscribeNativeWorkspaceSyncRetention: (() => void) | undefined; const fastResponses = app.locals.fastResponses as ReturnType; const fastResponseWorker = deliveryWork.register(DELIVERY_QUEUES.fastResponse, { @@ -1825,6 +1826,15 @@ async function startServerWithDatabaseTeardown( () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork, () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; + if (config.heartbeatSchedulerEnabled) { + const worker = heartbeat.registerRetainedNativeSessionCleanupWork(deliveryWork, + () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()); + unsubscribeRetainedNativeSessionCleanup = worker.stop; + server.on("close", worker.stop); + + + trackHeartbeatSchedulerWork(worker.ready); + } if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativeStatusReassessmentWork(deliveryWork, () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativeReviewResponseWork(deliveryWork, @@ -1976,7 +1986,7 @@ async function startServerWithDatabaseTeardown( // Periodically reap orphaned runs (5-min staleness threshold) and make sure // persisted queued work is still being driven forward. runPeriodicRecovery(() => heartbeat - .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true }) + .reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true }) .then(() => heartbeat.promoteDueScheduledRetries()) .then(async (promotion) => { await heartbeat.resumeQueuedRuns(); @@ -2173,6 +2183,7 @@ async function startServerWithDatabaseTeardown( await skillSourceSync.stop(); unsubscribeChatCompletions(); unsubscribeNativeResponsePresentation?.(); + unsubscribeRetainedNativeSessionCleanup?.(); await deliveryWork.stop(); await workspaceRecoveryWork.stop(); unregisterTerminalWorkspaceProbe?.(); diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 057f9a6c242..eaf9579001d 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; /** Topics for existing durable queues; register intent before their writes. */ export const DELIVERY_QUEUES = { + retainedNativeSessionCleanup: "retained-native-session-cleanup", nativeStatusReassessment: "native-status-reassessment", nativeReviewResponse: "native-review-response", nativeResponsePresentation: "native-response-presentation", diff --git a/server/src/services/environments.ts b/server/src/services/environments.ts index 91a6153d5f3..4a3e57f803b 100644 --- a/server/src/services/environments.ts +++ b/server/src/services/environments.ts @@ -1,3 +1,4 @@ +import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js"; import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js"; import { writeEnvironmentCleanup, notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js"; import { withoutCleanupVerification } from "./cleanup-verification-proof.js"; @@ -15,6 +16,7 @@ import { environmentLeases, environments, executionWorkspaces, + heartbeatRuns, instanceSettings, issues, projects, @@ -1690,6 +1692,12 @@ export function environmentService(db: Db) { companyId: previous.companyId, issueId: previous.issueId ?? undefined, }); } + if (previous.heartbeatRunId && ["active", "pending_cleanup"].includes(previous.status) + && status !== "pending_cleanup") { + await notifyRetainedNativeSessionCleanup(tx, { + companyId: previous.companyId, runId: previous.heartbeatRunId, + }); + } return tx.update(environmentLeases) .set({ status, @@ -1810,8 +1818,11 @@ export function environmentService(db: Db) { status: Extract = "released", ): Promise => { const now = new Date(); - const rows = await writeEnvironmentCleanup(db, tx => tx - .update(environmentLeases) + const rows = await writeEnvironmentCleanup(db, async tx => { + const [run] = await tx.select({ companyId: heartbeatRuns.companyId }).from(heartbeatRuns) + .where(eq(heartbeatRuns.id, heartbeatRunId)).limit(1); + if (run) await notifyRetainedNativeSessionCleanup(tx, { companyId: run.companyId, runId: heartbeatRunId }); + return tx.update(environmentLeases) .set({ status, releasedAt: now, @@ -1824,7 +1835,8 @@ export function environmentService(db: Db) { eq(environmentLeases.status, "active"), ), ) - .returning()); + .returning(); + }); return rows.map(toEnvironmentLease); }, }; diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 17574152073..05b54143118 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1023,6 +1023,7 @@ export function heartbeatService( sweepOrphanedActiveLeases, drainRunningRunsForShutdown, registerScheduledRetryCancellationWork, + registerRetainedNativeSessionCleanupWork, } = createHeartbeatRecovery(db, { pluginWorkerManager: options.pluginWorkerManager, enterShutdown: () => { shutdownInProgress = true; }, @@ -4092,6 +4093,9 @@ export function heartbeatService( triggerIssueMonitor, reportRunActivity: clearDetachedRunWarning, + registerRetainedNativeSessionCleanupWork: ( + coordinator: Parameters[0], canRun: () => boolean, + ) => registerRetainedNativeSessionCleanupWork(coordinator, () => canRun() && readTaskDrain(new Date()) === null), registerNativeStatusReassessmentWork: (coordinator: Parameters["register"]>[0], canRun: () => boolean) => nativeStatusReassessmentWork(db).register(coordinator, () => canRun() && readTaskDrain(new Date()) === null), registerNativeReviewResponseWork: (coordinator: Parameters[1], canRun: () => boolean) => diff --git a/server/src/services/heartbeat/recovery.test.ts b/server/src/services/heartbeat/recovery.test.ts index 3ab4dabb5fb..a98a413ef94 100644 --- a/server/src/services/heartbeat/recovery.test.ts +++ b/server/src/services/heartbeat/recovery.test.ts @@ -26,7 +26,7 @@ const native = vi.hoisted(() => ({ closeIdle: vi.fn(async () => ({ closed: 0, failed: 0 })), detach: vi.fn(async () => 0), finalizations: vi.fn(async (_db: Db, _ids?: string[], _options?: unknown) => undefined), - cleanups: vi.fn(async () => undefined), + cleanups: vi.fn<() => Promise>(async () => undefined), drainMaintenance: vi.fn<() => Promise>(async () => undefined), claims: vi.fn(async () => []), })); @@ -42,12 +42,12 @@ vi.mock("../native-runtime/index.js", async (importOriginal) => ({ closeIdleWarmNativeSessionsForRestart: native.closeIdle, detachNativeSessionsForRestart: native.detach, reconcileNativeFinalizations: native.finalizations, - reconcileRetainedNativeSessionCleanups: native.cleanups, claimNativeRestartRecoveries: native.claims, })); -vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({ - ...await importOriginal(), - drainRetainedRunnerdMaintenanceOperations: native.drainMaintenance, + +vi.mock("../native-runtime/retained-native-session-cleanup.js", async (importOriginal) => ({ + ...await importOriginal(), + reconcileRetainedNativeSessionCleanups: native.cleanups, })); function callbacks(db: Db) { @@ -107,7 +107,6 @@ beforeEach(() => { native.closeIdle.mockResolvedValue({ closed: 0, failed: 0 }); native.finalizations.mockResolvedValue(undefined); native.cleanups.mockResolvedValue(undefined); - native.drainMaintenance.mockResolvedValue(undefined); native.claims.mockResolvedValue([]); }); @@ -814,9 +813,9 @@ describePostgres("heartbeat recovery module ownership", () => { it("retains shutdown ownership until background native cleanup physically settles", async () => { const deps = callbacks(db); let settle!: () => void; - native.drainMaintenance.mockReturnValueOnce(new Promise(resolve => { settle = resolve; })); + native.cleanups.mockReturnValueOnce(new Promise(resolve => { settle = resolve; })); await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart(); - await vi.waitFor(() => expect(native.drainMaintenance).toHaveBeenCalled()); + await vi.waitFor(() => expect(native.cleanups).toHaveBeenCalled()); expect(deps.activeRunExecutionPromises.size).toBe(1); settle(); await Promise.all([...deps.activeRunExecutionPromises]); diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index e6c3e03c084..37628a4ca91 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -1,3 +1,5 @@ +import { retainedNativeSessionCleanupWork } from "../native-runtime/retained-native-session-cleanup-work.js"; +import { reconcileRetainedNativeSessionCleanups } from "../native-runtime/retained-native-session-cleanup.js"; import { createScheduledRetryCancellationRecovery } from "./scheduled-retry-cancellation.js"; import { pendingCleanupDeadlineSql } from "../environment-cleanup-state.js"; import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js"; @@ -57,9 +59,7 @@ import { type NativeRestartRecoveryClaim, reconcileNativeFinalizations, reconcileRetainedNativeSessionCleanup, - reconcileRetainedNativeSessionCleanups, } from "../native-runtime/index.js"; -import { drainRetainedRunnerdMaintenanceOperations } from "../../vendor/paperclip-runner/index.js"; import { runningProcesses } from "../../adapters/index.js"; import { parseObject } from "../../adapters/utils.js"; import { @@ -1920,12 +1920,23 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD ).catch(() => undefined); } + function cleanupRetainedNativeSession(input: { companyId: string; runId: string }) { + const operation = reconcileRetainedNativeSessionCleanup(db, input); + + + const tracked = operation.then(() => undefined, () => undefined); + activeRunExecutionPromises.add(tracked); + void tracked.then(() => activeRunExecutionPromises.delete(tracked)); + return operation; + } + const retainedNativeCleanup = retainedNativeSessionCleanupWork(db, cleanupRetainedNativeSession); + function scheduleRetainedNativeSessionCleanup() { // The per-database sweep joins startup and periodic callers. One bounded // control-only repair must not hold up unrelated provider ingress or the // entire orphan reaper, but shutdown must still await its physical owner. const cleanup = reconcileRetainedNativeSessionCleanups(db, { - cleanup: (input) => reconcileRetainedNativeSessionCleanup(db, input), + cleanup: cleanupRetainedNativeSession, onError: (error, runId) => { logger.warn( { err: error, runId }, @@ -1936,18 +1947,14 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD .then(() => undefined) .catch((error) => { logger.warn({ err: error }, "retained native cleanup discovery failed"); - }) - // The bounded maintenance attempt may fail before an already-started - // database callback settles. Keep shutdown ownership until the original - // operations finish; their timeout cannot authorize closing the database. - .finally(() => drainRetainedRunnerdMaintenanceOperations()); + }); activeRunExecutionPromises.add(cleanup); void cleanup.finally(() => activeRunExecutionPromises.delete(cleanup)); } let settledProcessesDiscovered = false; - async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean; skipNativeStatusReassessment?: boolean }) { + async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean; skipNativeStatusReassessment?: boolean; skipRetainedNativeSessionCleanup?: boolean }) { if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db); const staleThresholdMs = opts?.staleThresholdMs ?? 0; const now = new Date(); @@ -1981,7 +1988,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD "failed to reconcile persisted native finalizations before orphan reaping", ); }); - scheduleRetainedNativeSessionCleanup(); + if (!opts?.skipRetainedNativeSessionCleanup) scheduleRetainedNativeSessionCleanup(); await reconcileRetainedWorkspaces(db, environmentRuntime, resumeRemoteStopComments); await dispatchPendingNativeStatusWakeups().catch((error) => { logger.warn( @@ -2584,6 +2591,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD } return { + registerRetainedNativeSessionCleanupWork: retainedNativeCleanup.register, registerScheduledRetryCancellationWork: scheduledRetryCancellation.register, sweepPendingCleanupLeases, markNativeOwnershipUnverified, diff --git a/server/src/services/heartbeat/run-lifecycle.test.ts b/server/src/services/heartbeat/run-lifecycle.test.ts index 6d862394ed2..7efd1c71021 100644 --- a/server/src/services/heartbeat/run-lifecycle.test.ts +++ b/server/src/services/heartbeat/run-lifecycle.test.ts @@ -258,6 +258,38 @@ describe.skipIf(!support.supported)("heartbeat lifecycle database wiring", () => } finally { unsubscribe(); } }); + it.each(["setRunStatus", "setRunStatusIfRunning"] as const)("wakes retained cleanup only after the outer %s terminal write commits", async operation => { + const { run } = await fixture(); + const hints: unknown[] = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.retainedNativeSessionCleanup, (event, scopes) => { + if (event === "settled") for (const scope of scopes ?? []) hints.push(JSON.parse(scope)); + }); + try { + await db.transaction(async transaction => { + const tx = transaction as unknown as Db; + await createHeartbeatLifecycle(tx, callbacks(tx))[operation](run.id, "succeeded", { finishedAt: new Date() }); + expect(hints).toEqual([]); + expect((await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("succeeded"); + }); + expect(hints).toEqual([{ companyId: run.companyId, runId: run.id }]); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].finishedAt).not.toBeNull(); + hints.length = 0; + + await createHeartbeatLifecycle(db, callbacks(db)).setRunStatus(run.id, "succeeded", { error: "repaired metadata" }); + expect(hints).toEqual([{ companyId: run.companyId, runId: run.id }]); + } finally { unsubscribe(); } + }); + + it("does not wake retained cleanup for running progress", async () => { + const { run } = await fixture(); + const settled = vi.fn(); + const unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.retainedNativeSessionCleanup, settled); + try { + await createHeartbeatLifecycle(db, callbacks(db)).setRunStatus(run.id, "running"); + expect(settled).not.toHaveBeenCalled(); + } finally { unsubscribe(); } + }); + it("does not request accounting for an unchanged running status", async () => { const { run } = await fixture(); const settled = vi.fn(); diff --git a/server/src/services/heartbeat/run-lifecycle.ts b/server/src/services/heartbeat/run-lifecycle.ts index 0904b58a42f..43787f4ceea 100644 --- a/server/src/services/heartbeat/run-lifecycle.ts +++ b/server/src/services/heartbeat/run-lifecycle.ts @@ -1,3 +1,4 @@ +import { notifyRetainedNativeSessionCleanup } from "../retained-native-session-cleanup-notifications.js"; import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; @@ -1029,6 +1030,8 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl await notifyScheduledRetryCancellation(tx, { companyId: previous.companyId, runId: previous.id }); if (previous) await notifyInstructionCollection(tx, { companyId: previous.companyId, runId: previous.id }); if (previous) await notifyCancelledWorkspacePreservation(tx, { companyId: previous.companyId, runId: previous.id }); + if (previous && isHeartbeatRunTerminalStatus(status)) + await notifyRetainedNativeSessionCleanup(tx, { companyId: previous.companyId, runId: previous.id }); if (isHeartbeatRunTerminalStatus(status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, companyId); await notifyExecutionRunWork(tx); if (previous) await notifyChatRunMilestones(tx, { companyId: previous.companyId, runId: previous.id, diff --git a/server/src/services/native-finalization-notifications.ts b/server/src/services/native-finalization-notifications.ts index 3f95a4a67a4..2740baed7b4 100644 --- a/server/src/services/native-finalization-notifications.ts +++ b/server/src/services/native-finalization-notifications.ts @@ -1,3 +1,4 @@ +import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js"; import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyNativeReviewResponse } from "./native-review-response-notifications.js"; import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js"; @@ -7,6 +8,7 @@ export type NativeFinalizationWorkScope = { companyId: string; runId?: string; i export async function notifyNativeFinalizationWork(transaction: object, scope: NativeFinalizationWorkScope) { if (scope.issueId || scope.runId) { + await notifyRetainedNativeSessionCleanup(transaction, scope); await notifyNativeStatusReassessment(transaction, scope); await notifyNativeResponsePresentation(transaction, scope); await notifyNativeReviewResponse(transaction, scope); diff --git a/server/src/services/native-local-process-stop.ts b/server/src/services/native-local-process-stop.ts index 1ce70349175..0d31853d552 100644 --- a/server/src/services/native-local-process-stop.ts +++ b/server/src/services/native-local-process-stop.ts @@ -1,5 +1,6 @@ import { and, asc, desc, eq, gt, inArray, isNotNull, isNull, or, sql } from "drizzle-orm"; import { environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js"; import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js"; import { nativeSessionProviderEvidence } from "./native-runtime/native-session-resume.js"; import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js"; @@ -174,6 +175,7 @@ export async function releaseSettledNativeRunProcess(db: Db, runId: string): Pro identity.payload.processPid !== run.processPid || identity.payload.processGroupId !== run.processGroupId) return "skipped"; if (!(await recordNativeLocalProcessStop(tx, run))) return "pending"; } + await notifyRetainedNativeSessionCleanup(tx, { companyId: run.companyId, runId: run.id }); await tx.update(heartbeatRuns).set({ processPid: null, processGroupId: null, processStartedAt: null, updatedAt: new Date(), }).where(eq(heartbeatRuns.id, run.id)); diff --git a/server/src/services/native-runtime/native-finalization-reconciler.ts b/server/src/services/native-runtime/native-finalization-reconciler.ts index 48a31e52cc0..f8538648ff0 100644 --- a/server/src/services/native-runtime/native-finalization-reconciler.ts +++ b/server/src/services/native-runtime/native-finalization-reconciler.ts @@ -68,144 +68,7 @@ function record(value: unknown): Record { export type NativeSessionResumeClaim = { runId: string; leaseOwner: string }; -type NativeCleanupOutcome = { - runId: string; - status: "settled" | "not_eligible" | "operator_required"; -}; -type NativeCleanupSweep = { - cursor: string | null; - pending: Promise | null; -}; -const nativeCleanupSweeps = new WeakMap(); - -/** Candidate discovery is not cleanup authority. The exact-state operation - * claims its own durable lease and rechecks every physical owner. Keep this - * lane joined and bounded, and advance even past ineligible candidates so one - * damaged checkpoint cannot starve another company's recoverable session. */ -export function reconcileRetainedNativeSessionCleanups( - db: Db, - options: { - cleanup: (input: { - companyId: string; - runId: string; - }) => Promise; - onError?: (error: unknown, runId: string) => void; - limit?: number; - }, -): Promise { - let sweep = nativeCleanupSweeps.get(db); - if (!sweep) { - sweep = { cursor: null, pending: null }; - nativeCleanupSweeps.set(db, sweep); - } - if (sweep.pending) return sweep.pending; - const owned = sweep; - const limit = Number.isInteger(options.limit) - ? Math.max(1, Math.min(5, options.limit!)) - : 1; - const operation = async () => { - const selectCandidates = (cursor: string | null) => - db - .select({ - runId: heartbeatRuns.id, - companyId: heartbeatRuns.companyId, - }) - .from(heartbeatRuns) - .innerJoin( - nativeRunFinalizations, - and( - eq(nativeRunFinalizations.runId, heartbeatRuns.id), - eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId), - eq(nativeRunFinalizations.issueId, heartbeatRuns.nativeIssueId), - ), - ) - .innerJoin( - nativeRunResults, - and( - eq(nativeRunResults.id, nativeRunFinalizations.resultId), - eq(nativeRunResults.runId, heartbeatRuns.id), - eq(nativeRunResults.companyId, heartbeatRuns.companyId), - eq(nativeRunResults.issueId, heartbeatRuns.nativeIssueId), - ), - ) - .where( - and( - eq(heartbeatRuns.runtimeMode, "native"), - inArray(heartbeatRuns.status, ["succeeded", "failed"]), - isNotNull(heartbeatRuns.finishedAt), - nativeRunnerOwnershipNotHeldCondition(), - eq(nativeRunFinalizations.phase, "committed"), - eq(nativeRunResults.schemaStatus, "accepted"), - isNotNull(nativeRunFinalizations.assessmentId), - isNotNull(nativeRunFinalizations.decisionId), - isNull(nativeRunFinalizations.nextAttemptAt), - or( - isNull(nativeRunFinalizations.leaseOwner), - isNull(nativeRunFinalizations.leaseExpiresAt), - lte(nativeRunFinalizations.leaseExpiresAt, sql`now()`), - ), - // The accepted-result projector preserves a recovered close failure - // privately after clearing the visible successful run's stale error. - sql`coalesce(${heartbeatRuns.errorCode}, ${heartbeatRuns.resultJson}->'recoveredExecutionFailure'->>'errorCode') in ('adapter_failed', 'provider_transport_failed')`, - sql`coalesce(${heartbeatRuns.error}, ${heartbeatRuns.resultJson}->'recoveredExecutionFailure'->>'error') = 'provider_transport_failed: runner did not durably suspend before checkpoint'`, - sql`not (${nativeRunFinalizations.recoveryHistory} @> '[{"kind":"native_cleanup_runner_epoch"}]'::jsonb)`, - sql`not (${nativeRunFinalizations.recoveryHistory} @> '[{"kind":"native_cleanup_source_archive","phase":"operator_required"}]'::jsonb)`, - // One legacy pre-ownership attempt may be inspected by the closed, - // artifact-pinned no-launch verifier. Discovery grants no authority - // to reuse its directory or start a provider. New recorded epochs, - // staged attempts and ambiguous histories never enter this lane. - sql`( - select coalesce( - jsonb_array_length(history.entries) = 0 or ( - jsonb_array_length(history.entries) = 2 - and history.entries->0->>'version' = '1' - and history.entries->1->>'version' = '1' - and history.entries->0->>'phase' = 'started' - and history.entries->1->>'phase' = 'operator_required' - and history.entries->1->>'code' = 'native_cleanup_maintenance_unproven' - and history.entries->0->>'requestId' like 'native-cleanup:%' - and history.entries->0->>'requestId' = history.entries->1->>'requestId' - and history.entries->0->>'sourceFingerprint' ~ '^[a-f0-9]{64}$' - ), false - ) - from ( - select coalesce(jsonb_agg(entry.value order by entry.ordinal), '[]'::jsonb) as entries - from jsonb_array_elements(${nativeRunFinalizations.recoveryHistory}) - with ordinality as entry(value, ordinal) - where entry.value->>'kind' = 'native_cleanup_maintenance' - ) history - )`, - ...(cursor ? [gt(heartbeatRuns.id, cursor)] : []), - ), - ) - .orderBy(asc(heartbeatRuns.id)) - .limit(limit); - let candidates = await selectCandidates(owned.cursor); - if (candidates.length === 0 && owned.cursor !== null) { - owned.cursor = null; - candidates = await selectCandidates(null); - } - const outcomes: NativeCleanupOutcome[] = []; - for (const candidate of candidates) { - owned.cursor = candidate.runId; - try { - outcomes.push(await options.cleanup(candidate)); - } catch (error) { - options.onError?.(error, candidate.runId); - } - } - return outcomes; - }; - // Deferring the query one microtask installs the joined owner before any - // asynchronous work begins. Cleanup never creates a heartbeat or wake. - const pending = Promise.resolve() - .then(operation) - .finally(() => { - if (owned.pending === pending) owned.pending = null; - }); - owned.pending = pending; - return pending; -} +export { reconcileRetainedNativeSessionCleanups } from "./retained-native-session-cleanup.js"; export async function dispatchNativeSessionResumptions(input: { db: Db; diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index a1a6ac00c4e..5eff107290c 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -1,3 +1,4 @@ +import { notifyRetainedNativeSessionCleanup } from "../retained-native-session-cleanup-notifications.js"; import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; @@ -681,6 +682,7 @@ async function projectCommittedRun(input: { const now = new Date(); const [updatedRun] = await input.db.transaction(async transaction => { const tx = transaction as unknown as Db; + await notifyRetainedNativeSessionCleanup(tx, { companyId: input.run.companyId, runId: input.run.id }); await notifyInstructionCollection(tx, { companyId: input.run.companyId, runId: input.run.id }); await notifyCancelledWorkspacePreservation(tx, { companyId: input.run.companyId, runId: input.run.id }); await notifyUnsafeWorkspaceExportRunChange(tx, input.run); @@ -1746,6 +1748,7 @@ export async function finalizeNativeRun(input: { .returning(); const [updatedRun] = await input.db.transaction(async transaction => { const tx = transaction as unknown as Db; + await notifyRetainedNativeSessionCleanup(tx, { companyId: run.companyId, runId: run.id }); if (input.projectRunStatus) await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id }); if (input.projectRunStatus) await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id }); if (input.projectRunStatus) await notifyUnsafeWorkspaceExportRunChange(tx, run); diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 1c7e018ce20..eab9db6547b 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -1,3 +1,4 @@ +import { onRetainedNativeSessionCleanupRelease } from "../retained-native-session-cleanup-notifications.js"; import * as runnerCheckpoint from "./native-runner-checkpoint.js"; import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js"; import * as environmentTaskRuntime from "../environment-task-runtime.js"; @@ -34,6 +35,7 @@ import { inspect } from "node:util"; import { dirname, join } from "node:path"; import { heartbeatRuns, + environmentLeases, heartbeatRunEvents, issueThreadInteractions, issues, @@ -188,6 +190,7 @@ const state = vi.hoisted(() => ({ cleanup: vi.fn(), retireCleanup: vi.fn(), maintenanceIdle: vi.fn(() => true), + drainMaintenance: vi.fn(async (): Promise => undefined), createTransport: vi.fn((_options: RunnerTransportOptions) => ({ transport: {}, })), @@ -269,6 +272,7 @@ vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => { executeNativeSession: state.execute, settleRetainedRunnerdSession: state.cleanup, retainedRunnerdMaintenanceIsIdle: state.maintenanceIdle, + drainRetainedRunnerdMaintenanceOperations: state.drainMaintenance, completeRetainedNativeSessionCleanup: state.retireCleanup, parsePaperclipQuestionSet: (value: unknown) => value, }; @@ -403,6 +407,7 @@ import { import { onNativeRunnerEvidence } from "./native-runner-evidence-signal.js"; beforeEach(() => { + state.drainMaintenance.mockReset().mockResolvedValue(undefined); usageRecorder.capture.mockReset().mockImplementation(async receipt => receipt); usageRecorder.create.mockReset().mockResolvedValue({ capture: usageRecorder.capture, complete: vi.fn(async () => ({ complete: true })), persistFailure: vi.fn(async () => undefined) }); @@ -3880,6 +3885,35 @@ const execution = { } as NativeExecutionInputV1; describe("retained native cleanup activation", () => { + it("retains operation ownership through original callbacks after a cleanup failure", async () => { + const failure = new Error("cleanup database failure"); + const db = { transaction: vi.fn().mockRejectedValue(failure) } as unknown as Db; + let release!: () => void; + const original = new Promise(resolve => { release = resolve; }); + state.drainMaintenance.mockReturnValueOnce(original); + let finished = false; + const operation = reconcileRetainedNativeSessionCleanup(db, { companyId: "company", runId: "run" }) + .catch(error => error).finally(() => { finished = true; }); + try { + await vi.waitFor(() => expect(state.drainMaintenance).toHaveBeenCalledTimes(1)); + expect(finished).toBe(false); + release(); + expect(await operation).toBe(failure); + } finally { release(); await operation; } + }); + + it.each([ + Object.assign(new Error("database disconnected"), { code: "CONNECTION_CLOSED" }), + new Error("query failed", { cause: Object.assign(new Error("row locked"), { code: "55P03" }) }), + new Error("commit acknowledgement unavailable"), + ])("preserves a pre-claim database failure for recovery: %s", async failure => { + const db = { transaction: vi.fn().mockRejectedValue(failure) } as unknown as Db; + await expect(reconcileRetainedNativeSessionCleanup(db, { companyId: "company", runId: "run" })) + .rejects.toBe(failure); + expect(state.cleanup).not.toHaveBeenCalled(); + expect(state.retireCleanup).not.toHaveBeenCalled(); + }); + it.each([ "settled", "provider_transport_failed", @@ -3953,11 +3987,20 @@ describe("retained native cleanup activation", () => { "legacy_activation", "released_identity", "released_identity_unproven", + "invalid_execution_input", + "invalid_state_json", + "active_environment", + "active_finalization_lease", + "active_provider_process", ])("preserves exact original evidence for %s", async (mode) => { const directory = await mkdtemp( join(tmpdir(), "paperclip-maintenance-activation-"), ); let providerHomeDatabase: DatabaseSync | undefined; + let releaseMaintenance!: () => void; + const maintenance = new Promise(resolve => { releaseMaintenance = resolve; }); + let unsubscribeMaintenance: (() => void) | undefined; + const maintenanceReleased = vi.fn(); const preservedHomeFiles = [ "sessions/rollout-exact-thread.jsonl", "state_5.sqlite", @@ -4042,7 +4085,7 @@ describe("retained native cleanup activation", () => { payload: { providerSessionId: "exact-thread", providerAccountSessionId, - processId: 99_999_998, + processId: mode === "active_provider_process" ? process.pid : 99_999_998, }, }; const normalizedIdentity = { @@ -4223,13 +4266,14 @@ describe("retained native cleanup activation", () => { resultId: "result", assessmentId: "assessment", decisionId: "decision", - leaseOwner: null, - leaseExpiresAt: null, + leaseOwner: mode === "active_finalization_lease" ? "another-owner" : null, + leaseExpiresAt: mode === "active_finalization_lease" ? new Date(Date.now() + 60_000) : null, nextAttemptAt: null, recoveryHistory: [], }; let transactionOpen = false; let stopReceiptRead = false; + let environmentReleased = false; let releaseCommit!: () => void; const commitGate = new Promise((release) => { releaseCommit = release; @@ -4238,6 +4282,7 @@ describe("retained native cleanup activation", () => { select: () => ({ from: (table: unknown) => { const rows = + table === environmentLeases && mode === "active_environment" && !environmentReleased ? [{ id: "active-environment" }] : table === heartbeatRuns ? [run] : table === nativeRunFinalizations @@ -4679,6 +4724,8 @@ describe("retained native cleanup activation", () => { identity: { ...identity, runId: "foreign-run" }, }), ); + if (mode === "invalid_execution_input") run.runnerProfileJson.nativeExecutionInput = {} as typeof execution; + if (mode === "invalid_state_json") await writeFile(join(quarantine, source[0][0]), "{"); const original = await Promise.all( source.map(([file]) => readFile(join(quarantine, file), "utf8")), ); @@ -4966,6 +5013,10 @@ describe("retained native cleanup activation", () => { ).toBe("settled"); return 1; }); + if (mode === "legacy_busy_copy") { + state.drainMaintenance.mockReturnValueOnce(maintenance); + unsubscribeMaintenance = onRetainedNativeSessionCleanupRelease(db, maintenanceReleased); + } const pendingOutcome = reconcileRetainedNativeSessionCleanup( db as unknown as Db, { companyId: run.companyId, runId: run.id }, @@ -5013,7 +5064,25 @@ describe("retained native cleanup activation", () => { ).rejects.toThrow("native_session_supervisor_busy"); releaseCommit(); } + if (mode === "canonical_claim_commit_failure") { + await expect(pendingOutcome).rejects.toThrow("injected source claim commit failure"); + expect(state.cleanup).not.toHaveBeenCalled(); + expect(state.retireCleanup).not.toHaveBeenCalled(); + expect(coordinator.leaseOwner).toBeNull(); + expect(coordinator.recoveryHistory).toEqual([]); + expect(await Promise.all(source.map(([file]) => readFile(join(root, file), "utf8")))).toEqual(original); + return; + } + if (mode === "legacy_busy_copy") { + await vi.waitFor(() => expect(state.drainMaintenance).toHaveBeenCalledTimes(1)); + expect(maintenanceReleased).not.toHaveBeenCalled(); + releaseMaintenance(); + } const outcome = await pendingOutcome; + if (mode === "legacy_busy_copy") { + expect(maintenanceReleased).toHaveBeenCalledExactlyOnceWith({ companyId: run.companyId, runId: run.id }); + unsubscribeMaintenance?.(); + } if (canonicalSource) { const prepared = ( coordinator.recoveryHistory as Array> @@ -5049,6 +5118,14 @@ describe("retained native cleanup activation", () => { expect(cleanupEnvironment).not.toHaveProperty("OPENAI_API_KEY"); expect(cleanupEnvironment).not.toHaveProperty("CODEX_API_KEY"); } + const waitingFor = mode.endsWith("live_owner") || mode === "active_provider_process" ? "process_exit" + : mode === "legacy_busy_copy" ? "maintenance_settlement" + : mode === "active_environment" ? "environment_release" + : mode === "active_finalization_lease" ? "lease_expiry" : null; + if (waitingFor) { + expect(outcome).toMatchObject({ status: "deferred", waitFor: waitingFor, runId: run.id }); + if (waitingFor === "lease_expiry") expect(outcome).toHaveProperty("retryAt", coordinator.leaseExpiresAt); + } const ineligible = [ "canonical_live_owner", "canonical_foreign_owner", @@ -5078,6 +5155,8 @@ describe("retained native cleanup activation", () => { "foreign_semantic_scope", "wrong_provider_account", "released_identity_unproven", + "invalid_execution_input", + "invalid_state_json", ].includes(mode); const succeeds = [ "digest_only_command", @@ -5098,7 +5177,7 @@ describe("retained native cleanup activation", () => { "released_identity", ].includes(mode); expect(outcome.status).toBe( - succeeds + waitingFor ? "deferred" : succeeds ? "settled" : ineligible ? "not_eligible" @@ -5142,7 +5221,7 @@ describe("retained native cleanup activation", () => { "home_selected_reverted_rollout", ].includes(mode); expect(state.cleanup).toHaveBeenCalledTimes( - ineligible || deniedBeforeLaunch ? 0 : 1, + ineligible || waitingFor || deniedBeforeLaunch ? 0 : 1, ); expect(state.retireCleanup).toHaveBeenCalledTimes(succeeds ? 1 : 0); expect(coordinator.phase).toBe("committed"); @@ -5347,7 +5426,25 @@ describe("retained native cleanup activation", () => { ).toEqual(["started", "staged", "launch_intent", "operator_required"]); await expect(access(root)).rejects.toMatchObject({ code: "ENOENT" }); } + if (waitingFor && mode !== "active_provider_process") { + + + if (waitingFor === "process_exit") { + run.processPid = 99_999_999; + run.processGroupId = 99_999_999; + } + if (waitingFor === "maintenance_settlement") state.maintenanceIdle.mockReturnValue(true); + if (waitingFor === "environment_release") environmentReleased = true; + if (waitingFor === "lease_expiry") coordinator.leaseExpiresAt = new Date(Date.now() - 1000); + await expect(reconcileRetainedNativeSessionCleanup(db as unknown as Db, { + companyId: run.companyId, runId: run.id, + })).resolves.toEqual({ status: "settled", runId: run.id }); + expect(state.cleanup).toHaveBeenCalledTimes(1); + expect(state.retireCleanup).toHaveBeenCalledTimes(1); + } } finally { + releaseMaintenance(); + unsubscribeMaintenance?.(); providerHomeDatabase?.close(); proofSpy.mockRestore(); state.maintenanceIdle.mockReset().mockReturnValue(true); @@ -6297,11 +6394,16 @@ describe("native startup cancellation fence", () => { } throw new Error("provider should not have been submitted"); }); - await expect(executePaperclipNativeSession({ - db: leaseDb(execution, {}, resultJson), execution, runnerInstanceId: "startup-stop", - })).rejects.toThrow("native_cancellation_pending_recovery"); - expect(cancel).toHaveBeenCalledOnce(); - expect(submit).not.toHaveBeenCalled(); + const db = leaseDb(execution, {}, resultJson), released = vi.fn(); + const unsubscribe = onRetainedNativeSessionCleanupRelease(db, released); + try { + await expect(executePaperclipNativeSession({ + db, execution, runnerInstanceId: "startup-stop", + })).rejects.toThrow("native_cancellation_pending_recovery"); + expect(cancel).toHaveBeenCalledOnce(); + expect(submit).not.toHaveBeenCalled(); + expect(released).toHaveBeenCalledWith({ companyId: execution.binding.companyId, agentId: execution.binding.agentId }); + } finally { unsubscribe(); } }); }); @@ -9514,7 +9616,9 @@ describe("native warm session supervision", () => { }); it("reuses one session across distinct governed runs and closes it after idle expiry", async () => { - const close = vi.fn(async () => undefined); + let releaseClose!: () => void; + const closeBarrier = new Promise(resolve => { releaseClose = resolve; }); + const close = vi.fn(() => closeBarrier); const sharedSession = { close }; const base = { ...execution, @@ -9563,24 +9667,34 @@ describe("native warm session supervision", () => { return result; }); - await executePaperclipNativeSession({ - db: leaseDb(base), - execution: base, - runnerInstanceId: "runner", - }); - await executePaperclipNativeSession({ - db: leaseDb(second), - execution: second, - runnerInstanceId: "runner", - }); - expect(close).not.toHaveBeenCalled(); - await vi.waitFor( - () => - expect(close).toHaveBeenCalledWith({ - reason: "warm native session idle timeout", - }), - { timeout: 2_000 }, - ); + const db = leaseDb(base), released = vi.fn(); + const unsubscribe = onRetainedNativeSessionCleanupRelease(db, released); + try { + await executePaperclipNativeSession({ + db, + execution: base, + runnerInstanceId: "runner", + }); + await executePaperclipNativeSession({ + db: leaseDb(second), + execution: second, + runnerInstanceId: "runner", + }); + expect(close).not.toHaveBeenCalled(); + released.mockClear(); + await vi.waitFor( + () => + expect(close).toHaveBeenCalledWith({ + reason: "warm native session idle timeout", + }), + { timeout: 2_000 }, + ); + expect(released).not.toHaveBeenCalled(); + releaseClose(); + await vi.waitFor(() => expect(released).toHaveBeenCalledExactlyOnceWith({ + companyId: base.binding.companyId, agentId: base.binding.agentId, + })); + } finally { releaseClose(); unsubscribe(); } }); it("does not offer a quarantined warm session to the next run", async () => { @@ -12078,9 +12192,12 @@ describe("runnerd provider runtime wiring", () => { }); state.createBackend.mockClear(); state.createTransport.mockClear(); + const db = leaseDb(execution); + const released = vi.fn(); + const unsubscribe = onRetainedNativeSessionCleanupRelease(db, released); let settled = false; const preparation = createRunnerdBackend({ - db: leaseDb(execution), execution, runnerInstanceId: "runner-overlap", + db, execution, runnerInstanceId: "runner-overlap", runnerEnvironment, prepareRunnerEnvironment, runnerExecutionTarget: { kind: "remote", transport: "sandbox", providerKey: "daytona", @@ -12092,15 +12209,18 @@ describe("runnerd provider runtime wiring", () => { await assignedReady; expect(prepareRunnerEnvironment).toHaveBeenCalledOnce(); expect(settled).toBe(false); + expect(released).not.toHaveBeenCalled(); expect(state.createBackend).not.toHaveBeenCalled(); if (failure === "assigned") releaseAssigned(); else releaseEnvironment(); await new Promise(resolve => setImmediate(resolve)); expect(settled).toBe(false); + expect(released).not.toHaveBeenCalled(); expect(state.createBackend).not.toHaveBeenCalled(); releaseEnvironment(); releaseAssigned(); const result = await preparation; + expect(released).toHaveBeenCalledExactlyOnceWith({ companyId: execution.binding.companyId, agentId: execution.binding.agentId }); if (failure !== "success") { expect(result).toMatchObject({ error: { message: failure === "assigned" ? "assigned failed" : "environment failed" } }); expect(state.createBackend).not.toHaveBeenCalled(); @@ -12116,6 +12236,7 @@ describe("runnerd provider runtime wiring", () => { releaseEnvironment(); releaseAssigned(); await preparation; + unsubscribe(); } }); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index fe662bd7e09..be78a042cff 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,3 +1,5 @@ +import { releaseRetainedNativeSessionCleanup } from "../retained-native-session-cleanup-notifications.js"; +import type { NativeCleanupOutcome } from "./retained-native-session-cleanup.js"; import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js"; import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; @@ -132,6 +134,7 @@ import { resolveSourceCodexHome, readRunnerdArtifactBinding, retainedRunnerdMaintenanceIsIdle, + drainRetainedRunnerdMaintenanceOperations, settleRetainedRunnerdSession, validatePrpEvent, validatePrpStructuredRunResult, @@ -546,6 +549,7 @@ export type NativeInstructionWorkingCopy = { }; type WarmNativeSession = { + onReleased: () => void; pluginTaskId?: string; observabilityCompatibilityKey?: string; runId: string; @@ -593,6 +597,12 @@ async function closeWarmNativeSession(entry: WarmNativeSession, reason: string, } const warmNativeSessions = new Map(); +function forgetWarmNativeSession(scope: string, entry: WarmNativeSession) { + if (warmNativeSessions.get(scope) !== entry) return; + warmNativeSessions.delete(scope); + entry.onReleased(); +} + const pluginTaskSessionOwners = new WeakMap>(); // Closing a remote owner saves its checkpoint asynchronously. A new turn must // not inspect or quarantine that owner's state until the save has finished. @@ -630,7 +640,7 @@ export async function claimWarmNativeInstructionCopy(input: { entry.closeOnReleaseReason = reason; try { await closeWarmNativeSession(entry, reason); - if (warmNativeSessions.get(scope) === entry) warmNativeSessions.delete(scope); + forgetWarmNativeSession(scope, entry); } finally { entry.busy = false; } }; try { @@ -682,7 +692,7 @@ export async function reserveWarmNativeInstructionDirectory(input: { entry.closeOnReleaseReason = reason; try { await closeWarmNativeSession(entry, reason); - if (warmNativeSessions.get(id) === entry) warmNativeSessions.delete(id); + forgetWarmNativeSession(id, entry); } finally { // Failed containment remains registered and cannot be adopted. The next // admission retries retirement instead of launching over a live owner. @@ -765,10 +775,10 @@ async function closeIdleWarmNativeSessions(input: { closingWarmNativeSessions.set(sessionId, closing); try { await closing; - if (warmNativeSessions.get(sessionId) === entry) warmNativeSessions.delete(sessionId); + forgetWarmNativeSession(sessionId, entry); closed += 1; } catch { - if (!entry.instructionWorkingCopy?.runId) warmNativeSessions.delete(sessionId); + if (!entry.instructionWorkingCopy?.runId) forgetWarmNativeSession(sessionId, entry); failed += 1; } finally { entry.busy = false; @@ -1894,7 +1904,7 @@ async function retireSupersededWarmNativeSessions( entry.closeOnReleaseReason = "warm native session identity changed"; try { await closeWarmNativeSession(entry, entry.closeOnReleaseReason); - if (warmNativeSessions.get(scope) === entry) warmNativeSessions.delete(scope); + forgetWarmNativeSession(scope, entry); } finally { // A failed close retains the owner and prevents launch. A later attempt // must retry retirement, never adopt this partially closed transport. @@ -3043,14 +3053,22 @@ export async function verifyStoppedNativeSessionForReplacement( * inspection, not retried from an older snapshot with unknown process owners. */ export async function reconcileRetainedNativeSessionCleanup( db: Db, - input: { - companyId: string; - runId: string; - }, -): Promise<{ - status: "settled" | "not_eligible" | "operator_required"; - runId: string; -}> { + input: { companyId: string; runId: string }, +): Promise { + + + const outcome = await inspectRetainedNativeSessionCleanup(db, input) + .finally(() => drainRetainedRunnerdMaintenanceOperations()); + if (outcome.status === "deferred" && outcome.waitFor === "maintenance_settlement") { + releaseRetainedNativeSessionCleanup(db, input); + } + return outcome; +} + +async function inspectRetainedNativeSessionCleanup( + db: Db, + input: { companyId: string; runId: string }, +): Promise { const denied = () => new Error("native_cleanup_maintenance_unproven"); const leaseOwner = `native-cleanup:${randomUUID()}`; let reservedScope: string | null = null; @@ -3062,7 +3080,7 @@ export async function reconcileRetainedNativeSessionCleanup( executingRunnerdSessionScopes.delete(reservedScope); } }; - let claim: { + type CleanupClaim = { execution: NativeExecutionInput; run: typeof heartbeatRuns.$inferSelect; quarantine: string; @@ -3091,7 +3109,12 @@ export async function reconcileRetainedNativeSessionCleanup( itemId: string; }; history: Array>; - } | null = null; + }; + type DeferredCleanup = Extract; + const defer = (waitFor: Exclude): DeferredCleanup => ({ + runId: input.runId, status: "deferred", waitFor, + }); + let claim: CleanupClaim | DeferredCleanup | null = null; try { claim = await db.transaction(async (tx) => { const row = await tx @@ -3117,10 +3140,12 @@ export async function reconcileRetainedNativeSessionCleanup( !run.runnerInstanceId || !["succeeded", "failed"].includes(run.status) || !run.finishedAt || - !cleanupProcessAbsent(run.processPid) || + process.platform === "win32" || + !Number.isSafeInteger(run.processPid) || Number(run.processPid) <= 0 || run.processGroupId !== run.processPid ) return null; + if (!cleanupProcessAbsent(run.processPid)) return defer("process_exit"); const execution = parseNativeExecutionInput( record(run.runnerProfileJson).nativeExecutionInput, ); @@ -3149,7 +3174,7 @@ export async function reconcileRetainedNativeSessionCleanup( initializingSessionToolAuthorities.has(scope) || warmNativeSessions.has(scope) ) - return null; + return defer("session_release"); executingRunnerdSessionScopes.set(scope, leaseOwner); reservedScope = scope; const coordinator = await tx @@ -3172,14 +3197,14 @@ export async function reconcileRetainedNativeSessionCleanup( !coordinator.assessmentId || !coordinator.decisionId || coordinator.nextAttemptAt || - (coordinator.leaseOwner && - coordinator.leaseExpiresAt && - coordinator.leaseExpiresAt > new Date()) || coordinator.recoveryHistory.some( (event) => event.kind === "native_cleanup_runner_epoch", ) ) return null; + if (coordinator.leaseOwner && coordinator.leaseExpiresAt && coordinator.leaseExpiresAt > new Date()) { + return { runId: input.runId, status: "deferred" as const, waitFor: "lease_expiry" as const, retryAt: coordinator.leaseExpiresAt }; + } const result = await tx .select() .from(nativeRunResults) @@ -3206,7 +3231,7 @@ export async function reconcileRetainedNativeSessionCleanup( ), ) .limit(1); - if (environment.length) return null; + if (environment.length) return defer("environment_release"); const root = scopedRunnerdStateRoot(execution); const maintenanceHistory = coordinator.recoveryHistory.filter( (event) => event.kind === "native_cleanup_maintenance", @@ -3225,7 +3250,7 @@ export async function reconcileRetainedNativeSessionCleanup( let emptyRoot: ReturnType = null; let sourceDirectory: string; let quarantine: string; - let sourceArchive: NonNullable["sourceArchive"] = null; + let sourceArchive: CleanupClaim["sourceArchive"] = null; const rootExists = !!lstatSync(root, { throwIfNoEntry: false }); if (rootExists && !isSafeNativeStateDirectory(root)) return null; if (archiveHistory.length) { @@ -3352,7 +3377,7 @@ export async function reconcileRetainedNativeSessionCleanup( providerEvent.sourceKind !== "runner" || providerEvent.turnId !== identity.turnId || providerEvent.itemId !== identity.itemId || - !cleanupProcessAbsent(provider.processId) || + !Number.isSafeInteger(provider.processId) || Number(provider.processId) <= 0 || typeof provider.providerSessionId !== "string" || source.provider.threadId !== provider.providerSessionId || providerEvent.runId !== run.id || @@ -3361,6 +3386,7 @@ export async function reconcileRetainedNativeSessionCleanup( typeof providerEvent.sourceEventId !== "string" ) return null; + if (!cleanupProcessAbsent(provider.processId)) return defer("process_exit"); const persisted = await tx .select() .from(heartbeatRunEvents) @@ -3449,13 +3475,8 @@ export async function reconcileRetainedNativeSessionCleanup( ); if (attemptedNames.length !== 1) return null; const directory = resolve(runnerdStateBase(), attemptedNames[0]!); - if ( - !retainedRunnerdMaintenanceIsIdle(directory) || - lstatSync(resolve(directory, CLEANUP_ACTIVATION_FILE), { - throwIfNoEntry: false, - }) - ) - return null; + if (lstatSync(resolve(directory, CLEANUP_ACTIVATION_FILE), { throwIfNoEntry: false })) return null; + if (!retainedRunnerdMaintenanceIsIdle(directory)) return defer("maintenance_settlement"); const attempted = cleanupStateSnapshot(directory); const receipts = await tx .select() @@ -3557,14 +3578,27 @@ export async function reconcileRetainedNativeSessionCleanup( history, }; }); - } catch { + } catch (error) { releaseScope(); - return { status: "not_eligible", runId: input.runId }; + + + + const code = record(error).code; + if (error instanceof SyntaxError || code === "native_execution_input_invalid" + || code === "ENOENT" || code === "ENOTDIR" || code === "ELOOP" + || (error instanceof Error && ["native_cleanup_maintenance_unproven", "native_state_file_changed"].includes(error.message))) { + return { status: "not_eligible", runId: input.runId }; + } + throw error; } if (!claim) { releaseScope(); return { status: "not_eligible", runId: input.runId }; } + if ("status" in claim) { + releaseScope(); + return claim; + } const owned = claim; let stagingDirectory: string | null = null; let maintenanceStarted = !owned.sourceArchive; @@ -6527,9 +6561,9 @@ async function releaseWarmNativeSession( entry.busy = true; try { await closeWarmNativeSession(entry, entry.closeOnReleaseReason); - if (warmNativeSessions.get(sessionId) === entry) warmNativeSessions.delete(sessionId); + forgetWarmNativeSession(sessionId, entry); } catch (error) { - if (!entry.instructionWorkingCopy?.runId) warmNativeSessions.delete(sessionId); + if (!entry.instructionWorkingCopy?.runId) forgetWarmNativeSession(sessionId, entry); if (requireCleanClose) throw error; } finally { entry.busy = false; } return; @@ -6555,8 +6589,8 @@ function armWarmNativeSessionIdleTimer( current.busy = true; current.closeOnReleaseReason = "warm native session idle timeout"; const closing = closeWarmNativeSession(current, current.closeOnReleaseReason) - .then(() => { if (warmNativeSessions.get(sessionId) === current) warmNativeSessions.delete(sessionId); }) - .catch(() => { if (!current.instructionWorkingCopy?.runId && warmNativeSessions.get(sessionId) === current) warmNativeSessions.delete(sessionId); }) + .then(() => { forgetWarmNativeSession(sessionId, current); }) + .catch(() => { if (!current.instructionWorkingCopy?.runId) forgetWarmNativeSession(sessionId, current); }) .finally(() => { current.busy = false; if (closingWarmNativeSessions.get(sessionId) === closing) closingWarmNativeSessions.delete(sessionId); @@ -8003,6 +8037,9 @@ export async function executePaperclipNativeSession(input: { }) .catch(() => undefined); } + releaseRetainedNativeSessionCleanup(input.db, { + companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId, + }); } } @@ -8921,7 +8958,7 @@ async function executePaperclipNativeSessionWithinScope( entry.closeOnReleaseReason = "warm native session configuration changed"; try { await closeWarmNativeSession(entry, entry.closeOnReleaseReason, input.execution.binding.runId); - warmNativeSessions.delete(warmSessionId); + forgetWarmNativeSession(warmSessionId, entry); } finally { entry.busy = false; } if (preparedCopy) throw new Error("native_instruction_preparation_configuration_changed"); persistedWarmSession = loadWarmNativeCheckpoint( @@ -8938,7 +8975,7 @@ async function executePaperclipNativeSessionWithinScope( entry.closeOnReleaseReason = "warm instruction copy changed during preparation"; try { await closeWarmNativeSession(entry, entry.closeOnReleaseReason); - warmNativeSessions.delete(warmSessionId); + forgetWarmNativeSession(warmSessionId, entry); } finally { entry.busy = false; } throw new Error("native_instruction_preparation_copy_changed"); } @@ -9345,6 +9382,9 @@ async function executePaperclipNativeSessionWithinScope( existing.issueId = input.execution.binding.issueId; } else warmNativeSessions.set(warmSessionId, { + onReleased: () => releaseRetainedNativeSessionCleanup(input.db, { + companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId, + }), pluginTaskId: usesPluginTask && input.runnerExecutionTarget?.kind === "remote" && input.runnerExecutionTarget.transport === "sandbox" ? input.runnerExecutionTarget.sandboxLeaseAcquisition?.providerLeaseId : undefined, runId: input.execution.binding.runId, @@ -9392,7 +9432,7 @@ async function executePaperclipNativeSessionWithinScope( if (existing.idleTimer !== null) { clearTimeout(existing.idleTimer); } - warmNativeSessions.delete(warmSessionId); + forgetWarmNativeSession(warmSessionId, existing); } } if (session) { @@ -11852,6 +11892,9 @@ export async function createRunnerdBackend(input: { ); } finally { initializingSessionToolAuthorities.delete(sessionScopeId); + releaseRetainedNativeSessionCleanup(input.db, { + companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId, + }); } } diff --git a/server/src/services/native-runtime/native-workspace-export-resume.ts b/server/src/services/native-runtime/native-workspace-export-resume.ts index f6c150a29a1..c41930428d0 100644 --- a/server/src/services/native-runtime/native-workspace-export-resume.ts +++ b/server/src/services/native-runtime/native-workspace-export-resume.ts @@ -1,3 +1,4 @@ +import { notifyRetainedNativeSessionCleanup } from "../retained-native-session-cleanup-notifications.js"; import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js"; import { randomUUID } from "node:crypto"; import { and, eq, inArray, ne, sql } from "drizzle-orm"; @@ -107,20 +108,23 @@ export async function settleNativeWorkspaceExportResume(db: Db, lease: Lease, op const receipt = remoteTerminationReceipt(lease, options.receipt); const stopped = receipt?.state === "stopped"; const now = new Date(); - const [row] = await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ - status: stopped ? options.status ?? "released" : "pending_cleanup", cleanupStatus: stopped ? "success" : "failed", - releasedAt: now, lastUsedAt: now, updatedAt: now, - metadata: sql`(${environmentLeases.metadata} - 'remoteExecutionTermination' - ${stopped ? sql`- 'nativeWorkspaceExportResume'` : sql``}) || ${JSON.stringify({ - ...(stopped ? { remoteExecutionTermination: receipt } : {}), - pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0, - })}::jsonb`, - }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId), - eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.provider, lease.provider!), - eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), eq(environmentLeases.status, "pending_cleanup"), - sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.attemptId}`, - sql`${environmentLeases.metadata}->'nativeWorkspaceExportResume'->>'requestId' = ${marker.requestId}`, - sql`${environmentLeases.metadata}->>'pluginId' = ${marker.pluginId}`, - )).returning()); + const [row] = await writeEnvironmentCleanup(db, async tx => { + if (stopped) await notifyRetainedNativeSessionCleanup(tx, { companyId: lease.companyId, runId: lease.heartbeatRunId! }); + return tx.update(environmentLeases).set({ + status: stopped ? options.status ?? "released" : "pending_cleanup", cleanupStatus: stopped ? "success" : "failed", + releasedAt: now, lastUsedAt: now, updatedAt: now, + metadata: sql`(${environmentLeases.metadata} - 'remoteExecutionTermination' + ${stopped ? sql`- 'nativeWorkspaceExportResume'` : sql``}) || ${JSON.stringify({ + ...(stopped ? { remoteExecutionTermination: receipt } : {}), + pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0, + })}::jsonb`, + }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId), + eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.provider, lease.provider!), + eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), eq(environmentLeases.status, "pending_cleanup"), + sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.attemptId}`, + sql`${environmentLeases.metadata}->'nativeWorkspaceExportResume'->>'requestId' = ${marker.requestId}`, + sql`${environmentLeases.metadata}->>'pluginId' = ${marker.pluginId}`, + )).returning(); + }); return row ?? null; } diff --git a/server/src/services/native-runtime/retained-native-session-cleanup-work.ts b/server/src/services/native-runtime/retained-native-session-cleanup-work.ts new file mode 100644 index 00000000000..2a14f91c0ad --- /dev/null +++ b/server/src/services/native-runtime/retained-native-session-cleanup-work.ts @@ -0,0 +1,65 @@ +import type { Db } from "@paperclipai/db"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; +import { onRetainedNativeSessionCleanupRelease, type RetainedNativeSessionCleanupScope } from "../retained-native-session-cleanup-notifications.js"; +import { registerScopedDeadlineWork } from "../scoped-deadline-work.js"; +import { findRetainedNativeSessionCleanupCandidates, pendingRetainedNativeSessionCleanup, runRetainedNativeSessionCleanup, + type RetainedNativeSessionCleanup } from "./retained-native-session-cleanup.js"; + +type Entry = { id: string; companyId: string; issueId: string; agentId: string; dueAt: number }; + + +const PROCESS_EXIT_PROBE_MS = 30_000; + +export function retainedNativeSessionCleanupWork(db: Db, cleanup: RetainedNativeSessionCleanup) { + async function load(scope?: RetainedNativeSessionCleanupScope, signal?: AbortSignal, canRun = () => true): Promise { + const entries: Entry[] = []; + let after: string | undefined; + while (!signal?.aborted && canRun()) { + const rows = await findRetainedNativeSessionCleanupCandidates(db, { scope, after, limit: 100, includeLeased: true }); + for (const row of rows) entries.push({ id: row.runId, companyId: row.companyId, issueId: row.issueId, agentId: row.agentId, + dueAt: row.leaseOwner ? row.leaseExpiresAt?.getTime() ?? 0 : 0 }); + if (rows.length < 100) return entries; + after = rows[rows.length - 1].runId; + } + throw new Error("Retained native cleanup discovery stopped"); + } + async function check(entry: Entry, signal?: AbortSignal): Promise { + if (signal?.aborted) throw new Error("Retained native cleanup stopped"); + let pending = pendingRetainedNativeSessionCleanup(db, entry.id); + if (!pending) { + const [current] = await findRetainedNativeSessionCleanupCandidates(db, { + scope: { companyId: entry.companyId, runId: entry.id }, limit: 1, includeLeased: true, + }); + if (!current) return null; + if (current.leaseOwner && current.leaseExpiresAt && current.leaseExpiresAt.getTime() > Date.now()) { + return { ...entry, dueAt: current.leaseExpiresAt.getTime() }; + } + if (signal?.aborted) throw new Error("Retained native cleanup stopped"); + pending = runRetainedNativeSessionCleanup(db, { companyId: entry.companyId, runId: entry.id }, cleanup); + } + const result = await pending; + if (result.status !== "deferred") return null; + if (result.waitFor === "lease_expiry") return { ...entry, dueAt: result.retryAt.getTime() }; + if (result.waitFor === "process_exit") return { ...entry, dueAt: Date.now() + PROCESS_EXIT_PROBE_MS }; + + + return null; + } + function register(coordinator: ReturnType, canRun: () => boolean) { + const worker = registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.retainedNativeSessionCleanup, batchSize: 1, + decodeScope: (hint: string): RetainedNativeSessionCleanupScope | undefined => hint === "*" ? undefined : JSON.parse(hint), + matches: (entry: Entry, scope?: RetainedNativeSessionCleanupScope) => !scope || entry.companyId === scope.companyId + && (!scope.runId || entry.id === scope.runId) && (!scope.issueId || entry.issueId === scope.issueId) && (!scope.agentId || entry.agentId === scope.agentId), + load: (scope, signal) => load(scope, signal, canRun), + check: (entry, signal) => { + if (!canRun()) throw new Error("Retained native cleanup paused"); + return check(entry, signal); + }, + }, canRun); + const unsubscribe = onRetainedNativeSessionCleanupRelease(db, scope => worker.invalidate(JSON.stringify(scope))); + return { ...worker, stop: unsubscribe }; + } + return { load, check, register }; +} diff --git a/server/src/services/native-runtime/retained-native-session-cleanup.ts b/server/src/services/native-runtime/retained-native-session-cleanup.ts new file mode 100644 index 00000000000..3ddbe54b2c2 --- /dev/null +++ b/server/src/services/native-runtime/retained-native-session-cleanup.ts @@ -0,0 +1,200 @@ +import { and, asc, eq, gt, inArray, isNotNull, isNull, lte, or, sql } from "drizzle-orm"; +import { heartbeatRuns, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db"; +import { nativeRunnerOwnershipNotHeldCondition } from "./native-runner-ownership.js"; + +import type { RetainedNativeSessionCleanupScope } from "../retained-native-session-cleanup-notifications.js"; +export type { RetainedNativeSessionCleanupScope } from "../retained-native-session-cleanup-notifications.js"; + + + + +export function findRetainedNativeSessionCleanupCandidates(db: Db, options: { + scope?: RetainedNativeSessionCleanupScope; + after?: string; + limit: number; + includeLeased?: boolean; +}) { + return db + .select({ + runId: heartbeatRuns.id, + companyId: heartbeatRuns.companyId, + issueId: nativeRunFinalizations.issueId, + agentId: heartbeatRuns.agentId, + leaseOwner: nativeRunFinalizations.leaseOwner, + leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt, + }) + .from(heartbeatRuns) + .innerJoin( + nativeRunFinalizations, + and( + eq(nativeRunFinalizations.runId, heartbeatRuns.id), + eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId), + eq(nativeRunFinalizations.issueId, heartbeatRuns.nativeIssueId), + ), + ) + .innerJoin( + nativeRunResults, + and( + eq(nativeRunResults.id, nativeRunFinalizations.resultId), + eq(nativeRunResults.runId, heartbeatRuns.id), + eq(nativeRunResults.companyId, heartbeatRuns.companyId), + eq(nativeRunResults.issueId, heartbeatRuns.nativeIssueId), + ), + ) + .where( + and( + eq(heartbeatRuns.runtimeMode, "native"), + inArray(heartbeatRuns.status, ["succeeded", "failed"]), + isNotNull(heartbeatRuns.finishedAt), + nativeRunnerOwnershipNotHeldCondition(), + eq(nativeRunFinalizations.phase, "committed"), + eq(nativeRunResults.schemaStatus, "accepted"), + isNotNull(nativeRunFinalizations.assessmentId), + isNotNull(nativeRunFinalizations.decisionId), + isNull(nativeRunFinalizations.nextAttemptAt), + options.includeLeased ? undefined : or( + isNull(nativeRunFinalizations.leaseOwner), + isNull(nativeRunFinalizations.leaseExpiresAt), + lte(nativeRunFinalizations.leaseExpiresAt, sql`now()`), + ), + + + sql`coalesce(${heartbeatRuns.errorCode}, ${heartbeatRuns.resultJson}->'recoveredExecutionFailure'->>'errorCode') in ('adapter_failed', 'provider_transport_failed')`, + sql`coalesce(${heartbeatRuns.error}, ${heartbeatRuns.resultJson}->'recoveredExecutionFailure'->>'error') = 'provider_transport_failed: runner did not durably suspend before checkpoint'`, + sql`not (${nativeRunFinalizations.recoveryHistory} @> '[{"kind":"native_cleanup_runner_epoch"}]'::jsonb)`, + sql`not (${nativeRunFinalizations.recoveryHistory} @> '[{"kind":"native_cleanup_source_archive","phase":"operator_required"}]'::jsonb)`, + + + + + sql`( + select coalesce( + jsonb_array_length(history.entries) = 0 or ( + jsonb_array_length(history.entries) = 2 + and history.entries->0->>'version' = '1' + and history.entries->1->>'version' = '1' + and history.entries->0->>'phase' = 'started' + and history.entries->1->>'phase' = 'operator_required' + and history.entries->1->>'code' = 'native_cleanup_maintenance_unproven' + and history.entries->0->>'requestId' like 'native-cleanup:%' + and history.entries->0->>'requestId' = history.entries->1->>'requestId' + and history.entries->0->>'sourceFingerprint' ~ '^[a-f0-9]{64}$' + ), false + ) + from ( + select coalesce(jsonb_agg(entry.value order by entry.ordinal), '[]'::jsonb) as entries + from jsonb_array_elements(${nativeRunFinalizations.recoveryHistory}) + with ordinality as entry(value, ordinal) + where entry.value->>'kind' = 'native_cleanup_maintenance' + ) history + )`, + options.scope ? and( + eq(heartbeatRuns.companyId, options.scope.companyId), + options.scope.runId ? eq(heartbeatRuns.id, options.scope.runId) : undefined, + options.scope.issueId ? eq(nativeRunFinalizations.issueId, options.scope.issueId) : undefined, + options.scope.agentId ? eq(heartbeatRuns.agentId, options.scope.agentId) : undefined, + ) : undefined, + options.after ? gt(heartbeatRuns.id, options.after) : undefined, + ), + ) + .orderBy(asc(heartbeatRuns.id)) + .limit(options.limit); +} + +export type NativeCleanupOutcome = { + runId: string; + status: "settled" | "not_eligible" | "operator_required"; +} | { + runId: string; + status: "deferred"; + waitFor: "process_exit" | "session_release" | "environment_release" | "maintenance_settlement"; +} | { + runId: string; + status: "deferred"; + waitFor: "lease_expiry"; + retryAt: Date; +}; +export type RetainedNativeSessionCleanup = (input: { companyId: string; runId: string }) => Promise; + + + + + +const operations = new WeakMap; + runs: Map>; +}>(); +export function runRetainedNativeSessionCleanup(db: Db, input: { companyId: string; runId: string }, cleanup: RetainedNativeSessionCleanup) { + let owned = operations.get(db); + if (!owned) { owned = { tail: Promise.resolve(), runs: new Map() }; operations.set(db, owned); } + const existing = owned.runs.get(input.runId); + if (existing) return existing; + const pending = owned.tail.then(() => cleanup(input)).finally(() => { + if (owned.runs.get(input.runId) === pending) owned.runs.delete(input.runId); + if (!owned.runs.size && operations.get(db) === owned) operations.delete(db); + }); + owned.runs.set(input.runId, pending); + owned.tail = pending.then(() => undefined, () => undefined); + return pending; +} +export function pendingRetainedNativeSessionCleanup(db: Db, runId: string) { + return operations.get(db)?.runs.get(runId); +} + +type NativeCleanupSweep = { + cursor: string | null; + pending: Promise | null; +}; +const nativeCleanupSweeps = new WeakMap(); + + + + + +export function reconcileRetainedNativeSessionCleanups( + db: Db, + options: { + cleanup: RetainedNativeSessionCleanup; + onError?: (error: unknown, runId: string) => void; + limit?: number; + }, +): Promise { + let sweep = nativeCleanupSweeps.get(db); + if (!sweep) { + sweep = { cursor: null, pending: null }; + nativeCleanupSweeps.set(db, sweep); + } + if (sweep.pending) return sweep.pending; + const owned = sweep; + const limit = Number.isInteger(options.limit) + ? Math.max(1, Math.min(5, options.limit!)) + : 1; + const operation = async () => { + const selectCandidates = (cursor: string | null) => + findRetainedNativeSessionCleanupCandidates(db, { after: cursor ?? undefined, limit }); + let candidates = await selectCandidates(owned.cursor); + if (candidates.length === 0 && owned.cursor !== null) { + owned.cursor = null; + candidates = await selectCandidates(null); + } + const outcomes: NativeCleanupOutcome[] = []; + for (const candidate of candidates) { + owned.cursor = candidate.runId; + try { + outcomes.push(await runRetainedNativeSessionCleanup(db, { companyId: candidate.companyId, runId: candidate.runId }, options.cleanup)); + } catch (error) { + options.onError?.(error, candidate.runId); + } + } + return outcomes; + }; + + + const pending = Promise.resolve() + .then(operation) + .finally(() => { + if (owned.pending === pending) owned.pending = null; + }); + owned.pending = pending; + return pending; +} diff --git a/server/src/services/retained-native-session-cleanup-notifications.ts b/server/src/services/retained-native-session-cleanup-notifications.ts new file mode 100644 index 00000000000..a359e8ad2b2 --- /dev/null +++ b/server/src/services/retained-native-session-cleanup-notifications.ts @@ -0,0 +1,28 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +export type RetainedNativeSessionCleanupScope = { + companyId: string; + runId?: string; + issueId?: string; + agentId?: string; +}; + +export function notifyRetainedNativeSessionCleanup(tx: object, scope: RetainedNativeSessionCleanupScope) { + return notifyDeliveryWork(tx, DELIVERY_QUEUES.retainedNativeSessionCleanup, JSON.stringify({ companyId: scope.companyId, runId: scope.runId, issueId: scope.issueId, agentId: scope.agentId })); +} + + + +const listeners = new WeakMap void>>(); +export function onRetainedNativeSessionCleanupRelease(db: object, listener: (scope: RetainedNativeSessionCleanupScope) => void) { + let owned = listeners.get(db); + if (!owned) { owned = new Set(); listeners.set(db, owned); } + owned.add(listener); + return () => { + owned.delete(listener); + if (!owned.size && listeners.get(db) === owned) listeners.delete(db); + }; +} +export function releaseRetainedNativeSessionCleanup(db: object, scope: RetainedNativeSessionCleanupScope) { + for (const listener of listeners.get(db) ?? []) listener(scope); +} diff --git a/server/src/services/sandbox-stop-and-retain.ts b/server/src/services/sandbox-stop-and-retain.ts index acdb52295f9..2090ac4f7bd 100644 --- a/server/src/services/sandbox-stop-and-retain.ts +++ b/server/src/services/sandbox-stop-and-retain.ts @@ -1,3 +1,4 @@ +import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js"; import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js"; import { randomUUID } from "node:crypto"; import { and, eq, sql, type SQL } from "drizzle-orm"; @@ -126,28 +127,31 @@ export async function settleStopOnlyCleanup(db: Db, lease: Lease, options: { att const receipt = remoteTerminationReceipt(lease, options.receipt), stopped = receipt?.state === "stopped"; const now = new Date(); const removeIntent: SQL = stopped ? sql`- 'sandboxStopAndRetain'` : sql``; - const [updated] = await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({ - status: stopped ? "released" : "pending_cleanup", cleanupStatus: stopped ? "success" : "failed", - failureReason: stopped ? null : "sandbox_stop_pending", - releasedAt: now, lastUsedAt: now, updatedAt: now, - metadata: sql`(${environmentLeases.metadata} - 'remoteExecutionTermination' ${removeIntent}) || ${JSON.stringify({ - ...(stopped ? { remoteExecutionTermination: receipt, sandboxStopAndRetainReceipt: { - schema: "paperclip.sandbox-stop-and-retain-receipt.v1", requestId: intent.requestId, - companyId: lease.companyId, runId: lease.heartbeatRunId, leaseId: lease.id, - provider: lease.provider, providerLeaseId: lease.providerLeaseId, - ...(intent.pluginId ? { pluginId: intent.pluginId, method: "environmentStopLease" } - : { builtinProvider: lease.provider, method: "builtin.stopLease" }), - confirmedAt: receipt.confirmedAt, - } } : {}), - pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0, - })}::jsonb`, - }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId), - eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.provider, lease.provider!), - eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), eq(environmentLeases.status, "pending_cleanup"), - sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.attemptId}`, - sql`${environmentLeases.metadata}->'sandboxStopAndRetain' = ${JSON.stringify(lease.metadata?.[SANDBOX_STOP_AND_RETAIN_KEY])}::jsonb`, - sql`${environmentLeases.metadata}->>'pluginId' is not distinct from ${intent.pluginId}`, - )).returning()); + const [updated] = await writeEnvironmentCleanup(db, async tx => { + if (stopped) await notifyRetainedNativeSessionCleanup(tx, { companyId: lease.companyId, runId: lease.heartbeatRunId! }); + return tx.update(environmentLeases).set({ + status: stopped ? "released" : "pending_cleanup", cleanupStatus: stopped ? "success" : "failed", + failureReason: stopped ? null : "sandbox_stop_pending", + releasedAt: now, lastUsedAt: now, updatedAt: now, + metadata: sql`(${environmentLeases.metadata} - 'remoteExecutionTermination' ${removeIntent}) || ${JSON.stringify({ + ...(stopped ? { remoteExecutionTermination: receipt, sandboxStopAndRetainReceipt: { + schema: "paperclip.sandbox-stop-and-retain-receipt.v1", requestId: intent.requestId, + companyId: lease.companyId, runId: lease.heartbeatRunId, leaseId: lease.id, + provider: lease.provider, providerLeaseId: lease.providerLeaseId, + ...(intent.pluginId ? { pluginId: intent.pluginId, method: "environmentStopLease" } + : { builtinProvider: lease.provider, method: "builtin.stopLease" }), + confirmedAt: receipt.confirmedAt, + } } : {}), + pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0, + })}::jsonb`, + }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId), + eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.provider, lease.provider!), + eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), eq(environmentLeases.status, "pending_cleanup"), + sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.attemptId}`, + sql`${environmentLeases.metadata}->'sandboxStopAndRetain' = ${JSON.stringify(lease.metadata?.[SANDBOX_STOP_AND_RETAIN_KEY])}::jsonb`, + sql`${environmentLeases.metadata}->>'pluginId' is not distinct from ${intent.pluginId}`, + )).returning(); + }); return updated ?? null; } @@ -159,19 +163,22 @@ export async function settleAbsentSandbox(db: Db, lease: typeof environmentLease const termination = remoteTerminationReceipt(lease, { providerLeaseId: proof.providerLeaseId, state: "destroyed" }); if (!termination) return false; const now = new Date(); - const changed = await db.update(environmentLeases).set({ - status: "released", cleanupStatus: "success", failureReason: "workspace_source_unavailable", releasedAt: now, updatedAt: now, - metadata: sql`(${environmentLeases.metadata} - 'sandboxStopAndRetain' - 'nativeWorkspaceExportResume' - 'sandboxStopAndRetainReceipt') || ${JSON.stringify({ - remoteExecutionTermination: termination, sandboxAbsence: evidence, - pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0, - })}::jsonb`, - }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId), - eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.providerLeaseId, proof.providerLeaseId), - eq(environmentLeases.status, lease.status), - sql`exists (select 1 from ${heartbeatRuns} where ${heartbeatRuns.id} = ${lease.heartbeatRunId} - and ${heartbeatRuns.companyId} = ${lease.companyId} - and ${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceFinalizationOwner,token}' = ${proof.ownerToken})`, - sql`${environmentLeases.metadata} is not distinct from ${lease.metadata === null ? null : JSON.stringify(lease.metadata)}::jsonb`, - )).returning({ id: environmentLeases.id }); + const changed = await db.transaction(async tx => { + await notifyRetainedNativeSessionCleanup(tx, { companyId: lease.companyId, runId: lease.heartbeatRunId! }); + return tx.update(environmentLeases).set({ + status: "released", cleanupStatus: "success", failureReason: "workspace_source_unavailable", releasedAt: now, updatedAt: now, + metadata: sql`(${environmentLeases.metadata} - 'sandboxStopAndRetain' - 'nativeWorkspaceExportResume' - 'sandboxStopAndRetainReceipt') || ${JSON.stringify({ + remoteExecutionTermination: termination, sandboxAbsence: evidence, + pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0, + })}::jsonb`, + }).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId), + eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.providerLeaseId, proof.providerLeaseId), + eq(environmentLeases.status, lease.status), + sql`exists (select 1 from ${heartbeatRuns} where ${heartbeatRuns.id} = ${lease.heartbeatRunId} + and ${heartbeatRuns.companyId} = ${lease.companyId} + and ${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceFinalizationOwner,token}' = ${proof.ownerToken})`, + sql`${environmentLeases.metadata} is not distinct from ${lease.metadata === null ? null : JSON.stringify(lease.metadata)}::jsonb`, + )).returning({ id: environmentLeases.id }); + }); return changed.length === 1; }