mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 21:49:18 +01:00
fix(server): schedule retained native-session cleanup (#17717)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs can retain a local session after an interrupted durable suspend. > - The orphan reaper repeatedly scans completed history for that cleanup duty. > - Idle scans keep the database active. Removing them requires complete wake and ownership paths. > - This pull request moves retained-session cleanup to committed signals, local release events and saved deadlines. > - Existing physical proof checks still decide whether cleanup is safe. The shared scheduler does not grant execution authority. > - The benefit is quiet retained history with recoverable cleanup and owned shutdown. ## Linked Issues or Issue Description Refs #17709 and #18135. This is slice 51 of the existing recurring-server sleep stack. Both referenced dependencies have merged. The target is `master`. **What happened?** The periodic orphan pass scans retained native sessions even after their cleanup is complete or no automatic action is safe. Some eligible sessions also depend on a local session, environment lease or original maintenance callback releasing ownership. An earlier decision notification can be consumed before the run's later terminal projection commits. **Expected behavior** Reconstruct existing work at startup. Wake from committed writes and actual owner release. Register the saved claim expiry or a probe for a specific live process. Stop recurring discovery when no obligation remains. Preserve original callback ownership and all existing cleanup proof checks. **Steps to reproduce** 1. Retain completed native runs with a durable-suspend close failure. 2. Let the periodic orphan pass run with no new work. 3. Observe repeated cleanup discovery. Separately, delay terminal projection after its decision notification is consumed and observe the missing cleanup wake. **Paperclip version or commit** Parent #17709 at reviewed head `3b75f4bc1da1370304f069fc287af3d693575716`, landed as `df6156c2e37f542a69f393dde31e357b82b7cac1`. Current integrated master: `31b927e3013a8a4b3266854b740cf40a24753788`. ## What Changed - Move scoped, paged discovery to the retained cleanup owner. Preserve explicit manual recovery. - Register the duty with the shared scheduler and remove it from the periodic orphan pass. Keep concrete lease expiry and live-process probe obligations. - Use typed outcomes for settled/refused work and process, session, environment, maintenance or lease waits. Keep physical proof and quarantine decisions in the executor. - Wake from committed finalization and terminal projection writes, durable process/lease release, and local session or callback settlement. Routine company-only finalization hints leave dormant cleanup quiet. - Join duplicate cleanup and serialize inspections on one database lane. Keep original Runner callbacks owned until settlement and shutdown. - Preserve all master profile fixtures and the branch's historical identities. Save the colliding sleep profile 79 separately. Generate combined Copilot profile 80 from its source and retain master's other provider profiles. - Add the owner audit and regression coverage. No new table, migration or independent maintenance timer is added. ## Verification - Server acceptance: **2,342 distinct passing cases in 18 complete files**. The 14-file unaffected group contributes 1,532 passes. Full startup contributes 76, process recovery 410, environment service 51, and external-chat response 273. The 29-case cleanup worker suite and 977-case executor suite run in full. No case in these files is skipped. - Controlled scope regression: removing only the ordinary finalization issue/run guard wakes dormant retained cleanup; the same assertion passes after restoring it. A real local owner release still resumes the durable candidate. - Preserve the first broad failure: 1,607 passed and one startup shutdown case failed because the coordinator mock lacked `stop()`. Add only that mock method. All 76 unchanged startup cases then pass. Do not treat the first broad command as green. - Dedicated PostgreSQL 17 databases use the existing migration 413 schema. Embedded-only fixtures use the existing recovery helper with fresh per-call databases. All temporary imports are restored byte-exact. No assertion or timeout is relaxed. - Current-master integration preserves both complete source histories. The server, UI and database trees are identical to the server-tested source. All master historical profile fixtures remain byte-exact. The prepared sleep profile 79 remains independently inspectable. - Current runtime validation at `d08f764e3efe75e96929173afe7ee8739121ad9f` passes all 12 commands (22 installed ACPX/Grok contract cases and 26 native-provider Rust cases): `pnpm install --frozen-lockfile --ignore-scripts`; Copilot/Grok identity and generated profile checks; Runner and server typechecking; server build; Copilot/Pi installation, profile compatibility and transport settlement files (130 passed; one opt-in closed Pi snapshot case skipped because its distribution root is not configured); installed ACPX model/reconnect package contract and Grok identity tests; Rust native-provider backend tests; module boundaries and migration order. - Earlier 94-case installation/settlement acceptance applies to the prepared profile 79 source. It is retained as history and does not substitute for the current profile 80 validation above. - Failure history and duty-specific evidence remain in `doc/plans/2026-10-26-retained-native-session-cleanup-sleep-audit.md`. Exact-current-head acceptance: all 60 checks are clean, including protected CI and security; Greptile gives 5/5 for `281d579de0d08e907194b8b790a0597b07d16b7b` with no actionable findings. No Ponytail review is present under the authorized missing-review policy. - These checks do not claim a full local repository suite, live Copilot or Daytona qualification, deployed host/database suspension, or measured cost savings. ## Risks - Missing a supported producer could strand cleanup. Tests exercise real finalizer, heartbeat, environment and process writers. Process-local signals cover in-process writers; no multi-replica or external SQL notification transport is added. - A timeout can leave an original callback active. Cleanup and shutdown must keep waiting for it. The existing global maintenance drain is conservative across roots. - A retained live process still has a scoped 30-second absence probe. A saved lease keeps its actual deadline. Operational errors retain retry work. Immutable proof refusals and operator-required histories do not gain automatic retry authority. - Cleanup keeps all existing artifact, identity, claim, no-launch and quarantine checks. Terminal status alone does not authorize physical cleanup. - This PR does not expand Cloud alarm coverage or relax host sleep eligibility. It proves worker behavior, not deployed app/database suspension. - The Copilot source identity has changed; these checks do not claim new live provider qualification. - The parent has merged. This PR is merged only after its current-head protected CI and review gates pass under the existing campaign authorization. ## Model Used OpenAI Codex, GPT-6 family, with code execution and repository tools. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com>
This commit is contained in:
40 files changed
+2775
-358
No files matched your search
@@ -3052,3 +3052,115 @@ The owner audit preserves the wrong initial ordinary-release completion assertio
|
||||
and its authority-preserving correction. Fresh-head gates remain pending; nothing
|
||||
merged. Earlier E2E failures on #17490/#17572 and voice findings on #17576/#17580
|
||||
were revalidated and remain unresolved. All original remaining duties stay in scope.
|
||||
|
||||
|
||||
Slice 50 repaired-head acceptance: #17709 at
|
||||
`9a9e4262bb9beb39d67d26320c14e99651b6d6bd` now has all 53 checks passing,
|
||||
Greptile 5/5 at that exact head and zero unresolved review threads. Its PR body
|
||||
records the result and retains the local embedded-database limitation. It remains
|
||||
draft and unmerged. `/tmp/sleep50-release-latest-gates.json` records fresh CI.
|
||||
Older stack findings and master reconciliation remain outstanding.
|
||||
|
||||
Slice 51 is now `codex/sleep-51-retained-native-session-cleanup`, based on that
|
||||
repaired head. The existing joined discovery has a cohesive native cleanup owner;
|
||||
scoped reconstruction preserves future lease deadlines. All 35 database discovery/
|
||||
finalization cases, 72 exact physical-cleanup cases and server types pass. See
|
||||
[the audit](2026-10-26-retained-native-session-cleanup-sleep-audit.md) for producer,
|
||||
physical-owner, shutdown and outcome gaps. No scheduled cleanup worker is wired
|
||||
and periodic cleanup remains enabled. The full scope remains incomplete.
|
||||
|
||||
|
||||
Slice 51 outcome checkpoint: pre-claim database/commit failures now propagate
|
||||
instead of being discarded as ineligible history. Typed waiting outcomes preserve
|
||||
process/session/environment/maintenance ownership and the actual coordinator lease
|
||||
expiry. Four injected failures, a real PostgreSQL row-lock regression and six
|
||||
waiting-state regressions reproduced the old behavior. The repaired source passes
|
||||
798 executor cases (15 embedded-DB cases excluded), 36 database cases and server
|
||||
types; strengthened physical reactivation tests pass all 80 selected cases again.
|
||||
The owner audit preserves the reused-fixture failure and initial fixture type error.
|
||||
Worker registration, producer/release notifications, quiet/lifecycle/cost proof,
|
||||
periodic removal and a slice 51 PR remain outstanding. No polling duty is claimed
|
||||
removed by this checkpoint; all original stack scope stays active and unmerged.
|
||||
|
||||
|
||||
Slice 51 worker checkpoint: retained cleanup now has scoped shared-scheduler
|
||||
reconstruction, exact lease deadlines, a concrete process-exit probe and event
|
||||
waiting for other owners. Startup/manual/scheduled attempts join duplicates and
|
||||
serialize different inspections. A real finalization-notification missed wake was
|
||||
reproduced and repaired through the existing transaction fanout. All 19 worker
|
||||
lifecycle cases, 36 existing cleanup/finalization cases, 37 reassessment consumer
|
||||
cases and server types pass. The 103-item backlog becomes SQL-quiet after settlement.
|
||||
Actual local/environment/process/maintenance producer coverage, physical callback
|
||||
drain ownership, large-history costs, startup wiring, periodic removal, publication
|
||||
and remote acceptance remain open. Periodic cleanup is still enabled; no new PR
|
||||
or merge is claimed and the full original stack scope stays active.
|
||||
|
||||
|
||||
Slice 51 durable release checkpoint: six real-producer missed wakes were
|
||||
reproduced and repaired: single/run-wide lease release, settled process release,
|
||||
stop-only cleanup, export-resume settlement and confirmed sandbox absence.
|
||||
Notifications remain run/company-scoped and tied to outer transaction ownership.
|
||||
All 208 selected worker/environment/workspace consumer cases and server types pass.
|
||||
The owner audit retains the asynchronous rollback-test failure and correction,
|
||||
and distinguishes controlled stop evidence from actual physical/provider proof.
|
||||
Local session and maintenance releases, remaining finalization producers, callback
|
||||
ownership, startup wiring, cost proof, periodic removal and PR gates remain open.
|
||||
Periodic cleanup remains enabled. No PR or merge; all original scope stays open.
|
||||
|
||||
|
||||
Slice 51 local owner checkpoint: session execution/initialization releases and
|
||||
all warm-session retirement paths now emit local scoped hints. Warm removal checks
|
||||
the expected owner; the existing close/retention decisions remain authoritative.
|
||||
Three lifecycle regressions reproduced missing release notifications. Final
|
||||
executor verification passes 798 cases (15 embedded-DB cases excluded), the real
|
||||
PostgreSQL worker suite passes 26 cases, and server types pass. Retained maintenance
|
||||
completion and physical callback ownership are next; the audit records the exact
|
||||
boundary gap. Startup registration, periodic removal, cost proof, remaining
|
||||
producers and PR acceptance remain incomplete. Nothing published or merged;
|
||||
all original recurring-server duties and older stack gates remain open.
|
||||
|
||||
|
||||
Slice 51 maintenance and terminal-producer checkpoint: cleanup now owns original
|
||||
Runner maintenance attempts/callbacks through settlement, using Runner's existing
|
||||
drain boundary. Finalizer projection and heartbeat terminal writes now notify
|
||||
cleanup in their transactions. Three reproduced producer regressions include the
|
||||
real separate-commit decision/projection race. All 64 worker/lifecycle cases,
|
||||
146 finalizer corpus cases, 799 selected executor cases, 19 Runner settlement
|
||||
cases and server types pass; 15 embedded-DB cases remain excluded for the recorded
|
||||
host limitation. The audit records earlier failures and fixture corrections.
|
||||
Startup wiring, query cost, periodic removal, full gates and publication remain.
|
||||
No PR or merge; the complete recurring-server scope and older stack gates stay open.
|
||||
|
||||
|
||||
Slice 51 startup integration: the enabled server now registers retained cleanup
|
||||
and its shutdown subscription. The periodic orphan pass skips this duty; explicit
|
||||
and startup recovery retain their bounded joined sweep. Heartbeat joins original
|
||||
scheduled physical operations through shutdown. All 100 startup/worker cases and
|
||||
server types pass. After the user's explicitly approved cleanup of 11 orphaned
|
||||
semaphore sets, the full executor suite passes all 814 cases with no exclusions,
|
||||
including strengthened initialization-release coverage. A 10,000-completed-session
|
||||
probe measures broad discovery at 28–30 ms locally and run-scoped discovery at
|
||||
0.030 ms; broad discovery remains linear in history and no deployed claim is made.
|
||||
Full workspace acceptance, PR publication/review, older gates and every remaining
|
||||
duty stay open. No merge has occurred.
|
||||
|
||||
|
||||
Slice 51 publication: [#17717](https://github.com/paperclipai/paperclip/pull/17717)
|
||||
is draft on #17709. Published head `8c62a6b0c0` passed workspace types/build and
|
||||
received Greptile 5/5, but three CI gates failed: two tests retained the old
|
||||
heartbeat-owned maintenance drain, and the Copilot transport identity was stale.
|
||||
The repair follows the cleanup-owned lifetime and creates a new attestation while
|
||||
preserving old profile history. Full affected recovery/process suites (34 + 408)
|
||||
and profile tests (69) pass. The local whole-suite OAuth timeout is retained in
|
||||
the audit alongside a passing complete 472-case isolated rerun. Latest-head CI
|
||||
and review acceptance are still pending; no stack PR is merged.
|
||||
|
||||
The next native finalization audit identifies three distinct remaining duties:
|
||||
workspace copy-back/validation; accepted-result commit/projection; and resource
|
||||
settlement after finalization. Current committed-history upkeep repeatedly calls
|
||||
finalization and best-effort cleanup, including swallowed resource-release
|
||||
failures. A completed result cannot be used as proof that those residual duties
|
||||
are finished. Audit first-result publication in `paperclip-control-plane-port.ts`,
|
||||
workspace operation writes, durable workspace-owner release, retry updates and
|
||||
cleanup completion before disabling their existing recovery paths. No migration
|
||||
of these next duties is claimed by slice 51.
|
||||
@@ -0,0 +1,620 @@
|
||||
# Retained native-session cleanup: sleep migration audit
|
||||
|
||||
Parent: slice 50, #17709 at `9a9e4262bb9beb39d67d26320c14e99651b6d6bd`.
|
||||
Branch: `codex/sleep-51-retained-native-session-cleanup`.
|
||||
Inspected master: `1fbbef45a8d399771add7a27a04b12f3a20e264c`.
|
||||
|
||||
## Obligation and ownership
|
||||
|
||||
A finished native run with an accepted result can retain an exact local checkpoint
|
||||
following the specific durable-suspend close failure. Cleanup may settle that
|
||||
checkpoint only after its existing artifact, journal, identity, lease and physical
|
||||
owner proofs succeed. A terminal run alone grants no cleanup authority. Failed
|
||||
maintenance with uncertain provider ownership stays quarantined for inspection.
|
||||
|
||||
The old orphan pass calls a joined, per-database discovery sweep on every pass;
|
||||
startup native recovery calls it too. Its cursor wraps after the last row, so an
|
||||
ineligible retained history is inspected repeatedly. The sweep takes at most five
|
||||
candidates (normally one). It keeps one pending promise per database and continues
|
||||
past a failed candidate. Heartbeat tracks the attempt in its active execution
|
||||
promises and joins retained runner maintenance callbacks before releasing shutdown
|
||||
ownership. Preserve these lifecycle guarantees when moving to the shared scheduler.
|
||||
|
||||
The discovery and scheduling owner is now
|
||||
`server/src/services/native-runtime/retained-native-session-cleanup.ts`.
|
||||
`native-finalization-reconciler.ts` retains its export for compatibility; heartbeat
|
||||
imports the cohesive owner directly. Provider-specific exact-state cleanup remains
|
||||
in `native-session-executor.ts`, which already owns the physical session and its
|
||||
admission rules. No provider protocol belongs in index.ts or the generic scheduler.
|
||||
|
||||
## Current implementation checkpoint
|
||||
|
||||
- Extracted existing bounded, joined discovery without changing its behavior.
|
||||
- Added company/run/task/agent-scoped, keyset-paged candidate discovery. It returns
|
||||
existing lease metadata. Scheduled reconstruction can include future leases;
|
||||
ordinary sweep callers still exclude an unexpired claim.
|
||||
- Candidate discovery still excludes nonterminal runs, unaccepted results,
|
||||
uncommitted assessments/decisions, saved finalization retry work, recorded runner
|
||||
epochs, operator-required source archives and ambiguous maintenance history.
|
||||
- The exact legacy two-entry pre-ownership maintenance exception remains unchanged.
|
||||
- A separate shared-scheduler worker now consumes scoped candidates and typed
|
||||
outcomes. Its lease deadline comes from durable state; a concrete unowned PID
|
||||
gets a scoped 30-second absence probe. Session/environment/maintenance waits
|
||||
need release notifications; ineligible history has no periodic fallback.
|
||||
- The existing finalization notification fanout includes cleanup. There is also
|
||||
a per-database local release subscription boundary. Durable owner releases,
|
||||
local initialization/execution/warm-session releases and finalizer/heartbeat
|
||||
terminal projection are covered. The worker is registered in server startup.
|
||||
- No new table, migration, independent timer or provider operation is added.
|
||||
- **The periodic orphan pass now skips this cleanup duty. Startup and explicit
|
||||
manual recovery remain supported. Tests establish worker quiet under named
|
||||
conditions; deployed app/database suspension has not been qualified.**
|
||||
|
||||
## Reactivation and outcome audit still required
|
||||
|
||||
| Condition or producer | Existing owner / evidence | Required handling before periodic removal |
|
||||
| --- | --- | --- |
|
||||
| Accepted result, committed decision, terminal projection or recovered close failure | Finalizer, status committer, executor; existing native-finalization notification fanout | Audit every writer, register cleanup intent in the owning transaction, and test real commit/rollback |
|
||||
| Coordinator claim expires or clears | Native finalization row | Reconstruct its actual lease deadline; recheck claim ownership at execution; wake on early release |
|
||||
| Active/pending-cleanup environment lease releases | Environment runtime, workspace settlement and cleanup writers | Audit release/deletion writers, not only pending-cleanup notifications; wake affected runs after commit |
|
||||
| Process identity is cleared after verified stop | `native-local-process-stop.ts`, recovery and executor | Retain server-authored stop evidence and same-transaction notification; absence alone never authorizes replay |
|
||||
| Same session is active, initializing or warm | Executor-owned maps and release/close paths | Use local completion/release signals with company/agent or narrower scope; a different run can own the same session |
|
||||
| Prior retained callback is still running | Runner maintenance operation registry and drain function | Preserve original promise ownership and reactivate after settlement; a timeout is not proof the callback stopped |
|
||||
| Relevant PID is still live without a controllable local owner | Exact cleanup's process check | Decide the explicit scoped probe contract; if needed, retain only a concrete unresolved obligation, with stop condition and documented sleep impact |
|
||||
| Database failure or NOWAIT contention before claim | Executor now propagates operational failures; real lock regression passes | Shared worker must retain recovery and reconstruct durable state, especially after an unknown commit; it must not blindly replay physical effects |
|
||||
| Invalid/foreign/changed journal, contract, provider state or artifacts | Existing exact-state verifier | Preserve refusal and original evidence; no automatic provider launch or broad retry of unknown outcomes |
|
||||
| Cleanup already attempted with unknown outcome | Durable epoch/maintenance/archive history | Keep quarantine and existing operator-required policy, with no new automatic retry |
|
||||
|
||||
The original `not_eligible` outcome conflated transient local ownership, database
|
||||
errors and permanent physical refusal. The operation now returns a typed deferred
|
||||
outcome identifying process exit, session release, environment release, retained
|
||||
maintenance settlement or a real coordinator lease expiry. The lease outcome
|
||||
carries the saved expiry. Operational database/commit/unexpected I/O errors propagate;
|
||||
malformed stored input, malformed JSON and missing/unsafe/changed proof artifacts
|
||||
remain ineligible. Post-claim failures still retain the existing operator-required
|
||||
quarantine policy. No new physical retry is authorized by this outcome contract.
|
||||
|
||||
The shared worker consumes this contract, but is not enabled in startup yet.
|
||||
Before enabling it, complete the actual release producers for each waiting outcome.
|
||||
It must not drop deferred work without the corresponding release notification or
|
||||
concrete probe/deadline, and must not retry every ineligible retained history. A valid PID whose absence
|
||||
cannot be proven remains deferred and cannot grant cleanup authority. The supported
|
||||
process-exit observation and local session-release contracts remain to implement.
|
||||
|
||||
The implementation uses the existing scoped deadline worker for reconstruction,
|
||||
coalescing, fair bounded dispatch, retries and drain retention. Eligibility and
|
||||
execution remain with the native cleanup owner. Duplicate startup/manual/scheduled
|
||||
attempts join their original per-run promise; different cleanup inspections share
|
||||
one per-database FIFO lane. Production wiring must provide a cleanup callback that
|
||||
retains ownership through the original physical callbacks, including ones that
|
||||
outlive a bounded maintenance timeout. Tests currently prove the promise-joining
|
||||
boundary; heartbeat's retained-callback drain still needs integration.
|
||||
Preserve startup/manual recovery compatibility while removing only this duty from
|
||||
the periodic orphan pass after coverage is complete.
|
||||
|
||||
Process-local transaction notifications cover supported in-process writers only.
|
||||
No independent SQL-writer or multi-replica notification transport is being added.
|
||||
Cloud alarm coverage and sleep eligibility remain unchanged and conservative.
|
||||
|
||||
## Evidence
|
||||
|
||||
All commands ran in the dedicated stack worktree; no source edits occurred while
|
||||
local acceptance processes were running. External test databases are dedicated
|
||||
scratch databases, avoiding the separately diagnosed embedded-PostgreSQL host
|
||||
semaphore limit. No shared OS semaphore was removed.
|
||||
|
||||
- Before extraction: all 32 existing `native-finalization-recovery.test.ts` cases
|
||||
pass. `/tmp/sleep51-before.log` and `.status` (0).
|
||||
- After extraction: the same 32 pass against a second fresh database.
|
||||
`/tmp/sleep51-after.log` and `.status` (0).
|
||||
- All 72 retained native cleanup activation cases pass, including exact original
|
||||
evidence, foreign/live owners, archive/claim commit failures, stalled commits,
|
||||
lease loss, changed artifacts, quarantine and stopped-identity proof. Other
|
||||
executor tests were intentionally not selected (733 skipped).
|
||||
`/tmp/sleep51-physical.log` and `.status` (0).
|
||||
- Scoped discovery adds three database tests: future lease reconstruction without
|
||||
early legacy execution; company/run/task/agent isolation with a real second
|
||||
tenant; bounded keyset traversal excluding already-attempted maintenance.
|
||||
All 35 finalization/discovery cases pass on a third fresh database.
|
||||
`/tmp/sleep51-discovery.log` and `.status` (0).
|
||||
- Server typecheck passes after extraction and after scoped discovery.
|
||||
`/tmp/sleep51-types.log` and `/tmp/sleep51-discovery-types.log`, both status 0.
|
||||
|
||||
These are 107 distinct selected passing cases on the final source (35 discovery/
|
||||
finalization and 72 physical-operation cases; the physical owner was unchanged
|
||||
by the subsequent discovery extension). This is not full-suite acceptance,
|
||||
worker lifecycle proof, deployed sleep proof or PR readiness.
|
||||
|
||||
## Remaining acceptance
|
||||
|
||||
- [x] Explicit transient/dormant/terminal cleanup outcomes and operational failures.
|
||||
- [x] Producer audit and real commit/rollback/reactivation tests for the supported in-process writers.
|
||||
- [x] Local session-release and retained-callback completion ownership.
|
||||
- [x] Shared worker startup, first producer, retry, changes during sweep, drain,
|
||||
resume and shutdown; prove SQL/network quiet with retained history.
|
||||
- [x] Discovery/backlog cost and fairness under the measured 10,000-history fixture.
|
||||
- [x] Startup/periodic wiring and removal of this duty from the orphan pass.
|
||||
- [ ] Applicable broader verification, complete PR template, CI and 5/5 review.
|
||||
|
||||
All other duties in the recurring-server stack remain in scope. No merge is
|
||||
requested or authorized by this migration checkpoint.
|
||||
|
||||
|
||||
## Outcome contract checkpoint
|
||||
|
||||
The retained cleanup owner now distinguishes waiting, refusal and operational
|
||||
failure. This is a prerequisite to scheduling; periodic discovery is still enabled.
|
||||
The legacy sweep continues to isolate and log a failed candidate, advance its cursor,
|
||||
and release its joined owner. Shared scheduling and producer coverage remain pending.
|
||||
|
||||
Regression evidence:
|
||||
|
||||
- Three injected transaction failures (disconnection, nested NOWAIT, missing commit
|
||||
acknowledgement) and the existing rolled-back source-claim commit failure all
|
||||
incorrectly returned `not_eligible` before repair. Four expected failures in
|
||||
`/tmp/sleep51-outcome-before.log` / `.status` (1); 73 other selected cases passed.
|
||||
- The first real lock invocation reused a database with company prefix `NCS` and
|
||||
failed fixture setup with unique violation 23505. All 36 tests were skipped;
|
||||
this was not product evidence. `/tmp/sleep51-lock-before.log` / `.status` (1).
|
||||
- On a fresh dedicated database the real competing run-row lock reproduced the
|
||||
incorrect ineligible outcome. `/tmp/sleep51-lock-before-fresh.log` / `.status`
|
||||
(1), one failed regression and 35 deliberately unselected cases.
|
||||
- After preserving operational errors, all 77 cleanup-operation cases and all
|
||||
36 database discovery/finalization cases passed. Logs: `/tmp/sleep51-outcome-after`
|
||||
and `/tmp/sleep51-lock-after`, each `.log` and `.status` (0).
|
||||
- Six further regressions reproduced incorrectly discarded waiting states: live
|
||||
run/process ownership, live provider process, coordinator lease, active environment
|
||||
lease and busy legacy maintenance. `/tmp/sleep51-deferred-before.log` / `.status`
|
||||
(1), six failed and 74 passing selected cases.
|
||||
- The deferred outcome repair passes all 80 cleanup-operation cases. A typecheck
|
||||
caught the new malformed-input fixture casting its V1 field to the broader input
|
||||
union; correcting only that fixture's cast restored server typecheck. Preserve
|
||||
`/tmp/sleep51-outcomes-types.log` / `.status` (1) and the corrected
|
||||
`/tmp/sleep51-outcomes-types-final.log` / `.status` (0).
|
||||
- Broader executor verification passes 798 cases, with the 15 embedded-database
|
||||
remote warm-ownership cases deliberately excluded for the existing host resource
|
||||
limit. `/tmp/sleep51-executor-consumers.log` / `.status` (0). The full 36-case
|
||||
database suite also passes on a fresh database after the outcome changes:
|
||||
`/tmp/sleep51-discovery-final.log` / `.status` (0). Total: 834 distinct cases.
|
||||
- The final strengthened physical tests verify a deferred run can settle after
|
||||
its run process, coordinator lease, environment lease or retained maintenance
|
||||
blocker clears, including release of temporary local scope reservations.
|
||||
All 80 selected cases pass again: `/tmp/sleep51-reactivation.log` / `.status` (0).
|
||||
Provider-PID deferral is covered, but its exit-driven reactivation and local
|
||||
session-owner notification still require integration tests with the worker.
|
||||
- Final server typecheck passes after the strengthened reactivation assertions:
|
||||
`/tmp/sleep51-reactivation-types.log` / `.status` (0).
|
||||
|
||||
No source edits occurred during the cited local acceptance processes. No Cloud
|
||||
alarm coverage, sleep eligibility, quarantine policy or provider protocol changed.
|
||||
All other stack duties and older acceptance gaps remain open.
|
||||
|
||||
|
||||
## Shared worker checkpoint
|
||||
|
||||
New owner: `native-runtime/retained-native-session-cleanup-work.ts`.
|
||||
Notification boundary: `retained-native-session-cleanup-notifications.ts`.
|
||||
The public finalization notification helper now wakes this queue in the same
|
||||
transaction as its other finalization duties. Provider mechanics stay in the
|
||||
executor. The scheduler receives only candidate identity, normalized outcome and
|
||||
next deadline; no provider selection was added to shared scheduling.
|
||||
|
||||
- Discovery reconstructs scoped candidates in 100-row keyset pages and includes
|
||||
future coordinator leases. Execution checks the current row before dispatch.
|
||||
- Worker batches contain one physical cleanup. Duplicate attempts join one promise;
|
||||
different runs serialize through the same database lane. This preserves bounded
|
||||
inspection and avoids mistaking another cleanup's brief session reservation for
|
||||
a long-lived owner requiring a future release event.
|
||||
- A coordinator lease uses its saved expiry. `process_exit` retains one 30-second
|
||||
probe for that concrete candidate. This exception is for a retained process
|
||||
lacking a local exit callback, not idle-history polling; every attempt rechecks
|
||||
all authority and physical evidence. It stops when the candidate settles,
|
||||
becomes ineligible/operator-required, is deleted, or changes to an event wait.
|
||||
- Other deferred outcomes drop their scheduled entry until an owner release or
|
||||
committed mutation reactivates discovery. **Those production release producers
|
||||
must be installed and tested before periodic discovery is disabled.**
|
||||
- Local release subscriptions are scoped to the database owner and are explicitly
|
||||
removed on shutdown. Coordinator shutdown waits for its joined operation.
|
||||
- No startup registration or periodic skip flag has been added yet. Cloud alarm
|
||||
coverage and conservative sleep blockers remain unchanged.
|
||||
|
||||
Evidence on this source checkpoint:
|
||||
|
||||
- Initial worker tests: 12 passed, `/tmp/sleep51-worker-first.log` / `.status` (0).
|
||||
- Switching the first post-idle writer to the existing finalization notification
|
||||
helper reproduced a missed cleanup wake: `/tmp/sleep51-finalization-signal-before.log`
|
||||
/ `.status` (1), one failed regression with 16 cases deliberately unselected.
|
||||
Adding cleanup to that existing fanout repaired it. No new transaction wrapper
|
||||
was introduced.
|
||||
- The expanded selection passes 18 cases, then 19 after adding a 103-item backlog.
|
||||
`/tmp/sleep51-worker-second.log` and `/tmp/sleep51-worker-final.log`, status 0.
|
||||
The final suite covers empty-to-active-to-quiet, real physical refusal with
|
||||
retained history, named release signals, restart reconstruction, exact lease
|
||||
expiry, early release, cancellation, concrete process probing, outer rollback,
|
||||
first write during drain, failed work across drain, paused startup, discovery
|
||||
failure, a committed wake during cleanup, duplicate joining, serialization,
|
||||
shutdown, and completed history across discovery pages.
|
||||
- The release-signal tests exercise the notification boundary, not all real session,
|
||||
environment and maintenance producers. The main first-writer test exercises
|
||||
the existing finalization notification helper inside a real transaction, but
|
||||
the complete domain-writer audit is still pending.
|
||||
- Existing manual/startup cleanup and finalization: all 36 cases pass on a fresh
|
||||
database, `/tmp/sleep51-worker-legacy.log` / `.status` (0).
|
||||
- Existing reassessment worker: all 37 lifecycle/producer cases pass on a separate
|
||||
fresh database after extending finalization fanout,
|
||||
`/tmp/sleep51-worker-reassessment.log` / `.status` (0).
|
||||
- Server typecheck passes: `/tmp/sleep51-worker-types-final.log` / `.status` (0).
|
||||
Total current selection: 92 passing cases. Earlier physical-operation and broader
|
||||
executor results remain recorded above; they are not presented as rerun here.
|
||||
|
||||
The 103-item test proves paged reconstruction, actual bounded draining and SQL
|
||||
quiet. It is not the outstanding large-history/candidate cost qualification.
|
||||
Remaining work includes every actual producer/release path, operational ownership
|
||||
through retained callbacks, startup integration, periodic removal, full PR checks
|
||||
and the rest of the recurring-server stack. No new PR or merge is claimed.
|
||||
|
||||
|
||||
## Durable owner release checkpoint
|
||||
|
||||
The environment service owns lease release and its existing transaction wrapper.
|
||||
It now signals the affected run before single/run-wide release commits. Metadata
|
||||
updates and still-pending cleanup do not activate retained-session cleanup. The
|
||||
run-wide API resolves company scope from its authoritative run rather than an
|
||||
unscoped notification. It has no production call sites in the inspected checkout;
|
||||
its public service behavior remains covered.
|
||||
|
||||
Three direct settlement owners also bypass ordinary lease release: stop-only
|
||||
cleanup, workspace-export-resume settlement, and confirmed sandbox absence. Each
|
||||
now registers the same scoped intent before the status write. The first two reuse
|
||||
their transaction; absence settlement uses a short transaction that composes with
|
||||
a caller-owned transaction. No provider I/O enters those transactions. Existing
|
||||
receipt validation, ownership and compare-and-set predicates are unchanged.
|
||||
Verified settled-process release signals before clearing its mutable identity.
|
||||
|
||||
The environment writer audit found no production raw-SQL lease updates/deletes
|
||||
or direct lease deletes in server/src. Other direct writers either retain/add
|
||||
active or pending-cleanup ownership, change metadata/lease policy, or retire an
|
||||
already nonblocking lease. In particular, environment-task-runtime completion
|
||||
only changes retained -> released; retained is already excluded by this cleanup's
|
||||
active/pending-cleanup predicate. Reusable replacement also starts from released
|
||||
or retained. These transitions do not unblock this worker. This inventory does
|
||||
not finish the separate run/finalization writer or local-owner audit.
|
||||
|
||||
Evidence:
|
||||
|
||||
- `/tmp/sleep51-env-before.log` / `.status` (1): both real single and run-wide
|
||||
release fail to reactivate cleanup. `/tmp/sleep51-env-after.log` / `.status`
|
||||
(0): all 21 lifecycle cases pass after repair.
|
||||
- `/tmp/sleep51-process-before.log` / `.status` (1): verified process release
|
||||
fails to reactivate before the existing process probe. The process stop check
|
||||
is controlled in this integration test; database, receipt writer and service
|
||||
transactions are real.
|
||||
- `/tmp/sleep51-release-after.log` / `.status` (1): the initial process rollback
|
||||
test failed to await the coordinator's rollback reconciliation before opening
|
||||
the next transaction, then contaminated the following deadline assertion.
|
||||
The corrected barrier waits for that reconciliation and verifies the persisted
|
||||
original PID. `/tmp/sleep51-release-final.log` / `.status` (0): 23 cases pass.
|
||||
- `/tmp/sleep51-settlement-before.log` / `.status` (1): all three direct settlement
|
||||
producers reproduce the missing wake after commit. The absence test supplies
|
||||
already verified typed proof at the settlement boundary; it does not claim
|
||||
Cloud receipt signature/provider qualification.
|
||||
- `/tmp/sleep51-settlement-after.log` / `.status` (0): all 26 lifecycle cases pass.
|
||||
These prove outer rollback, no delivery before outer commit, release activation,
|
||||
successful settlement and subsequent SQL quiet. Failed cleanup, metadata-only
|
||||
updates and stale attempt IDs do not prematurely release the wait.
|
||||
- `/tmp/sleep51-env-consumer.log` / `.status` (0): all 51 existing environment
|
||||
service cases pass on a fresh isolated database.
|
||||
- `/tmp/sleep51-settlement-consumers.log` / `.status` (0): all 131 existing legacy
|
||||
restore/native failed-workspace recovery cases pass.
|
||||
- `/tmp/sleep51-settlement-types.log` / `.status` (0): server typecheck passes.
|
||||
|
||||
This checkpoint covers 208 selected cases. Actual process exit and physical
|
||||
cleanup safety tests from previous checkpoints remain distinct evidence. Session
|
||||
owner release, maintenance completion, full run/finalization producer coverage,
|
||||
physical callback ownership, startup wiring, large-history cost, periodic removal,
|
||||
publication and full PR acceptance are still open. No polling removal is claimed.
|
||||
|
||||
|
||||
## Local session-owner release checkpoint
|
||||
|
||||
The executor now signals company/agent scope after its execution scope and
|
||||
startup ownership settle, and after backend initialization relinquishes its claim.
|
||||
An older retained run can share the physical session with a newer run, so a
|
||||
run-only hint would miss supported waiters. Durable candidate discovery and all
|
||||
physical authority checks still determine whether cleanup can proceed.
|
||||
|
||||
Warm-session removal is centralized in `forgetWarmNativeSession`. It checks the
|
||||
expected entry identity, removes that entry, then invokes its captured release
|
||||
callback. All existing retirement paths use this owner boundary, including idle
|
||||
expiry, explicit shutdown, configuration/instruction changes and quarantine.
|
||||
Existing failed-close retention rules remain; a late close cannot delete a newer
|
||||
entry. The callback holds the database scope without adding database fields to
|
||||
the warm-session record or teaching the scheduler provider behavior. Cleanup's
|
||||
own temporary reservation release deliberately does not emit this hint, avoiding
|
||||
self-reactivation on every inspection of retained history.
|
||||
|
||||
Three real executor lifecycle regressions first failed with zero release hints:
|
||||
both startup cancellation variants and warm idle retirement. The strengthened
|
||||
warm test holds its original close promise unresolved and proves no release hint
|
||||
until that promise settles. These tests use the existing controlled executor/DB
|
||||
fixtures. They test actual owner paths; the separate PostgreSQL worker suite
|
||||
proves the notification-to-reconciliation lifecycle. They are not a single live
|
||||
provider-to-database end-to-end qualification.
|
||||
|
||||
- `/tmp/sleep51-local-release-before.log` / `.status` (1): 3 expected failures,
|
||||
810 deliberately unselected cases.
|
||||
- `/tmp/sleep51-local-release-after.log` / `.status` (0): 798 executor cases pass,
|
||||
15 embedded-PostgreSQL remote warm ownership cases excluded because of the
|
||||
previously diagnosed host semaphore limitation.
|
||||
- `/tmp/sleep51-local-release-types.log` / `.status` (0): server types pass.
|
||||
- `/tmp/sleep51-local-release-worker.log` / `.status` (0): 26 real-database worker
|
||||
lifecycle cases pass after local producer wiring.
|
||||
- `/tmp/sleep51-local-release-final.log` / `.status` (0): final 798-case executor
|
||||
rerun passes after removing duplicate entry guards from callers. The same 15
|
||||
embedded-database cases remain excluded, not reported as passing.
|
||||
|
||||
Next: retained maintenance still has a separate lifetime. Runner tracks active
|
||||
roots plus original bounded callbacks, but exposes only idle inspection and an
|
||||
all-roots drain. A timed-out callback can still own its root. The scheduler needs
|
||||
an owned completion signal for that concrete wait; heartbeat's existing outer
|
||||
sweep drain must move inside the common physical-operation lifetime so scheduled,
|
||||
manual and shutdown callers join the original work. Do not clear that obligation
|
||||
on timeout or add a recurring maintenance-state poll. Runner must expose its own
|
||||
completion contract; it must not import server notification policy.
|
||||
|
||||
Remaining finalization writer audit, maintenance ownership, initialization-path
|
||||
reactivation proof, startup integration, large-history cost, periodic removal,
|
||||
publication, and complete PR gates still stand. This slice remains unpublished;
|
||||
periodic cleanup remains enabled and the full stack remains unmerged.
|
||||
|
||||
|
||||
## Original maintenance ownership and terminal projection checkpoint
|
||||
|
||||
Runner's existing drain now joins active maintenance attempts as well as their
|
||||
original callbacks. A root-specific promise resolves only when neither remains;
|
||||
the global drain repeats discovery if another operation was registered meanwhile.
|
||||
This is local ownership settlement, not physical retirement or launch permission.
|
||||
The root waiter stays inside the Runner transport module; the server consumes the
|
||||
existing public drain contract. No server policy is imported into Runner.
|
||||
|
||||
The physical cleanup entry point now joins that drain in its own finally block,
|
||||
including on error. Manual, startup and future scheduled callers therefore own
|
||||
the complete operation. A maintenance-deferred result emits its scoped release
|
||||
hint only after the original callbacks settle. Heartbeat's sweep no longer owns
|
||||
a separate trailing drain. Existing timeout/quarantine and physical proof rules
|
||||
remain unchanged; an indefinitely unresolved original callback retains ownership.
|
||||
|
||||
Two more producer boundaries were missing cleanup intent: finalizer run
|
||||
projection (both committed-result repair and ordinary post-decision projection),
|
||||
and heartbeat's shared terminal run-status writer. They now signal within their
|
||||
existing transactions, including terminal metadata repairs. Running-progress
|
||||
writes do not wake this duty. The ordinary finalizer signal is unconditional in
|
||||
its projection transaction because metadata may make a previously terminal run
|
||||
eligible even when projectRunStatus is false.
|
||||
|
||||
Real PostgreSQL regressions cover outer commit, rollback, and a deterministic
|
||||
separate-commit race: delay the finalizer after its actual decision commit, allow
|
||||
the worker to consume that notification while the run is still running, then
|
||||
permit terminal projection. Before the new signal cleanup is stranded; afterward
|
||||
it runs once. No finalizer/provider decision logic was mocked for this race.
|
||||
|
||||
Evidence (all /tmp paths have matching .status files):
|
||||
|
||||
- sleep51-maintenance-before: invalid root Vitest selection, no matching Runner
|
||||
tests; not regression evidence. Correct package invocation
|
||||
sleep51-maintenance-before-runner failed the missing completion API test.
|
||||
- sleep51-maintenance-after: all 19 Runner settlement cases pass, including
|
||||
timeout/abort ownership, unrelated-root idleness and unresolved original callbacks.
|
||||
- sleep51-owned-maintenance-before: both new physical lifetime tests failed;
|
||||
sleep51-owned-maintenance-after passes 799 executor cases. The 15 embedded-DB
|
||||
remote warm cases remain excluded for the diagnosed host semaphore limit.
|
||||
- sleep51-maintenance-types initially failed the test mock's Promise<undefined>
|
||||
inference; an explicit Promise<void> fixes it. maintenance-types-final passes.
|
||||
- sleep51-maintenance-worker: all 26 existing real-DB worker cases pass.
|
||||
- sleep51-projection-before: committed-finalizer producer regression failed.
|
||||
- sleep51-lifecycle-before: both terminal writer variants failed; running progress
|
||||
remained quiet. The repaired sleep51-projection-after passes all 64 worker and
|
||||
heartbeat lifecycle cases, including outer rollback and terminal metadata repair.
|
||||
- sleep51-corpus-projection-before: the separate-commit race failed with zero
|
||||
cleanup calls after the terminal projection. sleep51-corpus-projection-after
|
||||
passes the entire 146-case finalizer corpus, including that regression.
|
||||
- sleep51-projection-types: server typecheck passes.
|
||||
|
||||
Fresh origin/master remains 1fbbef45a8d399771add7a27a04b12f3a20e264c.
|
||||
Remaining: final producer/initialization audit, startup integration, history/query
|
||||
cost, periodic removal, broader acceptance and PR gates. Nothing is published or
|
||||
merged for slice 51; periodic cleanup is still enabled at this checkpoint.
|
||||
|
||||
|
||||
## Startup integration and full executor verification
|
||||
|
||||
The enabled server path now registers the scoped worker with the shared delivery
|
||||
coordinator. Its admission predicate includes warm standby, task drain and server
|
||||
shutdown. Server shutdown removes local release subscriptions before stopping the
|
||||
coordinator. Heartbeat also tracks each original physical operation, so its direct
|
||||
execution drain joins scheduled cleanup independently of coordinator shutdown.
|
||||
The periodic orphan pass skips only retained-session cleanup. Startup recovery
|
||||
and explicit/manual recovery still call the bounded legacy sweep; all paths share
|
||||
the per-database lane and join identical run operations.
|
||||
|
||||
The startup and assembled-heartbeat regressions first failed because registration
|
||||
was absent. After wiring, all 100 selected startup/worker tests and server types
|
||||
pass (/tmp/sleep51-startup-{before,after,types}.{log,status}). The assembled test
|
||||
holds the original cleanup promise and proves heartbeat's drain cannot finish
|
||||
before it settles. Existing worker tests cover drain/resume, error/retry, rollback,
|
||||
concurrent hints and saved lease deadlines.
|
||||
|
||||
The user explicitly approved removal of 11 named orphaned PostgreSQL semaphore
|
||||
sets. Each exact ID/key and its recorded dead PIDs was rechecked; the corresponding
|
||||
shared-memory segment was absent. Only those 11 approved sets were removed.
|
||||
The complete executor suite then passed all 814 tests, with no exclusions
|
||||
(/tmp/sleep51-executor-full.log and .status). This supersedes the earlier 15-case
|
||||
host limitation for this suite; it does not erase the earlier failed setup evidence.
|
||||
The four overlapping initialization cases now also prove no cleanup release hint
|
||||
until both original preparation operations settle, on success and failure.
|
||||
|
||||
The query-cost probe cloned 10,000 settled retained sessions, with real result,
|
||||
assessment, decision and finalization foreign keys, in an isolated database.
|
||||
EXPLAIN ANALYZE measured startup at 29.764 ms/62,266 shared-hit blocks, company
|
||||
scope at 27.731 ms/63,190 blocks, agent-release scope at 27.672 ms/63,190 blocks,
|
||||
and run scope at 0.030 ms/6 blocks. All return zero candidates. Broad discovery
|
||||
still grows with retained history; these local warm-cache measurements are not a
|
||||
fleet latency guarantee. It runs on startup or a relevant scope change, not on a
|
||||
recurring idle scan. The separate 103-candidate test covers paging and settlement.
|
||||
No new index is claimed necessary or added from this bounded measurement.
|
||||
|
||||
Probe source: /tmp/sleep51-cost-probe.mts. Plans: /tmp/sleep51-cost-final.log
|
||||
(status 0). Initial root invocation failed because tsx is package-local; the
|
||||
corrected invocation then exposed a copied supersedes-assessment FK from the
|
||||
fixture. The seed transaction rolled back. Clearing that copied predecessor link
|
||||
produced the successful realistic-history fixture; no constraint was disabled.
|
||||
|
||||
Broader workspace verification, publication and all remote CI/review gates remain.
|
||||
The four older stack gates were refreshed and still need repair: #17490/#17572
|
||||
E2E shard 3, #17576 security/review findings, and #17580 review findings.
|
||||
Nothing in this stack is merged or authorized for merge.
|
||||
|
||||
|
||||
## Published PR and CI repair checkpoint
|
||||
|
||||
Draft [#17717](https://github.com/paperclipai/paperclip/pull/17717) was published
|
||||
at `8c62a6b0c0c381326ab94fd5c75c78e0dd426e1a`, based on #17709. Workspace
|
||||
typecheck and build passed there. Greptile reviewed that exact head at 5/5 with
|
||||
no actionable findings. CI run `40323191795` found three failures:
|
||||
|
||||
- The heartbeat recovery module test still mocked cleanup through the old barrel
|
||||
import and expected heartbeat to drain Runner maintenance itself. It now mocks
|
||||
the extracted cleanup owner and holds its operation promise. The complete
|
||||
34-case module suite passes after correcting the mock target.
|
||||
- The assembled heartbeat process-recovery test also expected the old outer
|
||||
drain call. Its controlled physical cleanup now retains its original callback
|
||||
inside the cleanup operation, matching the executor contract. It still proves
|
||||
startup/reaping stay responsive, duplicate cleanup joins, shutdown stays
|
||||
non-quiescent, no provider turn starts, and persisted results remain unchanged.
|
||||
The complete 408-case process-recovery file passes. Actual Runner callback
|
||||
settlement remains covered in the executor and Runner suites, rather than
|
||||
asserted through this controlled boundary alone.
|
||||
- The Copilot source attestation still bound the old transport bytes. Preserve
|
||||
the v58 fixture and add v59 using `copilot-profile-identity.mjs`; regenerate
|
||||
TypeScript/Rust admission from the shared manifest. Historical v58 remains
|
||||
decodable, but exact launch admission uses the new identity. The identity also
|
||||
catches inherited source drift in native runtime instructions, native-session
|
||||
runtime and transport contracts. No source changes to those three files are
|
||||
included here. All 69 installation/compatibility cases pass, including rejection
|
||||
of the old launch identity and preservation of its history. This is source
|
||||
identity verification, not a claim of new live Copilot qualification.
|
||||
|
||||
Local repair evidence preserves unsuccessful attempts: the first module repair
|
||||
still targeted the old import; the initial identity update retained an old digest
|
||||
literal; an overbroad replacement changed an unrelated historical expectation and
|
||||
was restored before acceptance. The first isolated Runner settlement attempt had
|
||||
no built `paperclip-runnerd` and failed all 19 cases at setup; the new checkout
|
||||
must build that binary before those tests count as executed.
|
||||
|
||||
The full default local suite at the published head remains separate. It reported
|
||||
an OAuth scope/catalog-refresh test timing out at 15 seconds. A separate complete
|
||||
472-case tool-access suite passed with a disposable PostgreSQL database under the
|
||||
same host load. This rerun does not erase the original full-suite failure or
|
||||
establish that its whole command passed. The original checkout remains unchanged
|
||||
while that command runs. Latest-head remote gates, complete suite disposition,
|
||||
and repaired-head review remain required before acceptance.
|
||||
|
||||
|
||||
## Master integration
|
||||
|
||||
Master advanced the source-bound Copilot profile to v63 while this branch
|
||||
published a different v62 identity for retained maintenance settlement. The
|
||||
combined runtime preserves master's uncertain-process and controller-retirement
|
||||
behavior plus this worker's event-driven idle waiters. Its generated profile is
|
||||
v64. Master's v62/v63 fixtures stay byte-for-byte unchanged; the stack's former
|
||||
v62 is retained separately as `copilot-profile-v62-sleep-maintenance-historical-identity.json`.
|
||||
Tests verify their checksums and exact changed source fields, historical decoding,
|
||||
and refusal to launch any historical identity through current installation
|
||||
admission. TypeScript and Rust declarations are generated together. This is source
|
||||
identity validation, not a claim of live provider qualification.
|
||||
|
||||
### Master integration follow-up (2026-10-27)
|
||||
|
||||
Pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` advances Copilot to v65.
|
||||
The combined transport retains its exhausted-reconnect cancellation behavior and
|
||||
this duty's per-root maintenance-idle completion notifications. The generated
|
||||
combined identity is v66; only the version, compatibility source, and runtime
|
||||
request bridge differ from master's v65 declaration. Master's v64/v65 fixtures
|
||||
remain unchanged. The stack's different former v64 is preserved as
|
||||
`copilot-profile-v64-sleep-maintenance-historical-identity.json` alongside its
|
||||
former v62 evidence. Installation tests retain both histories and prove historical
|
||||
profiles remain decodable but cannot launch as the current pinned profile.
|
||||
The profile generator's check, TypeScript/Rust agreement, and runner typecheck
|
||||
pass. This remains source identity verification, not live provider qualification.
|
||||
|
||||
## Merge acceptance on current master (2026-10-29)
|
||||
|
||||
This section supersedes earlier provisional enablement and readiness notes. The
|
||||
reviewed parent #17709 is merged. Its exact reviewed tree is unchanged in the
|
||||
landed squash. A normal merge adopts that ancestry without altering this child’s
|
||||
complete tree. This slice is now prepared directly for master.
|
||||
|
||||
Routine company-only finalization hints do not reconstruct retained cleanup.
|
||||
Ordinary fanout requires an issue or run identity; pending-effects serialization
|
||||
retains its existing company/run contract. Cleanup producers allocate their run
|
||||
identity before registering transaction intent and before writing work. The
|
||||
local release path can still wake a dormant candidate. A controlled regression
|
||||
removed only the scope guard and failed; restoring that guard passed the same
|
||||
assertions. The complete 29-case worker file also passes, including durable
|
||||
reconstruction, commit/rollback, drain/resume, changes during execution, physical
|
||||
owner settlement and quiet retained history.
|
||||
|
||||
Current parent integration preserved 12,693 independent parent entries, 14
|
||||
owned-only entries and 15 automatic shared entries. Ten manually reviewed paths
|
||||
retain both runtime histories and both sets of installation assertions. All 92
|
||||
parent historical fixtures remain byte-exact. The sleep branch’s different
|
||||
profile 66 is preserved under a separate historical filename, alongside its
|
||||
prior 62 and 64 variants. Forty added code comments were removed with identical
|
||||
nontrivia TypeScript token streams before generating source identities.
|
||||
|
||||
Local server acceptance covers **2,342 distinct passing cases in 18 complete
|
||||
files**. This includes 977 native executor cases, 146 arbiter corpus cases, 410
|
||||
process-recovery cases, 273 external-chat response cases, 76 startup cases and
|
||||
29 cleanup worker cases. Production server typechecking, build, module boundaries,
|
||||
migration order and profile generator checks pass on the stated source. Existing
|
||||
SQL and the database package tree are unchanged from parent migration 413.
|
||||
Disposable PostgreSQL 17 fixtures were prepared independently. Temporary imports
|
||||
use the existing fresh-per-call recovery fixture helper for embedded-only paths;
|
||||
every import was restored byte-for-byte. No test assertion or timeout was relaxed.
|
||||
|
||||
The first 15-file invocation reported 1,607 passes and one startup shutdown failure,
|
||||
plus its associated unhandled rejection. The shared coordinator test double lacked
|
||||
the real worker’s `stop()` method, which task-drain shutdown now calls. Only that
|
||||
test double changed. All 76 unchanged startup cases then passed in 6.17 seconds.
|
||||
The other 14 complete files account for 1,532 passes. The remaining three complete
|
||||
files passed separately: process recovery 410, environment service 51 and external
|
||||
chat response 273. The failed log remains part of the acceptance record; the first
|
||||
whole invocation is not reported as a successful command.
|
||||
|
||||
Master then advanced independently with Claude ACP reconnect replay and a lockfile
|
||||
refresh. Its server, UI and database trees are unchanged. Its published Copilot
|
||||
profile 79 differs from this branch’s prepared 79. The merge preserves master’s
|
||||
canonical fixture and saves the sleep variant as
|
||||
`copilot-profile-v79-sleep-maintenance-historical-identity.json`. Combined profile
|
||||
80 binds current compatibility decoding and retained-maintenance completion;
|
||||
other providers retain master’s declarations. Historical identities remain
|
||||
readable and fail current launch admission. This is source identity verification,
|
||||
not live Copilot qualification.
|
||||
|
||||
Current integration validation and exact review/CI results are recorded in the
|
||||
PR. Local evidence proves named worker behavior. It does not establish full
|
||||
repository acceptance, Cloud alarm coverage, deployed host/database suspension
|
||||
or measured cost savings. Concrete live-PID absence probes and durable lease
|
||||
expiries remain real obligations; this change does not discard them to claim quiet.
|
||||
|
||||
The current profile 80 runtime selection passes 130 cases across four files in
|
||||
125.90 seconds. One existing opt-in closed Pi snapshot case is skipped because
|
||||
`PAPERCLIP_TEST_PI_DISTRIBUTION_ROOT` is not set. This skip is not counted as
|
||||
acceptance of a live Pi installation. The Copilot installation/history and
|
||||
transport-settlement files have no skipped cases.
|
||||
|
||||
Current integration at `d08f764e3efe75e96929173afe7ee8739121ad9f` passes
|
||||
all 12 validation commands: frozen install, three source/profile checks, Runner
|
||||
and server typechecking, server build, the four-file runtime selection, installed
|
||||
ACPX/Grok contract tests (22 passes), native-provider Rust tests (26 passes),
|
||||
module boundaries and migration order. The live-provider limits above remain.
|
||||
@@ -32,12 +32,12 @@
|
||||
"requiresProviderPolicy": true
|
||||
},
|
||||
"copilot": {
|
||||
"agentProfileVersion": 79,
|
||||
"agentProfileVersion": 80,
|
||||
"agentServerPackage": "@github/copilot",
|
||||
"agentServerVersion": "1.0.88",
|
||||
"agentRuntimePackage": null,
|
||||
"agentRuntimeVersion": null,
|
||||
"commandDigest": "sha256:a0e9f339935f6ee616fee644c45cb09353e9aed1d1f22215739f9504153ceca1",
|
||||
"commandDigest": "sha256:7b8a6a31925256d225ea3d72fdcfd81b70a45c4b67d3eb0ee5073efff9ac8f8b",
|
||||
"requiresProviderPolicy": true
|
||||
},
|
||||
"claude": {
|
||||
|
||||
@@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option<AcpxReleaseProfile> {
|
||||
agent_runtime_package: None,
|
||||
agent_runtime_version: None,
|
||||
command_digest:
|
||||
"sha256:a0e9f339935f6ee616fee644c45cb09353e9aed1d1f22215739f9504153ceca1",
|
||||
"sha256:7b8a6a31925256d225ea3d72fdcfd81b70a45c4b67d3eb0ee5073efff9ac8f8b",
|
||||
requires_provider_policy: true,
|
||||
},
|
||||
"claude" => AcpxReleaseProfile {
|
||||
|
||||
@@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url";
|
||||
import { resolve, dirname } from "node:path";
|
||||
|
||||
const root = resolve(dirname(fileURLToPath(import.meta.url)), "..");
|
||||
const fixture = resolve(root, "test/fixtures/copilot-profile-v79-identity.json");
|
||||
const fixture = resolve(root, "test/fixtures/copilot-profile-v80-identity.json");
|
||||
const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8"));
|
||||
const declaration = { ...historical.declaration, agentProfileVersion: 79 };
|
||||
const declaration = { ...historical.declaration, agentProfileVersion: 80 };
|
||||
delete declaration.permissionIdentitySourceSha256;
|
||||
const sources = {
|
||||
displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts",
|
||||
|
||||
@@ -87,9 +87,71 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi
|
||||
|
||||
describe("Copilot build-owned installation", () => {
|
||||
it("admits the current profile with environment launch preparation", () => {
|
||||
expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(79);
|
||||
expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(80);
|
||||
expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined();
|
||||
expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:a0e9f339935f6ee616fee644c45cb09353e9aed1d1f22215739f9504153ceca1");
|
||||
expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:7b8a6a31925256d225ea3d72fdcfd81b70a45c4b67d3eb0ee5073efff9ac8f8b");
|
||||
});
|
||||
|
||||
it("preserves published v65 while binding retained-maintenance waits in v66", async () => {
|
||||
const priorBytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v65-identity.json", import.meta.url));
|
||||
expect(createHash("sha256").update(priorBytes).digest("hex")).toBe("ca1d95c21b3f49c99b6d7c6bd3c2bf149715db3263927ad653cc0c6f33fb4462");
|
||||
const prior = JSON.parse(priorBytes.toString("utf8"));
|
||||
const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v66-sleep-maintenance-historical-identity.json", import.meta.url), "utf8"));
|
||||
expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([
|
||||
"agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256",
|
||||
]);
|
||||
expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort());
|
||||
expect(isSupportedAcpxProfileVersion("copilot", 65)).toBe(true);
|
||||
await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot,
|
||||
agentProfileVersion: 65, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned");
|
||||
});
|
||||
|
||||
it("preserves the historical sleep-maintenance v64 identity", async () => {
|
||||
const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v64-sleep-maintenance-historical-identity.json", import.meta.url), "utf8"));
|
||||
for (const [name, checksum, changed] of [
|
||||
["copilot-profile-v63-identity.json", "4adb05bd642f0222a3bcc85cfefbeb2988de439f58125012e1c575caedd90798", ["agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256"]],
|
||||
["copilot-profile-v62-sleep-maintenance-historical-identity.json", "e0c93dd7210fbb0115e09b70887ec4ae40621fc5314bd1625e586caf2163c94e", ["agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256", "semanticCompletionContractSourceSha256", "semanticContinuationSourceSha256"]],
|
||||
] as const) {
|
||||
const bytes = readFileSync(new URL(`../../../test/fixtures/${name}`, import.meta.url));
|
||||
expect(createHash("sha256").update(bytes).digest("hex")).toBe(checksum);
|
||||
const prior = JSON.parse(bytes.toString("utf8"));
|
||||
expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual(changed);
|
||||
expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort());
|
||||
const sorted = Object.fromEntries(Object.entries(prior.declaration).sort(([a], [b]) => a.localeCompare(b)));
|
||||
expect(`sha256:${createHash("sha256").update(JSON.stringify(sorted)).digest("hex")}`).toBe(prior.commandDigest);
|
||||
expect(isSupportedAcpxProfileVersion("copilot", prior.declaration.agentProfileVersion)).toBe(true);
|
||||
await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot,
|
||||
agentProfileVersion: prior.declaration.agentProfileVersion, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned");
|
||||
}
|
||||
});
|
||||
|
||||
it("preserves profile 78 while binding retained-session waits in profile 79", async () => {
|
||||
const bytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v78-identity.json", import.meta.url));
|
||||
expect(createHash("sha256").update(bytes).digest("hex")).toBe("da2d7d33dbcdb900f3afb0ad0d0d1502c31f7638772a58009a3300f7e442fd1c");
|
||||
const prior = JSON.parse(bytes.toString("utf8"));
|
||||
const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v79-sleep-maintenance-historical-identity.json", import.meta.url), "utf8"));
|
||||
expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([
|
||||
"agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256",
|
||||
]);
|
||||
expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort());
|
||||
expect(isSupportedAcpxProfileVersion("copilot", 78)).toBe(true);
|
||||
await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot,
|
||||
agentProfileVersion: 78, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned");
|
||||
});
|
||||
|
||||
|
||||
it("preserves published profile 79 while binding retained cleanup in profile 80", async () => {
|
||||
const bytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v79-identity.json", import.meta.url));
|
||||
expect(createHash("sha256").update(bytes).digest("hex")).toBe("92fb6b5f1a312933f78d650fdc037ded4876d112c472fcfd5c1116cdbb0dae44");
|
||||
const prior = JSON.parse(bytes.toString("utf8"));
|
||||
const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v80-identity.json", import.meta.url), "utf8"));
|
||||
expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([
|
||||
"agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256",
|
||||
]);
|
||||
expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort());
|
||||
expect(isSupportedAcpxProfileVersion("copilot", 79)).toBe(true);
|
||||
await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot,
|
||||
agentProfileVersion: 79, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned");
|
||||
});
|
||||
|
||||
it("retains the published profile 75 fixture while advancing the event contract", async () => {
|
||||
|
||||
@@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = {
|
||||
"protocolVersion": 1,
|
||||
"acpxVersion": "0.13.1",
|
||||
"agent": "copilot",
|
||||
"agentProfileVersion": 79,
|
||||
"agentProfileVersion": 80,
|
||||
"agentServerPackage": "@github/copilot",
|
||||
"agentServerVersion": "1.0.88",
|
||||
"agentRuntimePackage": null,
|
||||
"agentRuntimeVersion": null,
|
||||
"commandDigest": "sha256:a0e9f339935f6ee616fee644c45cb09353e9aed1d1f22215739f9504153ceca1",
|
||||
"commandDigest": "sha256:7b8a6a31925256d225ea3d72fdcfd81b70a45c4b67d3eb0ee5073efff9ac8f8b",
|
||||
"permissionPolicy": "interactive"
|
||||
},
|
||||
"claude": {
|
||||
|
||||
@@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js";
|
||||
|
||||
/** Decodable historical profile revisions, not permission to launch them.
|
||||
* Exact current profile/package/digest admission is checked separately. */
|
||||
type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78;
|
||||
type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79;
|
||||
type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"];
|
||||
export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion;
|
||||
|
||||
const historicalVersions: Readonly<Record<string, readonly HistoricalAcpxProfileVersion[]>> = {
|
||||
pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5],
|
||||
grok: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78],
|
||||
grok: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79],
|
||||
cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15],
|
||||
};
|
||||
|
||||
|
||||
@@ -46,6 +46,7 @@ import {
|
||||
settleRetainedRunnerdSession,
|
||||
retainedRunnerdCleanupProofIsCurrent,
|
||||
retainedRunnerdMaintenanceIsIdle,
|
||||
waitForRetainedRunnerdMaintenanceIdle,
|
||||
} from "./runnerd-codex-transport.js";
|
||||
|
||||
// Explicit private-artifact test lane; production/default dist is never changed.
|
||||
@@ -605,6 +606,11 @@ it.each([
|
||||
authorize: () => stuckAuthorization,
|
||||
});
|
||||
const observed = blocked.catch((error: unknown) => error);
|
||||
let idle = false;
|
||||
const idleWait = waitForRetainedRunnerdMaintenanceIdle(copy).then(() => { idle = true; });
|
||||
|
||||
await waitForRetainedRunnerdMaintenanceIdle(join(copy, "unrelated"));
|
||||
expect(idle).toBe(false);
|
||||
if (interruption === "abort") abort.abort();
|
||||
else await vi.advanceTimersByTimeAsync(30_000);
|
||||
expect(await observed).toMatchObject({
|
||||
@@ -623,9 +629,11 @@ it.each([
|
||||
// The bounded wrapper has already failed, but its original callback
|
||||
// remains owned until it actually settles. No retry/proof is granted.
|
||||
expect(drained).toBe(false);
|
||||
expect(idle).toBe(false);
|
||||
releaseAuthorization();
|
||||
await drain;
|
||||
await Promise.all([drain, idleWait]);
|
||||
expect(drained).toBe(true);
|
||||
expect(idle).toBe(true);
|
||||
expect(retainedRunnerdMaintenanceIsIdle(copy)).toBe(true);
|
||||
} finally {
|
||||
releaseAuthorization();
|
||||
|
||||
@@ -2095,6 +2095,7 @@ export type RetainedRunnerdMaintenanceEpochReceipt = {
|
||||
const retainedRunnerdCleanupProofs = new WeakMap<object, readonly number[]>();
|
||||
const retainedMaintenanceOperations = new Map<string, Set<Promise<unknown>>>();
|
||||
const activeMaintenanceRoots = new Set<string>();
|
||||
const maintenanceIdleWaiters = new Map<string, { promise: Promise<void>; resolve: () => void }>();
|
||||
|
||||
/** No absence claim about an old child; only this controller's joined work. */
|
||||
export function retainedRunnerdMaintenanceIsIdle(directory: string): boolean {
|
||||
@@ -2102,16 +2103,33 @@ export function retainedRunnerdMaintenanceIsIdle(directory: string): boolean {
|
||||
return !activeMaintenanceRoots.has(root) && !retainedMaintenanceOperations.has(root);
|
||||
}
|
||||
|
||||
|
||||
|
||||
export function waitForRetainedRunnerdMaintenanceIdle(directory: string): Promise<void> {
|
||||
const root = resolve(directory);
|
||||
if (retainedRunnerdMaintenanceIsIdle(root)) return Promise.resolve();
|
||||
const existing = maintenanceIdleWaiters.get(root);
|
||||
if (existing) return existing.promise;
|
||||
let release!: () => void;
|
||||
const promise = new Promise<void>(done => { release = done; });
|
||||
maintenanceIdleWaiters.set(root, { promise, resolve: release });
|
||||
return promise;
|
||||
}
|
||||
|
||||
function releaseMaintenanceIdleWaiters(root: string) {
|
||||
if (!retainedRunnerdMaintenanceIsIdle(root)) return;
|
||||
const waiting = maintenanceIdleWaiters.get(root);
|
||||
maintenanceIdleWaiters.delete(root);
|
||||
waiting?.resolve();
|
||||
}
|
||||
|
||||
/** A timeout revokes cleanup authority, not ownership of an already-started
|
||||
* callback. Shutdown must join the original operations, including any that
|
||||
* another retained callback registers while the current snapshot settles. */
|
||||
export async function drainRetainedRunnerdMaintenanceOperations(): Promise<void> {
|
||||
while (retainedMaintenanceOperations.size > 0) {
|
||||
await Promise.allSettled(
|
||||
[...retainedMaintenanceOperations.values()].flatMap((operations) => [
|
||||
...operations,
|
||||
]),
|
||||
);
|
||||
while (activeMaintenanceRoots.size > 0 || retainedMaintenanceOperations.size > 0) {
|
||||
const roots = new Set([...activeMaintenanceRoots, ...retainedMaintenanceOperations.keys()]);
|
||||
await Promise.all([...roots].map(waitForRetainedRunnerdMaintenanceIdle));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2358,6 +2376,7 @@ export async function settleRetainedRunnerdSession(
|
||||
return await settleRetainedRunnerdSessionOwned(input);
|
||||
} finally {
|
||||
activeMaintenanceRoots.delete(root);
|
||||
releaseMaintenanceIdleWaiters(root);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2458,6 +2477,7 @@ async function settleRetainedRunnerdSessionOwned(
|
||||
retainedMaintenanceOperations.get(root) === retained
|
||||
)
|
||||
retainedMaintenanceOperations.delete(root);
|
||||
releaseMaintenanceIdleWaiters(root);
|
||||
};
|
||||
void operation.then(released, released);
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
|
||||
+85
@@ -0,0 +1,85 @@
|
||||
{
|
||||
"commandDigest": "sha256:4c5b7cd84ddbb5375ae044012a87391156633e309f60aba3a99acc80edaee2e0",
|
||||
"declaration": {
|
||||
"schema": "paperclip.acpx_profile_declaration.v1",
|
||||
"agent": "copilot",
|
||||
"agentProfileVersion": 62,
|
||||
"acpxVersion": "0.13.1",
|
||||
"agentServerVersion": "1.0.88",
|
||||
"protocolPolicyRevision": "copilot-agent-manual-v1",
|
||||
"acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1",
|
||||
"policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3",
|
||||
"distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107",
|
||||
"agentFilesBinding": "registered-runtime-context-v1",
|
||||
"systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1",
|
||||
"sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1",
|
||||
"innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a",
|
||||
"upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0",
|
||||
"patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679",
|
||||
"messageIdentityContract": "copilot-native-message-id-v1",
|
||||
"ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1",
|
||||
"permissionContextContract": "copilot-edit-permission-context-v1",
|
||||
"permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b",
|
||||
"permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668",
|
||||
"permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9",
|
||||
"permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f",
|
||||
"permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4",
|
||||
"semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2",
|
||||
"semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791",
|
||||
"semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5",
|
||||
"toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706",
|
||||
"semanticNormalizedInputContract": "validated-forwarded-input-commit-v1",
|
||||
"semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629",
|
||||
"semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103",
|
||||
"semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975",
|
||||
"semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e",
|
||||
"semanticCompletionContractSourceSha256": "3343df6362252d44d12b079ff78c8f6539d1544f3ce5ec36ff36db70ab0fac31",
|
||||
"semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9",
|
||||
"semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603",
|
||||
"semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236",
|
||||
"semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1",
|
||||
"displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877",
|
||||
"displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e",
|
||||
"configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff",
|
||||
"configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994",
|
||||
"profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282",
|
||||
"profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571",
|
||||
"profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88",
|
||||
"attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2",
|
||||
"singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f",
|
||||
"nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce",
|
||||
"nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842",
|
||||
"nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72",
|
||||
"nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90",
|
||||
"harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412",
|
||||
"harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249",
|
||||
"harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94",
|
||||
"transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523",
|
||||
"nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37",
|
||||
"nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44",
|
||||
"nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a",
|
||||
"providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c",
|
||||
"launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296",
|
||||
"runtimeRequestBridgeSourceSha256": "cd23577bfee9ab95bb8a177d1948917e96a7bf6245f320263fb6526ebf5a76cf",
|
||||
"runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295",
|
||||
"runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211",
|
||||
"guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f",
|
||||
"nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918",
|
||||
"profileCompatibilitySourceSha256": "be60656e66d7f61a290f8d1c9a3c9ec511fb13b6541825a087b470e971ccc020",
|
||||
"installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d",
|
||||
"assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c",
|
||||
"assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f",
|
||||
"credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161",
|
||||
"sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80",
|
||||
"profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456",
|
||||
"metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727",
|
||||
"permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1",
|
||||
"semanticContinuationSourceSha256": "d248e47a0fe7f2536d1e64cea0b0b410f34104800cd643d995b4473104ecc8e0",
|
||||
"runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec",
|
||||
"runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8",
|
||||
"runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea",
|
||||
"cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca",
|
||||
"agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba",
|
||||
"nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b"
|
||||
}
|
||||
}
|
||||
+85
@@ -0,0 +1,85 @@
|
||||
{
|
||||
"commandDigest": "sha256:83f8a5bfc7a619c72a2b7bee9e2ce15a0f779f4b483253186b590386186ac76d",
|
||||
"declaration": {
|
||||
"schema": "paperclip.acpx_profile_declaration.v1",
|
||||
"agent": "copilot",
|
||||
"agentProfileVersion": 64,
|
||||
"acpxVersion": "0.13.1",
|
||||
"agentServerVersion": "1.0.88",
|
||||
"protocolPolicyRevision": "copilot-agent-manual-v1",
|
||||
"acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1",
|
||||
"policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3",
|
||||
"distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107",
|
||||
"agentFilesBinding": "registered-runtime-context-v1",
|
||||
"systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1",
|
||||
"sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1",
|
||||
"innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a",
|
||||
"upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0",
|
||||
"patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679",
|
||||
"messageIdentityContract": "copilot-native-message-id-v1",
|
||||
"ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1",
|
||||
"permissionContextContract": "copilot-edit-permission-context-v1",
|
||||
"permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b",
|
||||
"permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668",
|
||||
"permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9",
|
||||
"permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f",
|
||||
"permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4",
|
||||
"semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2",
|
||||
"semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791",
|
||||
"semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5",
|
||||
"toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706",
|
||||
"semanticNormalizedInputContract": "validated-forwarded-input-commit-v1",
|
||||
"semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629",
|
||||
"semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103",
|
||||
"semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975",
|
||||
"semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e",
|
||||
"semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59",
|
||||
"semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9",
|
||||
"semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603",
|
||||
"semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236",
|
||||
"semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1",
|
||||
"displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877",
|
||||
"displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e",
|
||||
"configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff",
|
||||
"configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994",
|
||||
"profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282",
|
||||
"profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571",
|
||||
"profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88",
|
||||
"attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2",
|
||||
"singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f",
|
||||
"nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce",
|
||||
"nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842",
|
||||
"nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72",
|
||||
"nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90",
|
||||
"harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412",
|
||||
"harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249",
|
||||
"harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94",
|
||||
"transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523",
|
||||
"nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37",
|
||||
"nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44",
|
||||
"nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a",
|
||||
"providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c",
|
||||
"launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296",
|
||||
"runtimeRequestBridgeSourceSha256": "ccb98a1696366c67518a6be33d69c9e5e00324a74dee294193a6ba5d536ed53f",
|
||||
"runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295",
|
||||
"runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211",
|
||||
"guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f",
|
||||
"nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918",
|
||||
"profileCompatibilitySourceSha256": "2f875a2f39f4e99a3d03ffce9f325ce2461c51c6b8cb70218ce89ec008eaa5b8",
|
||||
"installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d",
|
||||
"assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c",
|
||||
"assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f",
|
||||
"credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161",
|
||||
"sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80",
|
||||
"profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456",
|
||||
"metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727",
|
||||
"permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1",
|
||||
"semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811",
|
||||
"runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec",
|
||||
"runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8",
|
||||
"runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea",
|
||||
"cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca",
|
||||
"agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba",
|
||||
"nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b"
|
||||
}
|
||||
}
|
||||
+85
@@ -0,0 +1,85 @@
|
||||
{
|
||||
"commandDigest": "sha256:ee0e229a3db5849b5f5528489a44e4450939d9854d6ef98d29a48611acf250f5",
|
||||
"declaration": {
|
||||
"schema": "paperclip.acpx_profile_declaration.v1",
|
||||
"agent": "copilot",
|
||||
"agentProfileVersion": 66,
|
||||
"acpxVersion": "0.13.1",
|
||||
"agentServerVersion": "1.0.88",
|
||||
"protocolPolicyRevision": "copilot-agent-manual-v1",
|
||||
"acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1",
|
||||
"policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3",
|
||||
"distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107",
|
||||
"agentFilesBinding": "registered-runtime-context-v1",
|
||||
"systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1",
|
||||
"sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1",
|
||||
"innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a",
|
||||
"upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0",
|
||||
"patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679",
|
||||
"messageIdentityContract": "copilot-native-message-id-v1",
|
||||
"ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1",
|
||||
"permissionContextContract": "copilot-edit-permission-context-v1",
|
||||
"permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b",
|
||||
"permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668",
|
||||
"permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9",
|
||||
"permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f",
|
||||
"permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4",
|
||||
"semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2",
|
||||
"semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791",
|
||||
"semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5",
|
||||
"toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706",
|
||||
"semanticNormalizedInputContract": "validated-forwarded-input-commit-v1",
|
||||
"semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629",
|
||||
"semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103",
|
||||
"semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975",
|
||||
"semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e",
|
||||
"semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59",
|
||||
"semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9",
|
||||
"semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603",
|
||||
"semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236",
|
||||
"semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1",
|
||||
"displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877",
|
||||
"displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e",
|
||||
"configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff",
|
||||
"configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994",
|
||||
"profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282",
|
||||
"profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571",
|
||||
"profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88",
|
||||
"attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2",
|
||||
"singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f",
|
||||
"nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce",
|
||||
"nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842",
|
||||
"nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1",
|
||||
"nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90",
|
||||
"harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412",
|
||||
"harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249",
|
||||
"harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94",
|
||||
"transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523",
|
||||
"nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37",
|
||||
"nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44",
|
||||
"nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a",
|
||||
"providerLaunchSourceSha256": "fa8db1c42c378898287c141d8cea9069166981e369c9b9c51422cb6177de5a2c",
|
||||
"launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296",
|
||||
"runtimeRequestBridgeSourceSha256": "cda99b35c428c4189c116c62b34349fc92e6f2de951d58847197b2d17bdb57de",
|
||||
"runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295",
|
||||
"runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211",
|
||||
"guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f",
|
||||
"nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918",
|
||||
"profileCompatibilitySourceSha256": "d96c3b809d083f9d43936af05e6bc14b1755c55f255c8e5967b8c16661f34ffa",
|
||||
"installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d",
|
||||
"assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c",
|
||||
"assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f",
|
||||
"credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161",
|
||||
"sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80",
|
||||
"profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456",
|
||||
"metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727",
|
||||
"permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1",
|
||||
"semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811",
|
||||
"runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec",
|
||||
"runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8",
|
||||
"runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea",
|
||||
"cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca",
|
||||
"agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba",
|
||||
"nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b"
|
||||
}
|
||||
}
|
||||
+88
@@ -0,0 +1,88 @@
|
||||
{
|
||||
"commandDigest": "sha256:9934065aaee959a19f05e102d9cf6fc41288c15be18f2769f62af6d7cc8a25b7",
|
||||
"declaration": {
|
||||
"schema": "paperclip.acpx_profile_declaration.v1",
|
||||
"agent": "copilot",
|
||||
"agentProfileVersion": 79,
|
||||
"acpxVersion": "0.13.1",
|
||||
"agentServerVersion": "1.0.88",
|
||||
"protocolPolicyRevision": "copilot-agent-manual-v1",
|
||||
"acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1",
|
||||
"policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3",
|
||||
"distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107",
|
||||
"agentFilesBinding": "registered-runtime-context-v1",
|
||||
"systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1",
|
||||
"sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1",
|
||||
"innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a",
|
||||
"upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0",
|
||||
"patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679",
|
||||
"messageIdentityContract": "copilot-native-message-id-v1",
|
||||
"ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1",
|
||||
"permissionContextContract": "copilot-edit-permission-context-v1",
|
||||
"permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b",
|
||||
"permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668",
|
||||
"permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9",
|
||||
"permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f",
|
||||
"permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4",
|
||||
"semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2",
|
||||
"semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791",
|
||||
"semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5",
|
||||
"toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706",
|
||||
"semanticNormalizedInputContract": "validated-forwarded-input-commit-v1",
|
||||
"semanticSidecarSourceSha256": "7d6e948b7cff4df52a9afc916db46ace0f42b4b2d3134ba0c21ee0a928857e62",
|
||||
"semanticDirectDriverSourceSha256": "d683429427dd8a1a5efe50952018ea486f808bdbf0a92fc5946ee267fe2c72ee",
|
||||
"semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975",
|
||||
"semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e",
|
||||
"semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59",
|
||||
"semanticValidatorsSourceSha256": "b20ec8ee6e7165117233ce4421a975bb9d3f276220ce31655c0b9a717cb2d9cb",
|
||||
"semanticSchemaBundleSourceSha256": "81ef98ca7b34896a0f76390ac7af3dd8ef7609fe6f38d54bd6a69373d50fe40b",
|
||||
"semanticSidecarProtocolSourceSha256": "26dd0cfbef8b2b401de6b7862a4af7f243fb703e23678a1a5ed8fa68d4784d42",
|
||||
"semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1",
|
||||
"displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877",
|
||||
"displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e",
|
||||
"configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff",
|
||||
"configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994",
|
||||
"profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282",
|
||||
"profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571",
|
||||
"profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88",
|
||||
"attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2",
|
||||
"singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f",
|
||||
"nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce",
|
||||
"nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842",
|
||||
"nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1",
|
||||
"nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90",
|
||||
"harnessSessionContractSourceSha256": "baba1a210b863518f01e0c875606ed37f9467c433222251438578bd999bc43b5",
|
||||
"harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249",
|
||||
"harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94",
|
||||
"transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523",
|
||||
"nativeBackendSourceSha256": "7495d9898d22dca1ebf6336361d332bafefc4498f6e49f1135025fad846c8586",
|
||||
"nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44",
|
||||
"nativeInstructionsSourceSha256": "82ac6e5f7275b4ef79fb166c7330efc504e80cab3d7e9081e4e40a8b570bc50b",
|
||||
"providerLaunchSourceSha256": "c744235b3067ea75973523f78f1e13399337bc155f1775af694e34782d5c9bef",
|
||||
"launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296",
|
||||
"runtimeRequestBridgeSourceSha256": "2d167741defe23d6b2b0647a54405331ae4457d298936f0796733628ad79d7bc",
|
||||
"runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295",
|
||||
"runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211",
|
||||
"guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f",
|
||||
"nativeExecutionSourceSha256": "d60b8ea63bad975c0144f0702b2749d7db9a7da50680a6147f20d4d11f2d5297",
|
||||
"profileCompatibilitySourceSha256": "9000cf5a7f4191cf28699d4deb23e17cd02507a3b2f12c12a7051a0fea782496",
|
||||
"installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d",
|
||||
"assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c",
|
||||
"assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f",
|
||||
"credentialBindingSourceSha256": "3a93267146bfc3c489f43262386d6dcb05ec4e3a8b1765414cd43d80d0c64d4a",
|
||||
"admissionDiagnosticsSourceSha256": "df8f1aa4f2d1bc63b2635bdd09cb8165c8b7681ddcc2618435c7c08ca90475af",
|
||||
"grokAdmissionSourceSha256": "f1e46cf2e1e36b7704bd797b678e3db8db2003f97bdb9d827cfa3d7662698777",
|
||||
"sidecarErrorSourceSha256": "17948ad74350dc487ab20d24a0fb89daf0b426b98396454fff0e3856e922d82c",
|
||||
"profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456",
|
||||
"metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727",
|
||||
"permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1",
|
||||
"semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811",
|
||||
"preparedHarnessHomeSourceSha256": "bb286f7ddc6e9597c56d46bf7ff6e3b556c5b680d7e147fd46518724432b083e",
|
||||
"runtimeHostSourceSha256": "13f9ac382a279a67adb8974c36bbed38fe43d9b93b87bd7a05e879ead7f03d63",
|
||||
"runtimeAdapterSourceSha256": "7c47c05306ebe7d1ca6b5d046a96f5726a48adec4d27e7c16788f505f25b5db2",
|
||||
"runtimeSandboxSourceSha256": "04a49a24e3828ce8fca908de15e31ae932b8b1b4f7ecec0308627f6a8da9ba20",
|
||||
"cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca",
|
||||
"agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba",
|
||||
"nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
{
|
||||
"commandDigest": "sha256:7b8a6a31925256d225ea3d72fdcfd81b70a45c4b67d3eb0ee5073efff9ac8f8b",
|
||||
"declaration": {
|
||||
"schema": "paperclip.acpx_profile_declaration.v1",
|
||||
"agent": "copilot",
|
||||
"agentProfileVersion": 80,
|
||||
"acpxVersion": "0.13.1",
|
||||
"agentServerVersion": "1.0.88",
|
||||
"protocolPolicyRevision": "copilot-agent-manual-v1",
|
||||
"acpxPatchSha256": "1e7dae8c83bf8620f53f2bea13c175eeff95e7e955783f6f529bb1dcbdcdb6d2",
|
||||
"policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3",
|
||||
"distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107",
|
||||
"agentFilesBinding": "registered-runtime-context-v1",
|
||||
"systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1",
|
||||
"sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1",
|
||||
"innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a",
|
||||
"upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0",
|
||||
"patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679",
|
||||
"messageIdentityContract": "copilot-native-message-id-v1",
|
||||
"ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1",
|
||||
"permissionContextContract": "copilot-edit-permission-context-v1",
|
||||
"permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b",
|
||||
"permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668",
|
||||
"permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9",
|
||||
"permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f",
|
||||
"permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4",
|
||||
"semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2",
|
||||
"semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791",
|
||||
"semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5",
|
||||
"toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706",
|
||||
"semanticNormalizedInputContract": "validated-forwarded-input-commit-v1",
|
||||
"semanticSidecarSourceSha256": "7d6e948b7cff4df52a9afc916db46ace0f42b4b2d3134ba0c21ee0a928857e62",
|
||||
"semanticDirectDriverSourceSha256": "d683429427dd8a1a5efe50952018ea486f808bdbf0a92fc5946ee267fe2c72ee",
|
||||
"semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975",
|
||||
"semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e",
|
||||
"semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59",
|
||||
"semanticValidatorsSourceSha256": "b20ec8ee6e7165117233ce4421a975bb9d3f276220ce31655c0b9a717cb2d9cb",
|
||||
"semanticSchemaBundleSourceSha256": "81ef98ca7b34896a0f76390ac7af3dd8ef7609fe6f38d54bd6a69373d50fe40b",
|
||||
"semanticSidecarProtocolSourceSha256": "26dd0cfbef8b2b401de6b7862a4af7f243fb703e23678a1a5ed8fa68d4784d42",
|
||||
"semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1",
|
||||
"displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877",
|
||||
"displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e",
|
||||
"configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff",
|
||||
"configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994",
|
||||
"profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282",
|
||||
"profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571",
|
||||
"profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88",
|
||||
"attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2",
|
||||
"singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f",
|
||||
"nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce",
|
||||
"nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842",
|
||||
"nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1",
|
||||
"nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90",
|
||||
"harnessSessionContractSourceSha256": "baba1a210b863518f01e0c875606ed37f9467c433222251438578bd999bc43b5",
|
||||
"harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249",
|
||||
"harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94",
|
||||
"transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523",
|
||||
"nativeBackendSourceSha256": "7495d9898d22dca1ebf6336361d332bafefc4498f6e49f1135025fad846c8586",
|
||||
"nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44",
|
||||
"nativeInstructionsSourceSha256": "82ac6e5f7275b4ef79fb166c7330efc504e80cab3d7e9081e4e40a8b570bc50b",
|
||||
"providerLaunchSourceSha256": "c744235b3067ea75973523f78f1e13399337bc155f1775af694e34782d5c9bef",
|
||||
"launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296",
|
||||
"runtimeRequestBridgeSourceSha256": "2d167741defe23d6b2b0647a54405331ae4457d298936f0796733628ad79d7bc",
|
||||
"runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295",
|
||||
"runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211",
|
||||
"guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f",
|
||||
"nativeExecutionSourceSha256": "d60b8ea63bad975c0144f0702b2749d7db9a7da50680a6147f20d4d11f2d5297",
|
||||
"profileCompatibilitySourceSha256": "c3bde2939555d6826b983928735d5a2e3d2949e4235d5910de16a1abf80ad7e9",
|
||||
"installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d",
|
||||
"assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c",
|
||||
"assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f",
|
||||
"credentialBindingSourceSha256": "3a93267146bfc3c489f43262386d6dcb05ec4e3a8b1765414cd43d80d0c64d4a",
|
||||
"admissionDiagnosticsSourceSha256": "df8f1aa4f2d1bc63b2635bdd09cb8165c8b7681ddcc2618435c7c08ca90475af",
|
||||
"grokAdmissionSourceSha256": "f1e46cf2e1e36b7704bd797b678e3db8db2003f97bdb9d827cfa3d7662698777",
|
||||
"sidecarErrorSourceSha256": "17948ad74350dc487ab20d24a0fb89daf0b426b98396454fff0e3856e922d82c",
|
||||
"profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456",
|
||||
"metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727",
|
||||
"permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1",
|
||||
"semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811",
|
||||
"preparedHarnessHomeSourceSha256": "bb286f7ddc6e9597c56d46bf7ff6e3b556c5b680d7e147fd46518724432b083e",
|
||||
"runtimeHostSourceSha256": "13f9ac382a279a67adb8974c36bbed38fe43d9b93b87bd7a05e879ead7f03d63",
|
||||
"runtimeAdapterSourceSha256": "7c47c05306ebe7d1ca6b5d046a96f5726a48adec4d27e7c16788f505f25b5db2",
|
||||
"runtimeSandboxSourceSha256": "04a49a24e3828ce8fca908de15e31ae932b8b1b4f7ecec0308627f6a8da9ba20",
|
||||
"cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca",
|
||||
"agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba",
|
||||
"nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b"
|
||||
}
|
||||
}
|
||||
@@ -7,7 +7,6 @@ import { eq, sql } from "drizzle-orm";
|
||||
import { createDatabaseResponseCapture, installDatabaseResponseCapture } from "@paperclipai/db/response-capture";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
activityLog,
|
||||
agents,
|
||||
approvals,
|
||||
assets,
|
||||
@@ -17,11 +16,8 @@ import {
|
||||
createDb,
|
||||
decisionQueueItems,
|
||||
decisionQueues,
|
||||
decisionArchiveNotificationOutbox,
|
||||
decisionRetention,
|
||||
decisions,
|
||||
decisionTriage,
|
||||
decisionTriageEvents,
|
||||
documents,
|
||||
heartbeatRunEvents,
|
||||
heartbeatRuns,
|
||||
@@ -70,35 +66,9 @@ describeEmbeddedPostgres("attention service", () => {
|
||||
}, 30_000);
|
||||
|
||||
afterEach(async () => {
|
||||
await db.delete(inboxDismissals);
|
||||
await db.delete(decisionArchiveNotificationOutbox);
|
||||
await db.delete(decisionRetention);
|
||||
await db.delete(decisionTriageEvents);
|
||||
await db.delete(decisionTriage);
|
||||
await db.delete(decisionQueueItems);
|
||||
await db.delete(decisionQueues);
|
||||
await db.delete(decisions);
|
||||
await db.delete(issueThreadInteractions);
|
||||
await db.delete(issueApprovals);
|
||||
await db.delete(issueAttachments);
|
||||
await db.delete(issueDocuments);
|
||||
await db.delete(heartbeatRunEvents);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(budgetIncidents);
|
||||
await db.delete(budgetPolicies);
|
||||
await db.delete(joinRequests);
|
||||
await db.delete(invites);
|
||||
await db.delete(issueRecoveryActions);
|
||||
await db.delete(issueRelations);
|
||||
await db.delete(activityLog);
|
||||
await db.delete(approvals);
|
||||
await db.delete(issues);
|
||||
await db.delete(assets);
|
||||
await db.delete(documents);
|
||||
await db.delete(projectWorkspaces);
|
||||
await db.delete(projects);
|
||||
await db.delete(agents);
|
||||
await db.delete(companies);
|
||||
|
||||
|
||||
await db.execute(sql`TRUNCATE TABLE companies CASCADE`);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
|
||||
@@ -2793,15 +2793,6 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
const lateCallback = new Promise<void>((resolve) => {
|
||||
releaseLateCallback = resolve;
|
||||
});
|
||||
const pendingOperationDrain =
|
||||
outcome === "late_callback"
|
||||
? vi
|
||||
.spyOn(
|
||||
paperclipRunner,
|
||||
"drainRetainedRunnerdMaintenanceOperations",
|
||||
)
|
||||
.mockImplementation(() => lateCallback)
|
||||
: null;
|
||||
let physicalCleanupFinished = false;
|
||||
mockRetainedNativeCleanup.mockImplementationOnce(
|
||||
async (cleanupDb, input) => {
|
||||
@@ -2809,6 +2800,9 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
expect(input).toEqual({ companyId, runId });
|
||||
await held;
|
||||
physicalCleanupFinished = true;
|
||||
|
||||
|
||||
if (outcome === "late_callback") await lateCallback;
|
||||
if (outcome === "rejected")
|
||||
throw new Error("maintenance-test-closed-failure");
|
||||
return {
|
||||
@@ -2873,10 +2867,8 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
expect(mockAdapterExecute).not.toHaveBeenCalled();
|
||||
expect(mockExecutePaperclipNativeSession).not.toHaveBeenCalled();
|
||||
release();
|
||||
if (pendingOperationDrain) {
|
||||
await vi.waitFor(() =>
|
||||
expect(pendingOperationDrain).toHaveBeenCalled(),
|
||||
);
|
||||
if (outcome === "late_callback") {
|
||||
await vi.waitFor(() => expect(physicalCleanupFinished).toBe(true));
|
||||
expect(physicalCleanupFinished).toBe(true);
|
||||
expect(drainFinished).toBe(false);
|
||||
expect(heartbeat.getTaskDrainStatus().quiescent).toBe(false);
|
||||
@@ -2916,7 +2908,6 @@ describeEmbeddedPostgres("heartbeat orphaned process recovery", () => {
|
||||
await startup;
|
||||
await drain;
|
||||
await heartbeat.drainActiveRunExecutions();
|
||||
pendingOperationDrain?.mockRestore();
|
||||
}
|
||||
});
|
||||
},
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { findRetainedNativeSessionCleanupCandidates } from "../services/native-runtime/retained-native-session-cleanup.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
@@ -27,7 +28,7 @@ import {
|
||||
reconcileRetainedNativeSessionCleanups,
|
||||
} from "../services/native-runtime/native-finalization-reconciler.js";
|
||||
import { PaperclipControlPlanePort } from "../services/native-runtime/paperclip-control-plane-port.js";
|
||||
import { assertRetainedNativeSourceArchiveSettled } from "../services/native-runtime/native-session-executor.js";
|
||||
import { assertRetainedNativeSourceArchiveSettled, reconcileRetainedNativeSessionCleanup } from "../services/native-runtime/native-session-executor.js";
|
||||
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_FINALIZATION_TEST_DATABASE_URL");
|
||||
|
||||
@@ -392,6 +393,7 @@ describe("retained native cleanup discovery", () => {
|
||||
|
||||
async function candidate(
|
||||
options: {
|
||||
owner?: { companyId: string; agentId: string };
|
||||
run?: Partial<typeof heartbeatRuns.$inferInsert>;
|
||||
coordinator?: Partial<typeof nativeRunFinalizations.$inferInsert>;
|
||||
schemaStatus?: string;
|
||||
@@ -399,6 +401,7 @@ describe("retained native cleanup discovery", () => {
|
||||
revision?: number;
|
||||
} = {},
|
||||
) {
|
||||
const owner = options.owner ?? { companyId, agentId };
|
||||
const runId = randomUUID();
|
||||
const issueId = options.issueId ?? randomUUID();
|
||||
const contractId = randomUUID();
|
||||
@@ -406,13 +409,13 @@ describe("retained native cleanup discovery", () => {
|
||||
.insert(issues)
|
||||
.values({
|
||||
id: issueId,
|
||||
companyId,
|
||||
companyId: owner.companyId,
|
||||
title: "Retained session",
|
||||
assigneeAgentId: agentId,
|
||||
assigneeAgentId: owner.agentId,
|
||||
});
|
||||
await db.insert(completionContracts).values({
|
||||
id: contractId,
|
||||
companyId,
|
||||
companyId: owner.companyId,
|
||||
issueId,
|
||||
revision: options.revision ?? 1,
|
||||
schemaVersion: "paperclip.completion-contract.v1",
|
||||
@@ -427,8 +430,8 @@ describe("retained native cleanup discovery", () => {
|
||||
});
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId,
|
||||
agentId,
|
||||
companyId: owner.companyId,
|
||||
agentId: owner.agentId,
|
||||
runtimeMode: "native",
|
||||
status: "succeeded",
|
||||
nativeIssueId: issueId,
|
||||
@@ -444,7 +447,7 @@ describe("retained native cleanup discovery", () => {
|
||||
const [result] = await db
|
||||
.insert(nativeRunResults)
|
||||
.values({
|
||||
companyId,
|
||||
companyId: owner.companyId,
|
||||
issueId,
|
||||
runId,
|
||||
completionContractId: contractId,
|
||||
@@ -457,13 +460,13 @@ describe("retained native cleanup discovery", () => {
|
||||
const [assessment] = await db
|
||||
.insert(workAssessments)
|
||||
.values({
|
||||
companyId,
|
||||
companyId: owner.companyId,
|
||||
issueId,
|
||||
runId,
|
||||
contractId,
|
||||
resultId: result!.id,
|
||||
triggerKind: "native_result",
|
||||
triggerActorCompanyId: companyId,
|
||||
triggerActorCompanyId: owner.companyId,
|
||||
priorIssueStatus: "in_progress",
|
||||
priorStatusVersion: 0,
|
||||
policyVersion: "phase6-v3",
|
||||
@@ -474,7 +477,7 @@ describe("retained native cleanup discovery", () => {
|
||||
const [decision] = await db
|
||||
.insert(statusDecisions)
|
||||
.values({
|
||||
companyId,
|
||||
companyId: owner.companyId,
|
||||
issueId,
|
||||
runId,
|
||||
assessmentId: assessment!.id,
|
||||
@@ -488,7 +491,7 @@ describe("retained native cleanup discovery", () => {
|
||||
})
|
||||
.returning();
|
||||
await db.insert(nativeRunFinalizations).values({
|
||||
companyId,
|
||||
companyId: owner.companyId,
|
||||
issueId,
|
||||
runId,
|
||||
phase: "committed",
|
||||
@@ -500,6 +503,73 @@ describe("retained native cleanup discovery", () => {
|
||||
return runId;
|
||||
}
|
||||
|
||||
it("does not misclassify a real competing row lock as ineligible retained history", async () => {
|
||||
const runId = await candidate();
|
||||
await db.transaction(async tx => {
|
||||
await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update");
|
||||
await expect(reconcileRetainedNativeSessionCleanup(db, { companyId, runId }))
|
||||
.rejects.toMatchObject({ cause: { code: "55P03" } });
|
||||
});
|
||||
});
|
||||
|
||||
it("reconstructs future lease deadlines without authorizing early legacy cleanup", async () => {
|
||||
const leaseExpiresAt = new Date(Date.now() + 60_000);
|
||||
const runId = await candidate({ coordinator: { leaseOwner: "other-owner", leaseExpiresAt } });
|
||||
const scope = { companyId, runId };
|
||||
expect(await findRetainedNativeSessionCleanupCandidates(db, { scope, limit: 100 })).toEqual([]);
|
||||
expect(await findRetainedNativeSessionCleanupCandidates(db, { scope, limit: 100, includeLeased: true }))
|
||||
.toEqual([expect.objectContaining({ runId, leaseOwner: "other-owner", leaseExpiresAt })]);
|
||||
const cleanup = vi.fn();
|
||||
await reconcileRetainedNativeSessionCleanups(db, { cleanup });
|
||||
expect(cleanup).not.toHaveBeenCalled();
|
||||
await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() - 1000) })
|
||||
.where(eq(nativeRunFinalizations.runId, runId));
|
||||
expect(await findRetainedNativeSessionCleanupCandidates(db, { scope, limit: 100 }))
|
||||
.toEqual([expect.objectContaining({ runId })]);
|
||||
});
|
||||
|
||||
it("scopes reconstruction to a company, run, task, or agent without leaking another tenant", async () => {
|
||||
const other = { companyId: randomUUID(), agentId: randomUUID() };
|
||||
await db.insert(companies).values({ id: other.companyId, name: "Other cleanup tenant", issuePrefix: "NCO" });
|
||||
await db.insert(agents).values({ id: other.agentId, companyId: other.companyId, name: "Other cleanup", adapterType: "codex_local" });
|
||||
try {
|
||||
const first = await candidate();
|
||||
const second = await candidate();
|
||||
const foreign = await candidate({ owner: other });
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, first));
|
||||
for (const scope of [{ companyId, runId: first }, { companyId, issueId: run.nativeIssueId! }]) {
|
||||
expect((await findRetainedNativeSessionCleanupCandidates(db, { scope, limit: 100, includeLeased: true }))
|
||||
.map(row => row.runId)).toEqual([first]);
|
||||
}
|
||||
expect((await findRetainedNativeSessionCleanupCandidates(db, { scope: { companyId, agentId }, limit: 100 }))
|
||||
.map(row => row.runId).sort()).toEqual([first, second].sort());
|
||||
expect(await findRetainedNativeSessionCleanupCandidates(db, {
|
||||
scope: { companyId, runId: foreign }, limit: 100,
|
||||
})).toEqual([]);
|
||||
expect((await findRetainedNativeSessionCleanupCandidates(db, { scope: other, limit: 100 }))
|
||||
.map(row => row.runId)).toEqual([foreign]);
|
||||
} finally {
|
||||
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, other.companyId));
|
||||
}
|
||||
});
|
||||
|
||||
it("pages reconstruction past retained history without wrapping or admitting previously attempted cleanup", async () => {
|
||||
const ids = [await candidate(), await candidate(), await candidate()].sort();
|
||||
await candidate({ coordinator: { recoveryHistory: [{ kind: "native_cleanup_runner_epoch", phase: "spawned" }] } });
|
||||
let after: string | undefined;
|
||||
const found: string[] = [];
|
||||
for (;;) {
|
||||
const rows = await findRetainedNativeSessionCleanupCandidates(db, {
|
||||
scope: { companyId }, after, limit: 1, includeLeased: true,
|
||||
});
|
||||
if (!rows.length) break;
|
||||
found.push(rows[0].runId);
|
||||
after = rows[0].runId;
|
||||
expect(found.length).toBeLessThanOrEqual(ids.length);
|
||||
}
|
||||
expect(found).toEqual(ids);
|
||||
});
|
||||
|
||||
it("discovers exact committed results even with stale nativePhase or privately recovered errors", async () => {
|
||||
const stale = await candidate();
|
||||
const recovered = await candidate({
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { retainedNativeSessionCleanupWork } from "../services/native-runtime/retained-native-session-cleanup-work.js";
|
||||
import { pendingNativeGovernance } from "../services/native-runtime/native-governance.js";
|
||||
import { withNativeWorkspaceFinalizationOwnership } from "../services/native-runtime/native-workspace-finalization-ownership.js";
|
||||
import { ensureNativeCompletionContract } from "../services/native-runtime/completion-contracts.js";
|
||||
@@ -3072,6 +3073,55 @@ describe("P6-31 Section 18.13 executable status-authority corpus", () => {
|
||||
expect((await issueService(db).getById(seeded.issueId))!.status).toBe(delivery === "queued" ? "in_progress" : "done");
|
||||
}, 30_000);
|
||||
|
||||
it("wakes retained cleanup after a decision signal was consumed before terminal projection", async () => {
|
||||
const seeded = await seedFixture({ ...corpus.fixtures[0]!, id: `cleanup-projection-${randomUUID()}` });
|
||||
await db.update(completionContracts).set({ risk: "low", completionAuthority: "agent_claim_policy" })
|
||||
.where(eq(completionContracts.id, seeded.contractId!));
|
||||
await db.update(heartbeatRuns).set({ status: "running", finishedAt: null, errorCode: "adapter_failed",
|
||||
error: "provider_transport_failed: runner did not durably suspend before checkpoint" })
|
||||
.where(eq(heartbeatRuns.id, seeded.runId));
|
||||
const cleanup = vi.fn(async (input: { companyId: string; runId: string }) => {
|
||||
await db.update(nativeRunFinalizations).set({ recoveryHistory: [{ kind: "native_cleanup_maintenance", phase: "settled" }] })
|
||||
.where(eq(nativeRunFinalizations.runId, input.runId));
|
||||
return { runId: input.runId, status: "settled" as const };
|
||||
});
|
||||
const errors: unknown[] = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
const worker = retainedNativeSessionCleanupWork(db, cleanup).register(coordinator, () => true);
|
||||
const flush = () => vi.waitFor(() => {
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 10_000 });
|
||||
let consumedBeforeProjection = false;
|
||||
const transaction = db.transaction.bind(db);
|
||||
const boundary = vi.spyOn(db, "transaction").mockImplementation(async (...args) => {
|
||||
const result = await transaction(...args);
|
||||
if (!consumedBeforeProjection) {
|
||||
const [row] = await db.select({ phase: nativeRunFinalizations.phase, status: heartbeatRuns.status })
|
||||
.from(nativeRunFinalizations).innerJoin(heartbeatRuns, eq(heartbeatRuns.id, nativeRunFinalizations.runId))
|
||||
.where(eq(heartbeatRuns.id, seeded.runId));
|
||||
if (row?.phase === "committed" && row.status === "running") {
|
||||
consumedBeforeProjection = true;
|
||||
|
||||
|
||||
await flush();
|
||||
expect(cleanup).not.toHaveBeenCalled();
|
||||
}
|
||||
}
|
||||
return result;
|
||||
});
|
||||
try {
|
||||
await worker.ready;
|
||||
await finalizeNativeRun({ db, runId: seeded.runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true,
|
||||
skipNativeResponsePresentation: true, skipNativeReviewResponse: true });
|
||||
expect(consumedBeforeProjection).toBe(true);
|
||||
await flush();
|
||||
expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId: seeded.runId });
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId)))[0].status).toBe("succeeded");
|
||||
expect(errors).toEqual([]);
|
||||
} finally { boundary.mockRestore(); worker.stop(); await coordinator.stop(); }
|
||||
}, 30_000);
|
||||
|
||||
it("completes a new merge run after an old CI review and bounds repeated incomplete results", async () => {
|
||||
const seeded = await seedAutomaticReview();
|
||||
const [sourceRun] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, seeded.runId));
|
||||
|
||||
@@ -0,0 +1,631 @@
|
||||
import { heartbeatService } from "../services/heartbeat.js";
|
||||
import * as nativeExecutor from "../services/native-runtime/native-session-executor.js";
|
||||
import { finalizeNativeRun } from "../services/native-runtime/native-run-finalizer.js";
|
||||
import { notifyNativeFinalizationWork } from "../services/native-finalization-notifications.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import os from "node:os";
|
||||
import { PROCESS_IDENTITY_RECORDED, releaseSettledNativeRunProcess } from "../services/native-local-process-stop.js";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { agents, companies, completionContracts, createDb, environmentLeases, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations,
|
||||
nativeRunResults, statusDecisions, workAssessments, type Db } from "@paperclipai/db";
|
||||
import { settleAbsentSandbox, settleStopOnlyCleanup } from "../services/sandbox-stop-and-retain.js";
|
||||
import { settleNativeWorkspaceExportResume } from "../services/native-runtime/native-workspace-export-resume.js";
|
||||
import type { SandboxAbsenceProof } from "../services/sandbox-absence-proof.js";
|
||||
import { environmentService } from "../services/environments.js";
|
||||
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
|
||||
import { idleWorkSnapshot } from "../services/task-admission.js";
|
||||
import { notifyRetainedNativeSessionCleanup, releaseRetainedNativeSessionCleanup } from "../services/retained-native-session-cleanup-notifications.js";
|
||||
import { retainedNativeSessionCleanupWork } from "../services/native-runtime/retained-native-session-cleanup-work.js";
|
||||
import { reconcileRetainedNativeSessionCleanups, runRetainedNativeSessionCleanup, type NativeCleanupOutcome } from "../services/native-runtime/retained-native-session-cleanup.js";
|
||||
import { reconcileRetainedNativeSessionCleanup } from "../services/native-runtime/native-session-executor.js";
|
||||
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
|
||||
|
||||
function deferred() {
|
||||
let resolve!: () => void;
|
||||
const promise = new Promise<void>(done => { resolve = done; });
|
||||
return { promise, resolve };
|
||||
}
|
||||
|
||||
describe("retained native cleanup scheduled lifecycle", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
|
||||
let worker: ReturnType<ReturnType<typeof retainedNativeSessionCleanupWork>["register"]> | undefined;
|
||||
const companyId = randomUUID(), agentId = randomUUID();
|
||||
const closeError = "provider_transport_failed: runner did not durably suspend before checkpoint";
|
||||
let admitted = true;
|
||||
const errors: unknown[] = [];
|
||||
async function settle(input: { runId: string; companyId: string }): Promise<NativeCleanupOutcome> {
|
||||
await db.update(nativeRunFinalizations).set({ leaseOwner: null, leaseExpiresAt: null,
|
||||
recoveryHistory: [{ kind: "native_cleanup_maintenance", phase: "settled" }] })
|
||||
.where(eq(nativeRunFinalizations.runId, input.runId));
|
||||
return { runId: input.runId, status: "settled" };
|
||||
}
|
||||
const cleanup = vi.fn(settle);
|
||||
beforeAll(async () => {
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_RETAINED_CLEANUP_TEST_DATABASE_URL");
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("paperclip-retained-cleanup-");
|
||||
db = createDb(external ?? temporary!.connectionString);
|
||||
await db.insert(companies).values({ id: companyId, name: "Retained cleanup lifecycle", issuePrefix: `RC${randomUUID().slice(0, 8)}` });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Cleanup agent", adapterType: "codex_local" });
|
||||
}, 30_000);
|
||||
beforeEach(() => {
|
||||
admitted = true; errors.length = 0; cleanup.mockReset().mockImplementation(settle);
|
||||
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true });
|
||||
});
|
||||
afterEach(async () => {
|
||||
worker?.stop(); await coordinator?.stop(); worker = undefined; coordinator = undefined;
|
||||
vi.useRealTimers(); vi.restoreAllMocks();
|
||||
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, companyId));
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
|
||||
async function start() {
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, canReconcile: () => true,
|
||||
onError: error => errors.push(error) });
|
||||
worker = retainedNativeSessionCleanupWork(db, cleanup).register(coordinator, () => admitted);
|
||||
await worker.ready;
|
||||
}
|
||||
async function flush() {
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect(coordinator!.nextWakeAt()).toBeNull();
|
||||
}, { timeout: 15_000 });
|
||||
}
|
||||
async function quiet(ms = 120_000) {
|
||||
const previous = db.$client.options.debug;
|
||||
const queries: string[] = [];
|
||||
db.$client.options.debug = (_id, query) => { queries.push(query); };
|
||||
try { await vi.advanceTimersByTimeAsync(ms); await delay(10); expect(queries).toEqual([]); }
|
||||
finally { db.$client.options.debug = previous; }
|
||||
}
|
||||
async function commitCandidate() {
|
||||
const runId = randomUUID();
|
||||
return db.transaction(async tx => {
|
||||
await notifyNativeFinalizationWork(tx, { companyId, runId });
|
||||
return candidate({ runId }, tx as unknown as Db);
|
||||
});
|
||||
}
|
||||
async function candidate(
|
||||
options: {
|
||||
runId?: string;
|
||||
owner?: { companyId: string; agentId: string };
|
||||
run?: Partial<typeof heartbeatRuns.$inferInsert>;
|
||||
coordinator?: Partial<typeof nativeRunFinalizations.$inferInsert>;
|
||||
schemaStatus?: string;
|
||||
issueId?: string;
|
||||
revision?: number;
|
||||
} = {},
|
||||
writer: Db = db,
|
||||
) {
|
||||
const owner = options.owner ?? { companyId, agentId };
|
||||
const runId = options.runId ?? randomUUID();
|
||||
const issueId = options.issueId ?? randomUUID();
|
||||
const contractId = randomUUID();
|
||||
if (!options.issueId) await writer
|
||||
.insert(issues)
|
||||
.values({
|
||||
id: issueId,
|
||||
companyId: owner.companyId,
|
||||
title: "Retained session",
|
||||
assigneeAgentId: owner.agentId,
|
||||
});
|
||||
await writer.insert(completionContracts).values({
|
||||
id: contractId,
|
||||
companyId: owner.companyId,
|
||||
issueId,
|
||||
revision: options.revision ?? 1,
|
||||
schemaVersion: "paperclip.completion-contract.v1",
|
||||
policyVersion: "phase6-v3",
|
||||
risk: "standard",
|
||||
completionAuthority: "server_arbiter",
|
||||
incompleteCriteriaPolicy: "preserve_non_terminal",
|
||||
contractJson: {},
|
||||
canonicalSha256: runId,
|
||||
createdByActorType: "system",
|
||||
createdByActorId: "test",
|
||||
});
|
||||
await writer.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId: owner.companyId,
|
||||
agentId: owner.agentId,
|
||||
runtimeMode: "native",
|
||||
status: "succeeded",
|
||||
nativeIssueId: issueId,
|
||||
nativeSessionId: runId,
|
||||
nativePhase: "terminal_failure",
|
||||
completionContractId: contractId,
|
||||
completionContractSha256: runId,
|
||||
finishedAt: new Date(),
|
||||
errorCode: "adapter_failed",
|
||||
error: closeError,
|
||||
...options.run,
|
||||
});
|
||||
const [result] = await writer
|
||||
.insert(nativeRunResults)
|
||||
.values({
|
||||
companyId: owner.companyId,
|
||||
issueId,
|
||||
runId,
|
||||
completionContractId: contractId,
|
||||
serverFingerprint: runId,
|
||||
canonicalSha256: runId,
|
||||
schemaStatus: options.schemaStatus ?? "accepted",
|
||||
resultJson: {},
|
||||
})
|
||||
.returning();
|
||||
const [assessment] = await writer
|
||||
.insert(workAssessments)
|
||||
.values({
|
||||
companyId: owner.companyId,
|
||||
issueId,
|
||||
runId,
|
||||
contractId,
|
||||
resultId: result!.id,
|
||||
triggerKind: "native_result",
|
||||
triggerActorCompanyId: owner.companyId,
|
||||
priorIssueStatus: "in_progress",
|
||||
priorStatusVersion: 0,
|
||||
policyVersion: "phase6-v3",
|
||||
assessmentJson: {},
|
||||
inputDigest: runId,
|
||||
})
|
||||
.returning();
|
||||
const [decision] = await writer
|
||||
.insert(statusDecisions)
|
||||
.values({
|
||||
companyId: owner.companyId,
|
||||
issueId,
|
||||
runId,
|
||||
assessmentId: assessment!.id,
|
||||
decisionVersion: options.revision ?? 1,
|
||||
policyVersion: "phase6-v3",
|
||||
fromStatus: "in_progress",
|
||||
toStatus: "in_review",
|
||||
reasonCode: "prior_status_terminal_preserved",
|
||||
decisionJson: {},
|
||||
decisionDigest: runId,
|
||||
})
|
||||
.returning();
|
||||
await writer.insert(nativeRunFinalizations).values({
|
||||
companyId: owner.companyId,
|
||||
issueId,
|
||||
runId,
|
||||
phase: "committed",
|
||||
resultId: result!.id,
|
||||
assessmentId: assessment!.id,
|
||||
decisionId: decision!.id,
|
||||
...options.coordinator,
|
||||
});
|
||||
return runId;
|
||||
}
|
||||
|
||||
|
||||
it("becomes SQL-quiet with no work, then processes the first committed candidate", async () => {
|
||||
await start(); await flush(); await quiet();
|
||||
const runId = await commitCandidate();
|
||||
await flush();
|
||||
expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId });
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("leaves retained unprovable history quiet through the real physical cleanup boundary", async () => {
|
||||
await candidate();
|
||||
cleanup.mockImplementation(input => reconcileRetainedNativeSessionCleanup(db, input));
|
||||
await start(); await flush();
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("keeps dormant retained cleanup quiet on unscoped finalization hints and resumes from local release", async () => {
|
||||
const runId = await candidate();
|
||||
cleanup.mockResolvedValue({ runId, status: "deferred", waitFor: "session_release" });
|
||||
await start();
|
||||
await flush();
|
||||
await quiet();
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
expect(await retainedNativeSessionCleanupWork(db, cleanup).load({ companyId, runId })).toHaveLength(1);
|
||||
const debug = db.$client.options.debug;
|
||||
const queries: string[] = [];
|
||||
db.$client.options.debug = (_connection, query) => {
|
||||
if (query.startsWith("select") && query.includes("native_run_finalizations")) queries.push(query);
|
||||
};
|
||||
try {
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await db.transaction(tx => notifyNativeFinalizationWork(tx, { companyId }));
|
||||
await flush();
|
||||
}
|
||||
expect(queries).toEqual([]);
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
} finally { db.$client.options.debug = debug; }
|
||||
cleanup.mockImplementation(settle);
|
||||
releaseRetainedNativeSessionCleanup(db, { companyId, runId });
|
||||
await flush();
|
||||
expect(cleanup).toHaveBeenCalledTimes(2);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it.each(["session_release", "environment_release", "maintenance_settlement"] as const)(
|
||||
"waits quietly for %s and then reactivates only the affected run", async waitFor => {
|
||||
const runId = await candidate();
|
||||
cleanup.mockResolvedValueOnce({ runId, status: "deferred", waitFor });
|
||||
await start(); await flush(); await quiet();
|
||||
releaseRetainedNativeSessionCleanup(db, { companyId, runId });
|
||||
await flush();
|
||||
expect(cleanup).toHaveBeenCalledTimes(2);
|
||||
await quiet();
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["single", "run"] as const)("reactivates after the real %s lease release commits, but not rollback", async method => {
|
||||
const runId = await candidate();
|
||||
const [lease] = await db.insert(environmentLeases).values({ companyId, heartbeatRunId: runId,
|
||||
status: "active", provider: "local" }).returning();
|
||||
cleanup.mockImplementation(async input => {
|
||||
const [current] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease!.id));
|
||||
return current!.status === "active"
|
||||
? { runId: input.runId, status: "deferred", waitFor: "environment_release" }
|
||||
: settle(input);
|
||||
});
|
||||
const release = async (writer: Db) => {
|
||||
const service = environmentService(writer);
|
||||
if (method === "single") await service.releaseLease(lease!.id);
|
||||
else await service.releaseLeasesForRun(runId);
|
||||
};
|
||||
await start(); await flush(); await quiet();
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
await expect(db.transaction(async tx => {
|
||||
await release(tx as unknown as Db);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
throw new Error("rollback lease release");
|
||||
})).rejects.toThrow("rollback lease release");
|
||||
await flush(); await quiet();
|
||||
const afterRollback = cleanup.mock.calls.length;
|
||||
await db.transaction(async tx => {
|
||||
await release(tx as unknown as Db);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(cleanup).toHaveBeenCalledTimes(afterRollback);
|
||||
});
|
||||
await flush();
|
||||
expect(cleanup).toHaveBeenCalledTimes(afterRollback + 1);
|
||||
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0]!.recoveryHistory)
|
||||
.toEqual([{ kind: "native_cleanup_maintenance", phase: "settled" }]);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("keeps pending cleanup quiet through metadata updates and stale release attempts", async () => {
|
||||
const runId = await candidate(), attemptId = randomUUID();
|
||||
const [lease] = await db.insert(environmentLeases).values({ companyId, heartbeatRunId: runId,
|
||||
status: "pending_cleanup", provider: "local", metadata: { pendingCleanupAttemptId: attemptId } }).returning();
|
||||
cleanup.mockResolvedValueOnce({ runId, status: "deferred", waitFor: "environment_release" });
|
||||
await start(); await flush();
|
||||
const service = environmentService(db);
|
||||
await service.updateLeaseMetadata(lease!.id, { pendingCleanupAttemptId: attemptId, pendingCleanupInFlight: true });
|
||||
await service.releaseLease(lease!.id, "pending_cleanup", { expectedPendingCleanupAttemptId: attemptId, cleanupStatus: "failed" });
|
||||
expect(await service.releaseLease(lease!.id, "released", { expectedPendingCleanupAttemptId: randomUUID(), cleanupStatus: "success" })).toBeNull();
|
||||
await flush(); await quiet();
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
expect(await service.releaseLease(lease!.id, "released", { expectedPendingCleanupAttemptId: attemptId, cleanupStatus: "success" })).not.toBeNull();
|
||||
await flush();
|
||||
expect(cleanup).toHaveBeenCalledTimes(2);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it.each(["stop-only", "export-resume", "absent"] as const)("reactivates after real %s settlement with outer transaction ownership", async method => {
|
||||
const ownerToken = randomUUID();
|
||||
const runId = await candidate({ run: { runnerProfileJson: { nativeWorkspaceFinalizationOwner: { token: ownerToken } } } });
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
const leaseId = randomUUID(), providerLeaseId = randomUUID(), attemptId = randomUUID(), pluginId = randomUUID();
|
||||
const binding = { companyId, runId, leaseId, provider: "daytona" as const, providerLeaseId };
|
||||
const marker = { ...binding, requestId: attemptId, pluginId,
|
||||
schema: method === "export-resume" ? "paperclip.workspace-export-resume.v2" : "paperclip.sandbox-stop-and-retain.v1",
|
||||
resultId: randomUUID() };
|
||||
const [lease] = await db.insert(environmentLeases).values({ id: leaseId, companyId, heartbeatRunId: runId,
|
||||
issueId: run!.nativeIssueId, provider: "daytona", providerLeaseId, status: "pending_cleanup", cleanupStatus: "failed",
|
||||
metadata: { pluginId, pendingCleanupAttemptId: attemptId,
|
||||
[method === "export-resume" ? "nativeWorkspaceExportResume" : "sandboxStopAndRetain"]: marker } }).returning();
|
||||
const proof: SandboxAbsenceProof = { ...binding, issueId: run!.nativeIssueId!, ownerToken, v: 1,
|
||||
iss: "paperclip-cloud", aud: "paperclip-sandbox-absence/v1", sub: "fixture", absent: true,
|
||||
iat: Math.floor(Date.now() / 1000), exp: Math.floor(Date.now() / 1000) + 60 };
|
||||
cleanup.mockImplementation(async input => {
|
||||
const [current] = await db.select().from(environmentLeases).where(eq(environmentLeases.id, leaseId));
|
||||
return current!.status === "pending_cleanup"
|
||||
? { runId: input.runId, status: "deferred", waitFor: "environment_release" }
|
||||
: settle(input);
|
||||
});
|
||||
const release = async (writer: Db) => {
|
||||
const options = { attemptId, receipt: { providerLeaseId, state: "stopped" } };
|
||||
if (method === "absent") expect(await settleAbsentSandbox(writer, lease!, { proof, receipt: "verified-upstream" })).toBe(true);
|
||||
else if (method === "export-resume") expect(await settleNativeWorkspaceExportResume(writer, lease!, options)).not.toBeNull();
|
||||
else expect(await settleStopOnlyCleanup(writer, lease!, options)).not.toBeNull();
|
||||
};
|
||||
await start(); await flush(); await quiet();
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
await expect(db.transaction(async tx => {
|
||||
await release(tx as unknown as Db);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
throw new Error("rollback settlement");
|
||||
})).rejects.toThrow("rollback settlement");
|
||||
await flush(); await quiet();
|
||||
const afterRollback = cleanup.mock.calls.length;
|
||||
await db.transaction(async tx => {
|
||||
await release(tx as unknown as Db);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(cleanup).toHaveBeenCalledTimes(afterRollback);
|
||||
});
|
||||
await flush();
|
||||
expect(cleanup).toHaveBeenCalledTimes(afterRollback + 1);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("reactivates immediately after verified process release commits without waiting for its probe", async () => {
|
||||
const pid = 2_000_000_000, startedAt = new Date();
|
||||
const runId = await candidate({ run: { status: "failed", processPid: pid, processGroupId: pid,
|
||||
processStartedAt: startedAt, runnerProfileJson: { sessionCheckpoint: {} } } });
|
||||
await db.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: 0,
|
||||
eventType: PROCESS_IDENTITY_RECORDED, payload: { targetKind: "local", hostname: os.hostname(),
|
||||
processPid: pid, processGroupId: pid, processStartedAt: startedAt.toISOString() } });
|
||||
await db.update(heartbeatRuns).set({ nextEventSeq: 1 }).where(eq(heartbeatRuns.id, runId));
|
||||
const originalKill = process.kill.bind(process);
|
||||
vi.spyOn(process, "kill").mockImplementation((id, signal) => {
|
||||
if ((id === pid || id === -pid) && signal === 0) throw Object.assign(new Error("gone"), { code: "ESRCH" });
|
||||
return originalKill(id, signal);
|
||||
});
|
||||
cleanup.mockImplementation(async input => {
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId));
|
||||
return run!.processPid !== null
|
||||
? { runId: input.runId, status: "deferred", waitFor: "process_exit" }
|
||||
: settle(input);
|
||||
});
|
||||
await start();
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
expect(coordinator!.nextWakeAt()).not.toBeNull();
|
||||
await expect(db.transaction(async tx => {
|
||||
expect(await releaseSettledNativeRunProcess(tx as unknown as Db, runId)).toBe("released");
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
throw new Error("rollback process release");
|
||||
})).rejects.toThrow("rollback process release");
|
||||
await vi.waitFor(async () => {
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(cleanup).toHaveBeenCalledTimes(2);
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
});
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]!.processPid).toBe(pid);
|
||||
const afterRollback = cleanup.mock.calls.length;
|
||||
await db.transaction(async tx => {
|
||||
expect(await releaseSettledNativeRunProcess(tx as unknown as Db, runId)).toBe("released");
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(cleanup).toHaveBeenCalledTimes(afterRollback);
|
||||
});
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
await vi.waitFor(() => expect(cleanup).toHaveBeenCalledTimes(afterRollback + 1));
|
||||
await flush(); await quiet();
|
||||
});
|
||||
|
||||
it("reactivates when the real finalizer projects an already committed result to a terminal run", async () => {
|
||||
const runId = await candidate({ run: { status: "running", finishedAt: null } });
|
||||
await db.update(nativeRunResults).set({ resultJson: { terminal: { runTerminalState: "succeeded" } } })
|
||||
.where(eq(nativeRunResults.runId, runId));
|
||||
await start(); await flush(); await quiet();
|
||||
expect(cleanup).not.toHaveBeenCalled();
|
||||
const finalize = (writer: Db) => finalizeNativeRun({ db: writer, runId, workspaceFinalizeStatus: "succeeded",
|
||||
projectRunStatus: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true });
|
||||
await expect(db.transaction(async tx => {
|
||||
await finalize(tx as unknown as Db);
|
||||
await vi.advanceTimersByTimeAsync(1000);
|
||||
expect(cleanup).not.toHaveBeenCalled();
|
||||
throw new Error("rollback final projection");
|
||||
})).rejects.toThrow("rollback final projection");
|
||||
await flush(); await quiet();
|
||||
expect(cleanup).not.toHaveBeenCalled();
|
||||
await finalize(db); await flush();
|
||||
expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId });
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
|
||||
expect(run!.status).toBe("succeeded");
|
||||
expect(run!.error).toBeNull();
|
||||
expect(run!.resultJson?.recoveredExecutionFailure).toMatchObject({ errorCode: "adapter_failed", error: closeError });
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("reconstructs a lease deadline across restart and does no SQL before it", async () => {
|
||||
const leaseExpiresAt = new Date(Date.now() + 60_000);
|
||||
const runId = await candidate({ coordinator: { leaseOwner: "other-owner", leaseExpiresAt } });
|
||||
await start();
|
||||
expect(cleanup).not.toHaveBeenCalled();
|
||||
expect(coordinator!.nextWakeAt()).toBe(leaseExpiresAt.getTime());
|
||||
await quiet(10_000);
|
||||
worker!.stop(); await coordinator!.stop();
|
||||
await start();
|
||||
expect(coordinator!.nextWakeAt()).toBe(leaseExpiresAt.getTime());
|
||||
await quiet(10_000);
|
||||
await vi.advanceTimersByTimeAsync(40_000); await flush();
|
||||
expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId });
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("retains only the concrete process-exit probe and disarms it after settlement", async () => {
|
||||
const runId = await candidate();
|
||||
cleanup.mockResolvedValueOnce({ runId, status: "deferred", waitFor: "process_exit" });
|
||||
await start();
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
await quiet(20_000);
|
||||
await vi.advanceTimersByTimeAsync(10_000); await flush();
|
||||
expect(cleanup).toHaveBeenCalledTimes(2);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("does not run before outer commit and clears rolled-back intent", async () => {
|
||||
await start(); await flush();
|
||||
const entered = deferred(), release = deferred();
|
||||
const transaction = db.transaction(async outer => {
|
||||
await outer.transaction(async tx => {
|
||||
await notifyRetainedNativeSessionCleanup(tx, { companyId });
|
||||
await candidate({}, tx as unknown as Db);
|
||||
});
|
||||
entered.resolve(); await release.promise;
|
||||
throw new Error("rollback fixture");
|
||||
});
|
||||
const rolledBack = expect(transaction).rejects.toThrow("rollback fixture");
|
||||
await entered.promise;
|
||||
await vi.advanceTimersByTimeAsync(10_000);
|
||||
expect(cleanup).not.toHaveBeenCalled();
|
||||
release.resolve(); await rolledBack; await flush(); await quiet();
|
||||
const runId = await commitCandidate(); await flush();
|
||||
expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId });
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("retains the first write during drain without passive SQL until resume", async () => {
|
||||
await start(); await flush(); admitted = false;
|
||||
const runId = await commitCandidate();
|
||||
await vi.advanceTimersByTimeAsync(1000); await delay(10);
|
||||
expect(cleanup).not.toHaveBeenCalled();
|
||||
await quiet();
|
||||
admitted = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
|
||||
expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId });
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("retains an operational failure across drain and retries after resume", async () => {
|
||||
await candidate(); cleanup.mockRejectedValueOnce(new Error("connection lost"));
|
||||
await start(); expect(errors).toHaveLength(1); errors.length = 0;
|
||||
admitted = false; await quiet();
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
admitted = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
|
||||
expect(cleanup).toHaveBeenCalledTimes(2);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("keeps a committed wake arriving during cleanup and never overlaps worker dispatch", async () => {
|
||||
await candidate();
|
||||
const entered = deferred(), release = deferred();
|
||||
let active = 0, maximum = 0;
|
||||
cleanup.mockImplementation(async input => {
|
||||
maximum = Math.max(maximum, ++active);
|
||||
try { entered.resolve(); await release.promise; return await settle(input); }
|
||||
finally { active--; }
|
||||
});
|
||||
const starting = start(); await entered.promise;
|
||||
await commitCandidate(); release.resolve(); await starting; await flush();
|
||||
expect(cleanup).toHaveBeenCalledTimes(2); expect(maximum).toBe(1);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("joins the original manual startup operation instead of performing cleanup twice", async () => {
|
||||
await candidate();
|
||||
const entered = deferred(), release = deferred();
|
||||
cleanup.mockImplementation(async input => { entered.resolve(); await release.promise; return settle(input); });
|
||||
const manual = reconcileRetainedNativeSessionCleanups(db, { cleanup });
|
||||
await entered.promise;
|
||||
const work = retainedNativeSessionCleanupWork(db, cleanup);
|
||||
const [entry] = await work.load();
|
||||
const joined = work.check(entry);
|
||||
try { expect(cleanup).toHaveBeenCalledTimes(1); }
|
||||
finally { release.resolve(); }
|
||||
await expect(joined).resolves.toBeNull(); await manual;
|
||||
expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("serializes different cleanup inspections sharing a database lane", async () => {
|
||||
const first = await candidate(), second = await candidate();
|
||||
const entered = deferred(), release = deferred();
|
||||
cleanup.mockImplementationOnce(async input => { entered.resolve(); await release.promise; return settle(input); });
|
||||
const active = runRetainedNativeSessionCleanup(db, { companyId, runId: first }, cleanup);
|
||||
await entered.promise;
|
||||
const queued = runRetainedNativeSessionCleanup(db, { companyId, runId: second }, cleanup);
|
||||
try { await delay(10); expect(cleanup).toHaveBeenCalledTimes(1); }
|
||||
finally { release.resolve(); }
|
||||
await Promise.all([active, queued]);
|
||||
expect(cleanup.mock.calls.map(([input]) => input.runId)).toEqual([first, second]);
|
||||
});
|
||||
|
||||
it("joins scheduled physical cleanup through the assembled heartbeat shutdown owner", async () => {
|
||||
const runId = await candidate();
|
||||
const entered = deferred(), release = deferred();
|
||||
const physical = vi.spyOn(nativeExecutor, "reconcileRetainedNativeSessionCleanup").mockImplementation(async (_db, input) => {
|
||||
entered.resolve(); await release.promise; return settle(input);
|
||||
});
|
||||
const heartbeat = heartbeatService(db);
|
||||
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => errors.push(error) });
|
||||
worker = heartbeat.registerRetainedNativeSessionCleanupWork(coordinator, () => true);
|
||||
await entered.promise;
|
||||
let drained = false;
|
||||
const draining = heartbeat.drainActiveRunExecutions().then(() => { drained = true; });
|
||||
try {
|
||||
await delay(10);
|
||||
expect(drained).toBe(false);
|
||||
expect(physical).toHaveBeenCalledExactlyOnceWith(db, { companyId, runId });
|
||||
} finally { release.resolve(); }
|
||||
await worker.ready; await draining; await flush();
|
||||
expect(drained).toBe(true); await quiet();
|
||||
});
|
||||
|
||||
it("waits on the original in-flight cleanup when shutdown begins", async () => {
|
||||
await candidate();
|
||||
const entered = deferred(), release = deferred();
|
||||
cleanup.mockImplementation(async input => { entered.resolve(); await release.promise; return settle(input); });
|
||||
const starting = start(); await entered.promise;
|
||||
let stopped = false;
|
||||
worker!.stop(); const stopping = coordinator!.stop().then(() => { stopped = true; });
|
||||
try { await delay(10); expect(stopped).toBe(false); }
|
||||
finally { release.resolve(); }
|
||||
await Promise.all([starting, stopping]);
|
||||
expect(stopped).toBe(true); expect(cleanup).toHaveBeenCalledTimes(1);
|
||||
releaseRetainedNativeSessionCleanup(db, { companyId });
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("replaces an old lease alarm when a committed change releases it early", async () => {
|
||||
const runId = await candidate({ coordinator: { leaseOwner: "old-owner", leaseExpiresAt: new Date(Date.now() + 60_000) } });
|
||||
await start(); expect(cleanup).not.toHaveBeenCalled();
|
||||
await db.transaction(async tx => {
|
||||
await notifyNativeFinalizationWork(tx, { companyId, runId });
|
||||
await tx.update(nativeRunFinalizations).set({ leaseOwner: null, leaseExpiresAt: null }).where(eq(nativeRunFinalizations.runId, runId));
|
||||
});
|
||||
await flush(); expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId });
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("cancels a saved alarm when a committed owner marks the candidate operator-required", async () => {
|
||||
const runId = await candidate({ coordinator: { leaseOwner: "old-owner", leaseExpiresAt: new Date(Date.now() + 60_000) } });
|
||||
await start();
|
||||
await db.transaction(async tx => {
|
||||
await notifyNativeFinalizationWork(tx, { companyId, runId });
|
||||
await tx.update(nativeRunFinalizations).set({ recoveryHistory: [{ kind: "native_cleanup_source_archive", phase: "operator_required" }] })
|
||||
.where(eq(nativeRunFinalizations.runId, runId));
|
||||
});
|
||||
await flush(); expect(cleanup).not.toHaveBeenCalled(); await quiet();
|
||||
});
|
||||
|
||||
it("retains startup discovery during drain and reconstructs only after admission resumes", async () => {
|
||||
const runId = await candidate(); admitted = false;
|
||||
await start(); await quiet(); expect(cleanup).not.toHaveBeenCalled();
|
||||
admitted = true; await vi.advanceTimersByTimeAsync(30_000); await flush();
|
||||
expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId }); await quiet();
|
||||
});
|
||||
|
||||
it("drains a backlog across discovery pages and becomes quiet with the completed history retained", async () => {
|
||||
const runIds: string[] = [];
|
||||
for (let index = 0; index < 103; index++) runIds.push(await candidate());
|
||||
await start(); await flush();
|
||||
expect(cleanup.mock.calls.map(([input]) => input.runId).sort()).toEqual(runIds.sort());
|
||||
expect(await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, companyId))).toHaveLength(103);
|
||||
await quiet();
|
||||
});
|
||||
|
||||
it("retries failed startup discovery without treating an unavailable database as empty", async () => {
|
||||
const runId = await candidate();
|
||||
vi.spyOn(db, "select").mockImplementationOnce(() => { throw new Error("discovery connection lost"); });
|
||||
await start(); expect(errors).toHaveLength(1); errors.length = 0;
|
||||
expect(cleanup).not.toHaveBeenCalled();
|
||||
await quiet(20_000); await vi.advanceTimersByTimeAsync(10_000); await flush();
|
||||
expect(cleanup).toHaveBeenCalledExactlyOnceWith({ companyId, runId }); await quiet();
|
||||
});
|
||||
});
|
||||
@@ -64,7 +64,7 @@ const {
|
||||
retryMs?: number;
|
||||
run: (signal?: AbortSignal) => Promise<unknown>;
|
||||
hasPending?: () => Promise<boolean>;
|
||||
}) => ({ ready: task.run(new AbortController().signal).catch(error => { deliveryWorkErrors.push({ queue, error }); }), wake: vi.fn() }));
|
||||
}) => ({ ready: task.run(new AbortController().signal).catch(error => { deliveryWorkErrors.push({ queue, error }); }), wake: vi.fn(), stop: vi.fn(async () => {}) }));
|
||||
const createAppMock = vi.fn(async () => Object.assign((_: unknown, __: unknown) => {}, {
|
||||
locals: {
|
||||
fastResponses: { hasPending: vi.fn(async () => false), sweepPending: vi.fn(async () => undefined) },
|
||||
@@ -103,6 +103,7 @@ const {
|
||||
reason: null,
|
||||
}));
|
||||
const heartbeatServiceMock = {
|
||||
drainActiveRunExecutions: vi.fn(async () => undefined),
|
||||
reconcileCostAccounting: vi.fn(async () => ({ scanned: 0, accounted: 0 })),
|
||||
resolveSchedulingSuppression: resolveHeartbeatSchedulingSuppressionMock,
|
||||
recoverNativeRunsAfterRestart: vi.fn(async () => ({
|
||||
@@ -116,6 +117,7 @@ const {
|
||||
reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })),
|
||||
registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerRetainedNativeSessionCleanupWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })),
|
||||
registerNativeStatusReassessmentWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerNativeReviewResponseWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })),
|
||||
registerNativeResponsePresentationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })),
|
||||
@@ -572,6 +574,17 @@ describe("startServer feedback export wiring", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it.each([false, true])("registers retained cleanup with its shutdown subscription only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
expect(heartbeatServiceMock.registerRetainedNativeSessionCleanupWork).toHaveBeenCalledTimes(enabled ? 1 : 0);
|
||||
if (enabled) {
|
||||
expect(heartbeatServiceMock.registerRetainedNativeSessionCleanupWork.mock.calls[0][1]()).toBe(true);
|
||||
const worker = heartbeatServiceMock.registerRetainedNativeSessionCleanupWork.mock.results[0].value;
|
||||
expect(fakeServer.on).toHaveBeenCalledWith("close", worker.stop);
|
||||
}
|
||||
});
|
||||
|
||||
it.each([false, true])("registers native reassessment only when maintenance is enabled (%s)", async enabled => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled }));
|
||||
await startServer(); await settleStartupHousekeeping();
|
||||
@@ -1033,7 +1046,7 @@ describe("startServer feedback export wiring", () => {
|
||||
intervalCallback?.();
|
||||
await new Promise<void>((resolve) => setImmediate(resolve));
|
||||
expect(heartbeatServiceMock.sweepStaleIssueLocks).toHaveBeenCalledTimes(2);
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true });
|
||||
expect(heartbeatServiceMock.reapOrphanedRuns).toHaveBeenLastCalledWith({ staleThresholdMs: 300_000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true });
|
||||
expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1);
|
||||
expect(retiredDetector).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
@@ -1189,6 +1202,37 @@ describe("startServer feedback export wiring", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("becomes ready while retained cleanup is held but joins it at shutdown", async () => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: true }));
|
||||
let releaseCleanup!: () => void;
|
||||
const ready = new Promise<void>(resolve => { releaseCleanup = resolve; });
|
||||
const stop = vi.fn();
|
||||
heartbeatServiceMock.registerRetainedNativeSessionCleanupWork.mockReturnValueOnce({ ready, stop });
|
||||
const timer = vi.spyOn(globalThis, "setInterval").mockImplementation(() => 1 as unknown as ReturnType<typeof setInterval>);
|
||||
let started = false;
|
||||
const starting = startServer().then(server => { started = true; return server; });
|
||||
let stopping: Promise<void> | undefined;
|
||||
let stopped = false;
|
||||
try {
|
||||
await vi.waitFor(() => expect(heartbeatServiceMock.registerRetainedNativeSessionCleanupWork).toHaveBeenCalledOnce());
|
||||
await settleStartupHousekeeping();
|
||||
expect(started).toBe(true);
|
||||
expect(getStartupRecoveryState().phase).toBe("ready");
|
||||
expect(heartbeatServiceMock.registerNativeStatusReassessmentWork).toHaveBeenCalledOnce();
|
||||
const server = await starting;
|
||||
stopping = server.shutdown().then(() => { stopped = true; });
|
||||
await settleStartupHousekeeping();
|
||||
expect(stop).toHaveBeenCalledOnce();
|
||||
expect(stopped).toBe(false);
|
||||
} finally {
|
||||
releaseCleanup();
|
||||
await starting;
|
||||
await stopping;
|
||||
timer.mockRestore();
|
||||
}
|
||||
expect(stopped).toBe(true);
|
||||
});
|
||||
|
||||
it("reports ready before startup housekeeping finishes and starts the scheduler after it", async () => {
|
||||
loadConfigMock.mockReturnValue(buildTestConfig({
|
||||
heartbeatSchedulerEnabled: true,
|
||||
|
||||
+12
-1
@@ -1404,6 +1404,7 @@ async function startServerWithDatabaseTeardown(
|
||||
});
|
||||
let unregisterTerminalWorkspaceProbe: (() => void) | undefined;
|
||||
let unsubscribeNativeResponsePresentation: (() => void) | undefined;
|
||||
let unsubscribeRetainedNativeSessionCleanup: (() => void) | undefined;
|
||||
let unsubscribeNativeWorkspaceSyncRetention: (() => void) | undefined;
|
||||
const fastResponses = app.locals.fastResponses as ReturnType<typeof import("./services/fast-responses.js").fastResponseService>;
|
||||
const fastResponseWorker = deliveryWork.register(DELIVERY_QUEUES.fastResponse, {
|
||||
@@ -1825,6 +1826,15 @@ async function startServerWithDatabaseTeardown(
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerInstructionDirectoryCleanupWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) {
|
||||
const worker = heartbeat.registerRetainedNativeSessionCleanupWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby());
|
||||
unsubscribeRetainedNativeSessionCleanup = worker.stop;
|
||||
server.on("close", worker.stop);
|
||||
|
||||
|
||||
trackHeartbeatSchedulerWork(worker.ready);
|
||||
}
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativeStatusReassessmentWork(deliveryWork,
|
||||
() => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready;
|
||||
if (config.heartbeatSchedulerEnabled) await heartbeat.registerNativeReviewResponseWork(deliveryWork,
|
||||
@@ -1976,7 +1986,7 @@ async function startServerWithDatabaseTeardown(
|
||||
// Periodically reap orphaned runs (5-min staleness threshold) and make sure
|
||||
// persisted queued work is still being driven forward.
|
||||
runPeriodicRecovery(() => heartbeat
|
||||
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true })
|
||||
.reapOrphanedRuns({ staleThresholdMs: 5 * 60 * 1000, skipScheduledRetryCancellation: true, skipCapturedInstructionCommit: true, skipInstructionDirectoryCleanup: true, skipStoppedInstructionCollection: true, skipCancelledWorkspacePreservation: true, skipObsoletePolicyReviews: true, skipAutomaticCompletionReviews: true, skipUnsafeWorkspaceExports: true, skipNativePendingEffects: true, skipNativeResponsePresentation: true, skipNativeReviewResponse: true, skipNativeStatusReassessment: true, skipRetainedNativeSessionCleanup: true })
|
||||
.then(() => heartbeat.promoteDueScheduledRetries())
|
||||
.then(async (promotion) => {
|
||||
await heartbeat.resumeQueuedRuns();
|
||||
@@ -2173,6 +2183,7 @@ async function startServerWithDatabaseTeardown(
|
||||
await skillSourceSync.stop();
|
||||
unsubscribeChatCompletions();
|
||||
unsubscribeNativeResponsePresentation?.();
|
||||
unsubscribeRetainedNativeSessionCleanup?.();
|
||||
await deliveryWork.stop();
|
||||
await workspaceRecoveryWork.stop();
|
||||
unregisterTerminalWorkspaceProbe?.();
|
||||
|
||||
@@ -2,6 +2,7 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db";
|
||||
|
||||
/** Topics for existing durable queues; register intent before their writes. */
|
||||
export const DELIVERY_QUEUES = {
|
||||
retainedNativeSessionCleanup: "retained-native-session-cleanup",
|
||||
nativeStatusReassessment: "native-status-reassessment",
|
||||
nativeReviewResponse: "native-review-response",
|
||||
nativeResponsePresentation: "native-response-presentation",
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js";
|
||||
import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js";
|
||||
import { writeEnvironmentCleanup, notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { withoutCleanupVerification } from "./cleanup-verification-proof.js";
|
||||
@@ -15,6 +16,7 @@ import {
|
||||
environmentLeases,
|
||||
environments,
|
||||
executionWorkspaces,
|
||||
heartbeatRuns,
|
||||
instanceSettings,
|
||||
issues,
|
||||
projects,
|
||||
@@ -1690,6 +1692,12 @@ export function environmentService(db: Db) {
|
||||
companyId: previous.companyId, issueId: previous.issueId ?? undefined,
|
||||
});
|
||||
}
|
||||
if (previous.heartbeatRunId && ["active", "pending_cleanup"].includes(previous.status)
|
||||
&& status !== "pending_cleanup") {
|
||||
await notifyRetainedNativeSessionCleanup(tx, {
|
||||
companyId: previous.companyId, runId: previous.heartbeatRunId,
|
||||
});
|
||||
}
|
||||
return tx.update(environmentLeases)
|
||||
.set({
|
||||
status,
|
||||
@@ -1810,8 +1818,11 @@ export function environmentService(db: Db) {
|
||||
status: Extract<EnvironmentLeaseStatus, "released" | "expired" | "failed"> = "released",
|
||||
): Promise<EnvironmentLease[]> => {
|
||||
const now = new Date();
|
||||
const rows = await writeEnvironmentCleanup(db, tx => tx
|
||||
.update(environmentLeases)
|
||||
const rows = await writeEnvironmentCleanup(db, async tx => {
|
||||
const [run] = await tx.select({ companyId: heartbeatRuns.companyId }).from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, heartbeatRunId)).limit(1);
|
||||
if (run) await notifyRetainedNativeSessionCleanup(tx, { companyId: run.companyId, runId: heartbeatRunId });
|
||||
return tx.update(environmentLeases)
|
||||
.set({
|
||||
status,
|
||||
releasedAt: now,
|
||||
@@ -1824,7 +1835,8 @@ export function environmentService(db: Db) {
|
||||
eq(environmentLeases.status, "active"),
|
||||
),
|
||||
)
|
||||
.returning());
|
||||
.returning();
|
||||
});
|
||||
return rows.map(toEnvironmentLease);
|
||||
},
|
||||
};
|
||||
|
||||
@@ -1023,6 +1023,7 @@ export function heartbeatService(
|
||||
sweepOrphanedActiveLeases,
|
||||
drainRunningRunsForShutdown,
|
||||
registerScheduledRetryCancellationWork,
|
||||
registerRetainedNativeSessionCleanupWork,
|
||||
} = createHeartbeatRecovery(db, {
|
||||
pluginWorkerManager: options.pluginWorkerManager,
|
||||
enterShutdown: () => { shutdownInProgress = true; },
|
||||
@@ -4092,6 +4093,9 @@ export function heartbeatService(
|
||||
triggerIssueMonitor,
|
||||
|
||||
reportRunActivity: clearDetachedRunWarning,
|
||||
registerRetainedNativeSessionCleanupWork: (
|
||||
coordinator: Parameters<typeof registerRetainedNativeSessionCleanupWork>[0], canRun: () => boolean,
|
||||
) => registerRetainedNativeSessionCleanupWork(coordinator, () => canRun() && readTaskDrain(new Date()) === null),
|
||||
registerNativeStatusReassessmentWork: (coordinator: Parameters<ReturnType<typeof nativeStatusReassessmentWork>["register"]>[0], canRun: () => boolean) =>
|
||||
nativeStatusReassessmentWork(db).register(coordinator, () => canRun() && readTaskDrain(new Date()) === null),
|
||||
registerNativeReviewResponseWork: (coordinator: Parameters<typeof registerNativeReviewResponseWork>[1], canRun: () => boolean) =>
|
||||
|
||||
@@ -26,7 +26,7 @@ const native = vi.hoisted(() => ({
|
||||
closeIdle: vi.fn(async () => ({ closed: 0, failed: 0 })),
|
||||
detach: vi.fn(async () => 0),
|
||||
finalizations: vi.fn(async (_db: Db, _ids?: string[], _options?: unknown) => undefined),
|
||||
cleanups: vi.fn(async () => undefined),
|
||||
cleanups: vi.fn<() => Promise<void>>(async () => undefined),
|
||||
drainMaintenance: vi.fn<() => Promise<void>>(async () => undefined),
|
||||
claims: vi.fn<typeof claimNativeRestartRecoveries>(async () => []),
|
||||
}));
|
||||
@@ -42,12 +42,12 @@ vi.mock("../native-runtime/index.js", async (importOriginal) => ({
|
||||
closeIdleWarmNativeSessionsForRestart: native.closeIdle,
|
||||
detachNativeSessionsForRestart: native.detach,
|
||||
reconcileNativeFinalizations: native.finalizations,
|
||||
reconcileRetainedNativeSessionCleanups: native.cleanups,
|
||||
claimNativeRestartRecoveries: native.claims,
|
||||
}));
|
||||
vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => ({
|
||||
...await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>(),
|
||||
drainRetainedRunnerdMaintenanceOperations: native.drainMaintenance,
|
||||
|
||||
vi.mock("../native-runtime/retained-native-session-cleanup.js", async (importOriginal) => ({
|
||||
...await importOriginal<typeof import("../native-runtime/retained-native-session-cleanup.js")>(),
|
||||
reconcileRetainedNativeSessionCleanups: native.cleanups,
|
||||
}));
|
||||
|
||||
function callbacks(db: Db) {
|
||||
@@ -107,7 +107,6 @@ beforeEach(() => {
|
||||
native.closeIdle.mockResolvedValue({ closed: 0, failed: 0 });
|
||||
native.finalizations.mockResolvedValue(undefined);
|
||||
native.cleanups.mockResolvedValue(undefined);
|
||||
native.drainMaintenance.mockResolvedValue(undefined);
|
||||
native.claims.mockResolvedValue([]);
|
||||
});
|
||||
|
||||
@@ -814,9 +813,9 @@ describePostgres("heartbeat recovery module ownership", () => {
|
||||
it("retains shutdown ownership until background native cleanup physically settles", async () => {
|
||||
const deps = callbacks(db);
|
||||
let settle!: () => void;
|
||||
native.drainMaintenance.mockReturnValueOnce(new Promise<void>(resolve => { settle = resolve; }));
|
||||
native.cleanups.mockReturnValueOnce(new Promise<void>(resolve => { settle = resolve; }));
|
||||
await createHeartbeatRecovery(db, deps).recoverNativeRunsAfterRestart();
|
||||
await vi.waitFor(() => expect(native.drainMaintenance).toHaveBeenCalled());
|
||||
await vi.waitFor(() => expect(native.cleanups).toHaveBeenCalled());
|
||||
expect(deps.activeRunExecutionPromises.size).toBe(1);
|
||||
settle();
|
||||
await Promise.all([...deps.activeRunExecutionPromises]);
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { retainedNativeSessionCleanupWork } from "../native-runtime/retained-native-session-cleanup-work.js";
|
||||
import { reconcileRetainedNativeSessionCleanups } from "../native-runtime/retained-native-session-cleanup.js";
|
||||
import { createScheduledRetryCancellationRecovery } from "./scheduled-retry-cancellation.js";
|
||||
import { pendingCleanupDeadlineSql } from "../environment-cleanup-state.js";
|
||||
import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js";
|
||||
@@ -57,9 +59,7 @@ import {
|
||||
type NativeRestartRecoveryClaim,
|
||||
reconcileNativeFinalizations,
|
||||
reconcileRetainedNativeSessionCleanup,
|
||||
reconcileRetainedNativeSessionCleanups,
|
||||
} from "../native-runtime/index.js";
|
||||
import { drainRetainedRunnerdMaintenanceOperations } from "../../vendor/paperclip-runner/index.js";
|
||||
import { runningProcesses } from "../../adapters/index.js";
|
||||
import { parseObject } from "../../adapters/utils.js";
|
||||
import {
|
||||
@@ -1920,12 +1920,23 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
).catch(() => undefined);
|
||||
}
|
||||
|
||||
function cleanupRetainedNativeSession(input: { companyId: string; runId: string }) {
|
||||
const operation = reconcileRetainedNativeSessionCleanup(db, input);
|
||||
|
||||
|
||||
const tracked = operation.then(() => undefined, () => undefined);
|
||||
activeRunExecutionPromises.add(tracked);
|
||||
void tracked.then(() => activeRunExecutionPromises.delete(tracked));
|
||||
return operation;
|
||||
}
|
||||
const retainedNativeCleanup = retainedNativeSessionCleanupWork(db, cleanupRetainedNativeSession);
|
||||
|
||||
function scheduleRetainedNativeSessionCleanup() {
|
||||
// The per-database sweep joins startup and periodic callers. One bounded
|
||||
// control-only repair must not hold up unrelated provider ingress or the
|
||||
// entire orphan reaper, but shutdown must still await its physical owner.
|
||||
const cleanup = reconcileRetainedNativeSessionCleanups(db, {
|
||||
cleanup: (input) => reconcileRetainedNativeSessionCleanup(db, input),
|
||||
cleanup: cleanupRetainedNativeSession,
|
||||
onError: (error, runId) => {
|
||||
logger.warn(
|
||||
{ err: error, runId },
|
||||
@@ -1936,18 +1947,14 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
.then(() => undefined)
|
||||
.catch((error) => {
|
||||
logger.warn({ err: error }, "retained native cleanup discovery failed");
|
||||
})
|
||||
// The bounded maintenance attempt may fail before an already-started
|
||||
// database callback settles. Keep shutdown ownership until the original
|
||||
// operations finish; their timeout cannot authorize closing the database.
|
||||
.finally(() => drainRetainedRunnerdMaintenanceOperations());
|
||||
});
|
||||
activeRunExecutionPromises.add(cleanup);
|
||||
void cleanup.finally(() => activeRunExecutionPromises.delete(cleanup));
|
||||
}
|
||||
|
||||
let settledProcessesDiscovered = false;
|
||||
|
||||
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean; skipNativeStatusReassessment?: boolean }) {
|
||||
async function reapOrphanedRuns(opts?: { staleThresholdMs?: number; skipScheduledRetryCancellation?: boolean; skipCapturedInstructionCommit?: boolean; skipInstructionDirectoryCleanup?: boolean; skipStoppedInstructionCollection?: boolean; skipCancelledWorkspacePreservation?: boolean; skipObsoletePolicyReviews?: boolean; skipAutomaticCompletionReviews?: boolean; skipUnsafeWorkspaceExports?: boolean; skipNativePendingEffects?: boolean; skipNativeResponsePresentation?: boolean; skipNativeReviewResponse?: boolean; skipNativeStatusReassessment?: boolean; skipRetainedNativeSessionCleanup?: boolean }) {
|
||||
if (!opts?.skipCancelledWorkspacePreservation) await preserveCancelledNativeWorkspaces(db);
|
||||
const staleThresholdMs = opts?.staleThresholdMs ?? 0;
|
||||
const now = new Date();
|
||||
@@ -1981,7 +1988,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
"failed to reconcile persisted native finalizations before orphan reaping",
|
||||
);
|
||||
});
|
||||
scheduleRetainedNativeSessionCleanup();
|
||||
if (!opts?.skipRetainedNativeSessionCleanup) scheduleRetainedNativeSessionCleanup();
|
||||
await reconcileRetainedWorkspaces(db, environmentRuntime, resumeRemoteStopComments);
|
||||
await dispatchPendingNativeStatusWakeups().catch((error) => {
|
||||
logger.warn(
|
||||
@@ -2584,6 +2591,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD
|
||||
}
|
||||
|
||||
return {
|
||||
registerRetainedNativeSessionCleanupWork: retainedNativeCleanup.register,
|
||||
registerScheduledRetryCancellationWork: scheduledRetryCancellation.register,
|
||||
sweepPendingCleanupLeases,
|
||||
markNativeOwnershipUnverified,
|
||||
|
||||
@@ -258,6 +258,38 @@ describe.skipIf(!support.supported)("heartbeat lifecycle database wiring", () =>
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
|
||||
it.each(["setRunStatus", "setRunStatusIfRunning"] as const)("wakes retained cleanup only after the outer %s terminal write commits", async operation => {
|
||||
const { run } = await fixture();
|
||||
const hints: unknown[] = [];
|
||||
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.retainedNativeSessionCleanup, (event, scopes) => {
|
||||
if (event === "settled") for (const scope of scopes ?? []) hints.push(JSON.parse(scope));
|
||||
});
|
||||
try {
|
||||
await db.transaction(async transaction => {
|
||||
const tx = transaction as unknown as Db;
|
||||
await createHeartbeatLifecycle(tx, callbacks(tx))[operation](run.id, "succeeded", { finishedAt: new Date() });
|
||||
expect(hints).toEqual([]);
|
||||
expect((await tx.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].status).toBe("succeeded");
|
||||
});
|
||||
expect(hints).toEqual([{ companyId: run.companyId, runId: run.id }]);
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0].finishedAt).not.toBeNull();
|
||||
hints.length = 0;
|
||||
|
||||
await createHeartbeatLifecycle(db, callbacks(db)).setRunStatus(run.id, "succeeded", { error: "repaired metadata" });
|
||||
expect(hints).toEqual([{ companyId: run.companyId, runId: run.id }]);
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
|
||||
it("does not wake retained cleanup for running progress", async () => {
|
||||
const { run } = await fixture();
|
||||
const settled = vi.fn();
|
||||
const unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.retainedNativeSessionCleanup, settled);
|
||||
try {
|
||||
await createHeartbeatLifecycle(db, callbacks(db)).setRunStatus(run.id, "running");
|
||||
expect(settled).not.toHaveBeenCalled();
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
|
||||
it("does not request accounting for an unchanged running status", async () => {
|
||||
const { run } = await fixture();
|
||||
const settled = vi.fn();
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyRetainedNativeSessionCleanup } from "../retained-native-session-cleanup-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
|
||||
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
||||
import { notifyInstructionCollection } from "../instruction-collection-notifications.js";
|
||||
@@ -1029,6 +1030,8 @@ export function createHeartbeatLifecycle(db: Db, dependencies: HeartbeatLifecycl
|
||||
await notifyScheduledRetryCancellation(tx, { companyId: previous.companyId, runId: previous.id });
|
||||
if (previous) await notifyInstructionCollection(tx, { companyId: previous.companyId, runId: previous.id });
|
||||
if (previous) await notifyCancelledWorkspacePreservation(tx, { companyId: previous.companyId, runId: previous.id });
|
||||
if (previous && isHeartbeatRunTerminalStatus(status))
|
||||
await notifyRetainedNativeSessionCleanup(tx, { companyId: previous.companyId, runId: previous.id });
|
||||
if (isHeartbeatRunTerminalStatus(status)) await notifyDeliveryWork(tx, DELIVERY_QUEUES.accounting, companyId);
|
||||
await notifyExecutionRunWork(tx);
|
||||
if (previous) await notifyChatRunMilestones(tx, { companyId: previous.companyId, runId: previous.id,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js";
|
||||
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
|
||||
import { notifyNativeReviewResponse } from "./native-review-response-notifications.js";
|
||||
import { notifyNativeResponsePresentation } from "./native-response-presentation-notifications.js";
|
||||
@@ -7,6 +8,7 @@ export type NativeFinalizationWorkScope = { companyId: string; runId?: string; i
|
||||
|
||||
export async function notifyNativeFinalizationWork(transaction: object, scope: NativeFinalizationWorkScope) {
|
||||
if (scope.issueId || scope.runId) {
|
||||
await notifyRetainedNativeSessionCleanup(transaction, scope);
|
||||
await notifyNativeStatusReassessment(transaction, scope);
|
||||
await notifyNativeResponsePresentation(transaction, scope);
|
||||
await notifyNativeReviewResponse(transaction, scope);
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { and, asc, desc, eq, gt, inArray, isNotNull, isNull, or, sql } from "drizzle-orm";
|
||||
import { environmentLeases, heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js";
|
||||
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
|
||||
import { nativeSessionProviderEvidence } from "./native-runtime/native-session-resume.js";
|
||||
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
@@ -174,6 +175,7 @@ export async function releaseSettledNativeRunProcess(db: Db, runId: string): Pro
|
||||
identity.payload.processPid !== run.processPid || identity.payload.processGroupId !== run.processGroupId) return "skipped";
|
||||
if (!(await recordNativeLocalProcessStop(tx, run))) return "pending";
|
||||
}
|
||||
await notifyRetainedNativeSessionCleanup(tx, { companyId: run.companyId, runId: run.id });
|
||||
await tx.update(heartbeatRuns).set({
|
||||
processPid: null, processGroupId: null, processStartedAt: null, updatedAt: new Date(),
|
||||
}).where(eq(heartbeatRuns.id, run.id));
|
||||
|
||||
@@ -68,144 +68,7 @@ function record(value: unknown): Record<string, unknown> {
|
||||
|
||||
export type NativeSessionResumeClaim = { runId: string; leaseOwner: string };
|
||||
|
||||
type NativeCleanupOutcome = {
|
||||
runId: string;
|
||||
status: "settled" | "not_eligible" | "operator_required";
|
||||
};
|
||||
type NativeCleanupSweep = {
|
||||
cursor: string | null;
|
||||
pending: Promise<NativeCleanupOutcome[]> | null;
|
||||
};
|
||||
const nativeCleanupSweeps = new WeakMap<Db, NativeCleanupSweep>();
|
||||
|
||||
/** Candidate discovery is not cleanup authority. The exact-state operation
|
||||
* claims its own durable lease and rechecks every physical owner. Keep this
|
||||
* lane joined and bounded, and advance even past ineligible candidates so one
|
||||
* damaged checkpoint cannot starve another company's recoverable session. */
|
||||
export function reconcileRetainedNativeSessionCleanups(
|
||||
db: Db,
|
||||
options: {
|
||||
cleanup: (input: {
|
||||
companyId: string;
|
||||
runId: string;
|
||||
}) => Promise<NativeCleanupOutcome>;
|
||||
onError?: (error: unknown, runId: string) => void;
|
||||
limit?: number;
|
||||
},
|
||||
): Promise<NativeCleanupOutcome[]> {
|
||||
let sweep = nativeCleanupSweeps.get(db);
|
||||
if (!sweep) {
|
||||
sweep = { cursor: null, pending: null };
|
||||
nativeCleanupSweeps.set(db, sweep);
|
||||
}
|
||||
if (sweep.pending) return sweep.pending;
|
||||
const owned = sweep;
|
||||
const limit = Number.isInteger(options.limit)
|
||||
? Math.max(1, Math.min(5, options.limit!))
|
||||
: 1;
|
||||
const operation = async () => {
|
||||
const selectCandidates = (cursor: string | null) =>
|
||||
db
|
||||
.select({
|
||||
runId: heartbeatRuns.id,
|
||||
companyId: heartbeatRuns.companyId,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.innerJoin(
|
||||
nativeRunFinalizations,
|
||||
and(
|
||||
eq(nativeRunFinalizations.runId, heartbeatRuns.id),
|
||||
eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
|
||||
eq(nativeRunFinalizations.issueId, heartbeatRuns.nativeIssueId),
|
||||
),
|
||||
)
|
||||
.innerJoin(
|
||||
nativeRunResults,
|
||||
and(
|
||||
eq(nativeRunResults.id, nativeRunFinalizations.resultId),
|
||||
eq(nativeRunResults.runId, heartbeatRuns.id),
|
||||
eq(nativeRunResults.companyId, heartbeatRuns.companyId),
|
||||
eq(nativeRunResults.issueId, heartbeatRuns.nativeIssueId),
|
||||
),
|
||||
)
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRuns.runtimeMode, "native"),
|
||||
inArray(heartbeatRuns.status, ["succeeded", "failed"]),
|
||||
isNotNull(heartbeatRuns.finishedAt),
|
||||
nativeRunnerOwnershipNotHeldCondition(),
|
||||
eq(nativeRunFinalizations.phase, "committed"),
|
||||
eq(nativeRunResults.schemaStatus, "accepted"),
|
||||
isNotNull(nativeRunFinalizations.assessmentId),
|
||||
isNotNull(nativeRunFinalizations.decisionId),
|
||||
isNull(nativeRunFinalizations.nextAttemptAt),
|
||||
or(
|
||||
isNull(nativeRunFinalizations.leaseOwner),
|
||||
isNull(nativeRunFinalizations.leaseExpiresAt),
|
||||
lte(nativeRunFinalizations.leaseExpiresAt, sql`now()`),
|
||||
),
|
||||
// The accepted-result projector preserves a recovered close failure
|
||||
// privately after clearing the visible successful run's stale error.
|
||||
sql`coalesce(${heartbeatRuns.errorCode}, ${heartbeatRuns.resultJson}->'recoveredExecutionFailure'->>'errorCode') in ('adapter_failed', 'provider_transport_failed')`,
|
||||
sql`coalesce(${heartbeatRuns.error}, ${heartbeatRuns.resultJson}->'recoveredExecutionFailure'->>'error') = 'provider_transport_failed: runner did not durably suspend before checkpoint'`,
|
||||
sql`not (${nativeRunFinalizations.recoveryHistory} @> '[{"kind":"native_cleanup_runner_epoch"}]'::jsonb)`,
|
||||
sql`not (${nativeRunFinalizations.recoveryHistory} @> '[{"kind":"native_cleanup_source_archive","phase":"operator_required"}]'::jsonb)`,
|
||||
// One legacy pre-ownership attempt may be inspected by the closed,
|
||||
// artifact-pinned no-launch verifier. Discovery grants no authority
|
||||
// to reuse its directory or start a provider. New recorded epochs,
|
||||
// staged attempts and ambiguous histories never enter this lane.
|
||||
sql`(
|
||||
select coalesce(
|
||||
jsonb_array_length(history.entries) = 0 or (
|
||||
jsonb_array_length(history.entries) = 2
|
||||
and history.entries->0->>'version' = '1'
|
||||
and history.entries->1->>'version' = '1'
|
||||
and history.entries->0->>'phase' = 'started'
|
||||
and history.entries->1->>'phase' = 'operator_required'
|
||||
and history.entries->1->>'code' = 'native_cleanup_maintenance_unproven'
|
||||
and history.entries->0->>'requestId' like 'native-cleanup:%'
|
||||
and history.entries->0->>'requestId' = history.entries->1->>'requestId'
|
||||
and history.entries->0->>'sourceFingerprint' ~ '^[a-f0-9]{64}$'
|
||||
), false
|
||||
)
|
||||
from (
|
||||
select coalesce(jsonb_agg(entry.value order by entry.ordinal), '[]'::jsonb) as entries
|
||||
from jsonb_array_elements(${nativeRunFinalizations.recoveryHistory})
|
||||
with ordinality as entry(value, ordinal)
|
||||
where entry.value->>'kind' = 'native_cleanup_maintenance'
|
||||
) history
|
||||
)`,
|
||||
...(cursor ? [gt(heartbeatRuns.id, cursor)] : []),
|
||||
),
|
||||
)
|
||||
.orderBy(asc(heartbeatRuns.id))
|
||||
.limit(limit);
|
||||
let candidates = await selectCandidates(owned.cursor);
|
||||
if (candidates.length === 0 && owned.cursor !== null) {
|
||||
owned.cursor = null;
|
||||
candidates = await selectCandidates(null);
|
||||
}
|
||||
const outcomes: NativeCleanupOutcome[] = [];
|
||||
for (const candidate of candidates) {
|
||||
owned.cursor = candidate.runId;
|
||||
try {
|
||||
outcomes.push(await options.cleanup(candidate));
|
||||
} catch (error) {
|
||||
options.onError?.(error, candidate.runId);
|
||||
}
|
||||
}
|
||||
return outcomes;
|
||||
};
|
||||
// Deferring the query one microtask installs the joined owner before any
|
||||
// asynchronous work begins. Cleanup never creates a heartbeat or wake.
|
||||
const pending = Promise.resolve()
|
||||
.then(operation)
|
||||
.finally(() => {
|
||||
if (owned.pending === pending) owned.pending = null;
|
||||
});
|
||||
owned.pending = pending;
|
||||
return pending;
|
||||
}
|
||||
export { reconcileRetainedNativeSessionCleanups } from "./retained-native-session-cleanup.js";
|
||||
|
||||
export async function dispatchNativeSessionResumptions(input: {
|
||||
db: Db;
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyRetainedNativeSessionCleanup } from "../retained-native-session-cleanup-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRunChange } from "../unsafe-workspace-export-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
|
||||
@@ -681,6 +682,7 @@ async function projectCommittedRun(input: {
|
||||
const now = new Date();
|
||||
const [updatedRun] = await input.db.transaction(async transaction => {
|
||||
const tx = transaction as unknown as Db;
|
||||
await notifyRetainedNativeSessionCleanup(tx, { companyId: input.run.companyId, runId: input.run.id });
|
||||
await notifyInstructionCollection(tx, { companyId: input.run.companyId, runId: input.run.id });
|
||||
await notifyCancelledWorkspacePreservation(tx, { companyId: input.run.companyId, runId: input.run.id });
|
||||
await notifyUnsafeWorkspaceExportRunChange(tx, input.run);
|
||||
@@ -1746,6 +1748,7 @@ export async function finalizeNativeRun(input: {
|
||||
.returning();
|
||||
const [updatedRun] = await input.db.transaction(async transaction => {
|
||||
const tx = transaction as unknown as Db;
|
||||
await notifyRetainedNativeSessionCleanup(tx, { companyId: run.companyId, runId: run.id });
|
||||
if (input.projectRunStatus) await notifyInstructionCollection(tx, { companyId: run.companyId, runId: run.id });
|
||||
if (input.projectRunStatus) await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId: run.id });
|
||||
if (input.projectRunStatus) await notifyUnsafeWorkspaceExportRunChange(tx, run);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { onRetainedNativeSessionCleanupRelease } from "../retained-native-session-cleanup-notifications.js";
|
||||
import * as runnerCheckpoint from "./native-runner-checkpoint.js";
|
||||
import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js";
|
||||
import * as environmentTaskRuntime from "../environment-task-runtime.js";
|
||||
@@ -34,6 +35,7 @@ import { inspect } from "node:util";
|
||||
import { dirname, join } from "node:path";
|
||||
import {
|
||||
heartbeatRuns,
|
||||
environmentLeases,
|
||||
heartbeatRunEvents,
|
||||
issueThreadInteractions,
|
||||
issues,
|
||||
@@ -188,6 +190,7 @@ const state = vi.hoisted(() => ({
|
||||
cleanup: vi.fn(),
|
||||
retireCleanup: vi.fn(),
|
||||
maintenanceIdle: vi.fn(() => true),
|
||||
drainMaintenance: vi.fn(async (): Promise<void> => undefined),
|
||||
createTransport: vi.fn((_options: RunnerTransportOptions) => ({
|
||||
transport: {},
|
||||
})),
|
||||
@@ -269,6 +272,7 @@ vi.mock("../../vendor/paperclip-runner/index.js", async (importOriginal) => {
|
||||
executeNativeSession: state.execute,
|
||||
settleRetainedRunnerdSession: state.cleanup,
|
||||
retainedRunnerdMaintenanceIsIdle: state.maintenanceIdle,
|
||||
drainRetainedRunnerdMaintenanceOperations: state.drainMaintenance,
|
||||
completeRetainedNativeSessionCleanup: state.retireCleanup,
|
||||
parsePaperclipQuestionSet: (value: unknown) => value,
|
||||
};
|
||||
@@ -403,6 +407,7 @@ import {
|
||||
import { onNativeRunnerEvidence } from "./native-runner-evidence-signal.js";
|
||||
|
||||
beforeEach(() => {
|
||||
state.drainMaintenance.mockReset().mockResolvedValue(undefined);
|
||||
usageRecorder.capture.mockReset().mockImplementation(async receipt => receipt);
|
||||
usageRecorder.create.mockReset().mockResolvedValue({ capture: usageRecorder.capture,
|
||||
complete: vi.fn(async () => ({ complete: true })), persistFailure: vi.fn(async () => undefined) });
|
||||
@@ -3880,6 +3885,35 @@ const execution = {
|
||||
} as NativeExecutionInputV1;
|
||||
|
||||
describe("retained native cleanup activation", () => {
|
||||
it("retains operation ownership through original callbacks after a cleanup failure", async () => {
|
||||
const failure = new Error("cleanup database failure");
|
||||
const db = { transaction: vi.fn().mockRejectedValue(failure) } as unknown as Db;
|
||||
let release!: () => void;
|
||||
const original = new Promise<void>(resolve => { release = resolve; });
|
||||
state.drainMaintenance.mockReturnValueOnce(original);
|
||||
let finished = false;
|
||||
const operation = reconcileRetainedNativeSessionCleanup(db, { companyId: "company", runId: "run" })
|
||||
.catch(error => error).finally(() => { finished = true; });
|
||||
try {
|
||||
await vi.waitFor(() => expect(state.drainMaintenance).toHaveBeenCalledTimes(1));
|
||||
expect(finished).toBe(false);
|
||||
release();
|
||||
expect(await operation).toBe(failure);
|
||||
} finally { release(); await operation; }
|
||||
});
|
||||
|
||||
it.each([
|
||||
Object.assign(new Error("database disconnected"), { code: "CONNECTION_CLOSED" }),
|
||||
new Error("query failed", { cause: Object.assign(new Error("row locked"), { code: "55P03" }) }),
|
||||
new Error("commit acknowledgement unavailable"),
|
||||
])("preserves a pre-claim database failure for recovery: %s", async failure => {
|
||||
const db = { transaction: vi.fn().mockRejectedValue(failure) } as unknown as Db;
|
||||
await expect(reconcileRetainedNativeSessionCleanup(db, { companyId: "company", runId: "run" }))
|
||||
.rejects.toBe(failure);
|
||||
expect(state.cleanup).not.toHaveBeenCalled();
|
||||
expect(state.retireCleanup).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([
|
||||
"settled",
|
||||
"provider_transport_failed",
|
||||
@@ -3953,11 +3987,20 @@ describe("retained native cleanup activation", () => {
|
||||
"legacy_activation",
|
||||
"released_identity",
|
||||
"released_identity_unproven",
|
||||
"invalid_execution_input",
|
||||
"invalid_state_json",
|
||||
"active_environment",
|
||||
"active_finalization_lease",
|
||||
"active_provider_process",
|
||||
])("preserves exact original evidence for %s", async (mode) => {
|
||||
const directory = await mkdtemp(
|
||||
join(tmpdir(), "paperclip-maintenance-activation-"),
|
||||
);
|
||||
let providerHomeDatabase: DatabaseSync | undefined;
|
||||
let releaseMaintenance!: () => void;
|
||||
const maintenance = new Promise<void>(resolve => { releaseMaintenance = resolve; });
|
||||
let unsubscribeMaintenance: (() => void) | undefined;
|
||||
const maintenanceReleased = vi.fn();
|
||||
const preservedHomeFiles = [
|
||||
"sessions/rollout-exact-thread.jsonl",
|
||||
"state_5.sqlite",
|
||||
@@ -4042,7 +4085,7 @@ describe("retained native cleanup activation", () => {
|
||||
payload: {
|
||||
providerSessionId: "exact-thread",
|
||||
providerAccountSessionId,
|
||||
processId: 99_999_998,
|
||||
processId: mode === "active_provider_process" ? process.pid : 99_999_998,
|
||||
},
|
||||
};
|
||||
const normalizedIdentity = {
|
||||
@@ -4223,13 +4266,14 @@ describe("retained native cleanup activation", () => {
|
||||
resultId: "result",
|
||||
assessmentId: "assessment",
|
||||
decisionId: "decision",
|
||||
leaseOwner: null,
|
||||
leaseExpiresAt: null,
|
||||
leaseOwner: mode === "active_finalization_lease" ? "another-owner" : null,
|
||||
leaseExpiresAt: mode === "active_finalization_lease" ? new Date(Date.now() + 60_000) : null,
|
||||
nextAttemptAt: null,
|
||||
recoveryHistory: [],
|
||||
};
|
||||
let transactionOpen = false;
|
||||
let stopReceiptRead = false;
|
||||
let environmentReleased = false;
|
||||
let releaseCommit!: () => void;
|
||||
const commitGate = new Promise<void>((release) => {
|
||||
releaseCommit = release;
|
||||
@@ -4238,6 +4282,7 @@ describe("retained native cleanup activation", () => {
|
||||
select: () => ({
|
||||
from: (table: unknown) => {
|
||||
const rows =
|
||||
table === environmentLeases && mode === "active_environment" && !environmentReleased ? [{ id: "active-environment" }] :
|
||||
table === heartbeatRuns
|
||||
? [run]
|
||||
: table === nativeRunFinalizations
|
||||
@@ -4679,6 +4724,8 @@ describe("retained native cleanup activation", () => {
|
||||
identity: { ...identity, runId: "foreign-run" },
|
||||
}),
|
||||
);
|
||||
if (mode === "invalid_execution_input") run.runnerProfileJson.nativeExecutionInput = {} as typeof execution;
|
||||
if (mode === "invalid_state_json") await writeFile(join(quarantine, source[0][0]), "{");
|
||||
const original = await Promise.all(
|
||||
source.map(([file]) => readFile(join(quarantine, file), "utf8")),
|
||||
);
|
||||
@@ -4966,6 +5013,10 @@ describe("retained native cleanup activation", () => {
|
||||
).toBe("settled");
|
||||
return 1;
|
||||
});
|
||||
if (mode === "legacy_busy_copy") {
|
||||
state.drainMaintenance.mockReturnValueOnce(maintenance);
|
||||
unsubscribeMaintenance = onRetainedNativeSessionCleanupRelease(db, maintenanceReleased);
|
||||
}
|
||||
const pendingOutcome = reconcileRetainedNativeSessionCleanup(
|
||||
db as unknown as Db,
|
||||
{ companyId: run.companyId, runId: run.id },
|
||||
@@ -5013,7 +5064,25 @@ describe("retained native cleanup activation", () => {
|
||||
).rejects.toThrow("native_session_supervisor_busy");
|
||||
releaseCommit();
|
||||
}
|
||||
if (mode === "canonical_claim_commit_failure") {
|
||||
await expect(pendingOutcome).rejects.toThrow("injected source claim commit failure");
|
||||
expect(state.cleanup).not.toHaveBeenCalled();
|
||||
expect(state.retireCleanup).not.toHaveBeenCalled();
|
||||
expect(coordinator.leaseOwner).toBeNull();
|
||||
expect(coordinator.recoveryHistory).toEqual([]);
|
||||
expect(await Promise.all(source.map(([file]) => readFile(join(root, file), "utf8")))).toEqual(original);
|
||||
return;
|
||||
}
|
||||
if (mode === "legacy_busy_copy") {
|
||||
await vi.waitFor(() => expect(state.drainMaintenance).toHaveBeenCalledTimes(1));
|
||||
expect(maintenanceReleased).not.toHaveBeenCalled();
|
||||
releaseMaintenance();
|
||||
}
|
||||
const outcome = await pendingOutcome;
|
||||
if (mode === "legacy_busy_copy") {
|
||||
expect(maintenanceReleased).toHaveBeenCalledExactlyOnceWith({ companyId: run.companyId, runId: run.id });
|
||||
unsubscribeMaintenance?.();
|
||||
}
|
||||
if (canonicalSource) {
|
||||
const prepared = (
|
||||
coordinator.recoveryHistory as Array<Record<string, unknown>>
|
||||
@@ -5049,6 +5118,14 @@ describe("retained native cleanup activation", () => {
|
||||
expect(cleanupEnvironment).not.toHaveProperty("OPENAI_API_KEY");
|
||||
expect(cleanupEnvironment).not.toHaveProperty("CODEX_API_KEY");
|
||||
}
|
||||
const waitingFor = mode.endsWith("live_owner") || mode === "active_provider_process" ? "process_exit"
|
||||
: mode === "legacy_busy_copy" ? "maintenance_settlement"
|
||||
: mode === "active_environment" ? "environment_release"
|
||||
: mode === "active_finalization_lease" ? "lease_expiry" : null;
|
||||
if (waitingFor) {
|
||||
expect(outcome).toMatchObject({ status: "deferred", waitFor: waitingFor, runId: run.id });
|
||||
if (waitingFor === "lease_expiry") expect(outcome).toHaveProperty("retryAt", coordinator.leaseExpiresAt);
|
||||
}
|
||||
const ineligible = [
|
||||
"canonical_live_owner",
|
||||
"canonical_foreign_owner",
|
||||
@@ -5078,6 +5155,8 @@ describe("retained native cleanup activation", () => {
|
||||
"foreign_semantic_scope",
|
||||
"wrong_provider_account",
|
||||
"released_identity_unproven",
|
||||
"invalid_execution_input",
|
||||
"invalid_state_json",
|
||||
].includes(mode);
|
||||
const succeeds = [
|
||||
"digest_only_command",
|
||||
@@ -5098,7 +5177,7 @@ describe("retained native cleanup activation", () => {
|
||||
"released_identity",
|
||||
].includes(mode);
|
||||
expect(outcome.status).toBe(
|
||||
succeeds
|
||||
waitingFor ? "deferred" : succeeds
|
||||
? "settled"
|
||||
: ineligible
|
||||
? "not_eligible"
|
||||
@@ -5142,7 +5221,7 @@ describe("retained native cleanup activation", () => {
|
||||
"home_selected_reverted_rollout",
|
||||
].includes(mode);
|
||||
expect(state.cleanup).toHaveBeenCalledTimes(
|
||||
ineligible || deniedBeforeLaunch ? 0 : 1,
|
||||
ineligible || waitingFor || deniedBeforeLaunch ? 0 : 1,
|
||||
);
|
||||
expect(state.retireCleanup).toHaveBeenCalledTimes(succeeds ? 1 : 0);
|
||||
expect(coordinator.phase).toBe("committed");
|
||||
@@ -5347,7 +5426,25 @@ describe("retained native cleanup activation", () => {
|
||||
).toEqual(["started", "staged", "launch_intent", "operator_required"]);
|
||||
await expect(access(root)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
}
|
||||
if (waitingFor && mode !== "active_provider_process") {
|
||||
|
||||
|
||||
if (waitingFor === "process_exit") {
|
||||
run.processPid = 99_999_999;
|
||||
run.processGroupId = 99_999_999;
|
||||
}
|
||||
if (waitingFor === "maintenance_settlement") state.maintenanceIdle.mockReturnValue(true);
|
||||
if (waitingFor === "environment_release") environmentReleased = true;
|
||||
if (waitingFor === "lease_expiry") coordinator.leaseExpiresAt = new Date(Date.now() - 1000);
|
||||
await expect(reconcileRetainedNativeSessionCleanup(db as unknown as Db, {
|
||||
companyId: run.companyId, runId: run.id,
|
||||
})).resolves.toEqual({ status: "settled", runId: run.id });
|
||||
expect(state.cleanup).toHaveBeenCalledTimes(1);
|
||||
expect(state.retireCleanup).toHaveBeenCalledTimes(1);
|
||||
}
|
||||
} finally {
|
||||
releaseMaintenance();
|
||||
unsubscribeMaintenance?.();
|
||||
providerHomeDatabase?.close();
|
||||
proofSpy.mockRestore();
|
||||
state.maintenanceIdle.mockReset().mockReturnValue(true);
|
||||
@@ -6297,11 +6394,16 @@ describe("native startup cancellation fence", () => {
|
||||
}
|
||||
throw new Error("provider should not have been submitted");
|
||||
});
|
||||
await expect(executePaperclipNativeSession({
|
||||
db: leaseDb(execution, {}, resultJson), execution, runnerInstanceId: "startup-stop",
|
||||
})).rejects.toThrow("native_cancellation_pending_recovery");
|
||||
expect(cancel).toHaveBeenCalledOnce();
|
||||
expect(submit).not.toHaveBeenCalled();
|
||||
const db = leaseDb(execution, {}, resultJson), released = vi.fn();
|
||||
const unsubscribe = onRetainedNativeSessionCleanupRelease(db, released);
|
||||
try {
|
||||
await expect(executePaperclipNativeSession({
|
||||
db, execution, runnerInstanceId: "startup-stop",
|
||||
})).rejects.toThrow("native_cancellation_pending_recovery");
|
||||
expect(cancel).toHaveBeenCalledOnce();
|
||||
expect(submit).not.toHaveBeenCalled();
|
||||
expect(released).toHaveBeenCalledWith({ companyId: execution.binding.companyId, agentId: execution.binding.agentId });
|
||||
} finally { unsubscribe(); }
|
||||
});
|
||||
});
|
||||
|
||||
@@ -9514,7 +9616,9 @@ describe("native warm session supervision", () => {
|
||||
});
|
||||
|
||||
it("reuses one session across distinct governed runs and closes it after idle expiry", async () => {
|
||||
const close = vi.fn(async () => undefined);
|
||||
let releaseClose!: () => void;
|
||||
const closeBarrier = new Promise<void>(resolve => { releaseClose = resolve; });
|
||||
const close = vi.fn(() => closeBarrier);
|
||||
const sharedSession = { close };
|
||||
const base = {
|
||||
...execution,
|
||||
@@ -9563,24 +9667,34 @@ describe("native warm session supervision", () => {
|
||||
return result;
|
||||
});
|
||||
|
||||
await executePaperclipNativeSession({
|
||||
db: leaseDb(base),
|
||||
execution: base,
|
||||
runnerInstanceId: "runner",
|
||||
});
|
||||
await executePaperclipNativeSession({
|
||||
db: leaseDb(second),
|
||||
execution: second,
|
||||
runnerInstanceId: "runner",
|
||||
});
|
||||
expect(close).not.toHaveBeenCalled();
|
||||
await vi.waitFor(
|
||||
() =>
|
||||
expect(close).toHaveBeenCalledWith({
|
||||
reason: "warm native session idle timeout",
|
||||
}),
|
||||
{ timeout: 2_000 },
|
||||
);
|
||||
const db = leaseDb(base), released = vi.fn();
|
||||
const unsubscribe = onRetainedNativeSessionCleanupRelease(db, released);
|
||||
try {
|
||||
await executePaperclipNativeSession({
|
||||
db,
|
||||
execution: base,
|
||||
runnerInstanceId: "runner",
|
||||
});
|
||||
await executePaperclipNativeSession({
|
||||
db: leaseDb(second),
|
||||
execution: second,
|
||||
runnerInstanceId: "runner",
|
||||
});
|
||||
expect(close).not.toHaveBeenCalled();
|
||||
released.mockClear();
|
||||
await vi.waitFor(
|
||||
() =>
|
||||
expect(close).toHaveBeenCalledWith({
|
||||
reason: "warm native session idle timeout",
|
||||
}),
|
||||
{ timeout: 2_000 },
|
||||
);
|
||||
expect(released).not.toHaveBeenCalled();
|
||||
releaseClose();
|
||||
await vi.waitFor(() => expect(released).toHaveBeenCalledExactlyOnceWith({
|
||||
companyId: base.binding.companyId, agentId: base.binding.agentId,
|
||||
}));
|
||||
} finally { releaseClose(); unsubscribe(); }
|
||||
});
|
||||
|
||||
it("does not offer a quarantined warm session to the next run", async () => {
|
||||
@@ -12078,9 +12192,12 @@ describe("runnerd provider runtime wiring", () => {
|
||||
});
|
||||
state.createBackend.mockClear();
|
||||
state.createTransport.mockClear();
|
||||
const db = leaseDb(execution);
|
||||
const released = vi.fn();
|
||||
const unsubscribe = onRetainedNativeSessionCleanupRelease(db, released);
|
||||
let settled = false;
|
||||
const preparation = createRunnerdBackend({
|
||||
db: leaseDb(execution), execution, runnerInstanceId: "runner-overlap",
|
||||
db, execution, runnerInstanceId: "runner-overlap",
|
||||
runnerEnvironment, prepareRunnerEnvironment,
|
||||
runnerExecutionTarget: {
|
||||
kind: "remote", transport: "sandbox", providerKey: "daytona",
|
||||
@@ -12092,15 +12209,18 @@ describe("runnerd provider runtime wiring", () => {
|
||||
await assignedReady;
|
||||
expect(prepareRunnerEnvironment).toHaveBeenCalledOnce();
|
||||
expect(settled).toBe(false);
|
||||
expect(released).not.toHaveBeenCalled();
|
||||
expect(state.createBackend).not.toHaveBeenCalled();
|
||||
if (failure === "assigned") releaseAssigned();
|
||||
else releaseEnvironment();
|
||||
await new Promise<void>(resolve => setImmediate(resolve));
|
||||
expect(settled).toBe(false);
|
||||
expect(released).not.toHaveBeenCalled();
|
||||
expect(state.createBackend).not.toHaveBeenCalled();
|
||||
releaseEnvironment();
|
||||
releaseAssigned();
|
||||
const result = await preparation;
|
||||
expect(released).toHaveBeenCalledExactlyOnceWith({ companyId: execution.binding.companyId, agentId: execution.binding.agentId });
|
||||
if (failure !== "success") {
|
||||
expect(result).toMatchObject({ error: { message: failure === "assigned" ? "assigned failed" : "environment failed" } });
|
||||
expect(state.createBackend).not.toHaveBeenCalled();
|
||||
@@ -12116,6 +12236,7 @@ describe("runnerd provider runtime wiring", () => {
|
||||
releaseEnvironment();
|
||||
releaseAssigned();
|
||||
await preparation;
|
||||
unsubscribe();
|
||||
}
|
||||
});
|
||||
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { releaseRetainedNativeSessionCleanup } from "../retained-native-session-cleanup-notifications.js";
|
||||
import type { NativeCleanupOutcome } from "./retained-native-session-cleanup.js";
|
||||
import { notifyNativeStatusReassessment } from "../native-status-reassessment-notifications.js";
|
||||
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
||||
import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js";
|
||||
@@ -132,6 +134,7 @@ import {
|
||||
resolveSourceCodexHome,
|
||||
readRunnerdArtifactBinding,
|
||||
retainedRunnerdMaintenanceIsIdle,
|
||||
drainRetainedRunnerdMaintenanceOperations,
|
||||
settleRetainedRunnerdSession,
|
||||
validatePrpEvent,
|
||||
validatePrpStructuredRunResult,
|
||||
@@ -546,6 +549,7 @@ export type NativeInstructionWorkingCopy = {
|
||||
};
|
||||
|
||||
type WarmNativeSession = {
|
||||
onReleased: () => void;
|
||||
pluginTaskId?: string;
|
||||
observabilityCompatibilityKey?: string;
|
||||
runId: string;
|
||||
@@ -593,6 +597,12 @@ async function closeWarmNativeSession(entry: WarmNativeSession, reason: string,
|
||||
}
|
||||
|
||||
const warmNativeSessions = new Map<string, WarmNativeSession>();
|
||||
function forgetWarmNativeSession(scope: string, entry: WarmNativeSession) {
|
||||
if (warmNativeSessions.get(scope) !== entry) return;
|
||||
warmNativeSessions.delete(scope);
|
||||
entry.onReleased();
|
||||
}
|
||||
|
||||
const pluginTaskSessionOwners = new WeakMap<NativeSession, ReturnType<typeof createPluginRunnerTaskOwner>>();
|
||||
// Closing a remote owner saves its checkpoint asynchronously. A new turn must
|
||||
// not inspect or quarantine that owner's state until the save has finished.
|
||||
@@ -630,7 +640,7 @@ export async function claimWarmNativeInstructionCopy(input: {
|
||||
entry.closeOnReleaseReason = reason;
|
||||
try {
|
||||
await closeWarmNativeSession(entry, reason);
|
||||
if (warmNativeSessions.get(scope) === entry) warmNativeSessions.delete(scope);
|
||||
forgetWarmNativeSession(scope, entry);
|
||||
} finally { entry.busy = false; }
|
||||
};
|
||||
try {
|
||||
@@ -682,7 +692,7 @@ export async function reserveWarmNativeInstructionDirectory(input: {
|
||||
entry.closeOnReleaseReason = reason;
|
||||
try {
|
||||
await closeWarmNativeSession(entry, reason);
|
||||
if (warmNativeSessions.get(id) === entry) warmNativeSessions.delete(id);
|
||||
forgetWarmNativeSession(id, entry);
|
||||
} finally {
|
||||
// Failed containment remains registered and cannot be adopted. The next
|
||||
// admission retries retirement instead of launching over a live owner.
|
||||
@@ -765,10 +775,10 @@ async function closeIdleWarmNativeSessions(input: {
|
||||
closingWarmNativeSessions.set(sessionId, closing);
|
||||
try {
|
||||
await closing;
|
||||
if (warmNativeSessions.get(sessionId) === entry) warmNativeSessions.delete(sessionId);
|
||||
forgetWarmNativeSession(sessionId, entry);
|
||||
closed += 1;
|
||||
} catch {
|
||||
if (!entry.instructionWorkingCopy?.runId) warmNativeSessions.delete(sessionId);
|
||||
if (!entry.instructionWorkingCopy?.runId) forgetWarmNativeSession(sessionId, entry);
|
||||
failed += 1;
|
||||
} finally {
|
||||
entry.busy = false;
|
||||
@@ -1894,7 +1904,7 @@ async function retireSupersededWarmNativeSessions(
|
||||
entry.closeOnReleaseReason = "warm native session identity changed";
|
||||
try {
|
||||
await closeWarmNativeSession(entry, entry.closeOnReleaseReason);
|
||||
if (warmNativeSessions.get(scope) === entry) warmNativeSessions.delete(scope);
|
||||
forgetWarmNativeSession(scope, entry);
|
||||
} finally {
|
||||
// A failed close retains the owner and prevents launch. A later attempt
|
||||
// must retry retirement, never adopt this partially closed transport.
|
||||
@@ -3043,14 +3053,22 @@ export async function verifyStoppedNativeSessionForReplacement(
|
||||
* inspection, not retried from an older snapshot with unknown process owners. */
|
||||
export async function reconcileRetainedNativeSessionCleanup(
|
||||
db: Db,
|
||||
input: {
|
||||
companyId: string;
|
||||
runId: string;
|
||||
},
|
||||
): Promise<{
|
||||
status: "settled" | "not_eligible" | "operator_required";
|
||||
runId: string;
|
||||
}> {
|
||||
input: { companyId: string; runId: string },
|
||||
): Promise<NativeCleanupOutcome> {
|
||||
|
||||
|
||||
const outcome = await inspectRetainedNativeSessionCleanup(db, input)
|
||||
.finally(() => drainRetainedRunnerdMaintenanceOperations());
|
||||
if (outcome.status === "deferred" && outcome.waitFor === "maintenance_settlement") {
|
||||
releaseRetainedNativeSessionCleanup(db, input);
|
||||
}
|
||||
return outcome;
|
||||
}
|
||||
|
||||
async function inspectRetainedNativeSessionCleanup(
|
||||
db: Db,
|
||||
input: { companyId: string; runId: string },
|
||||
): Promise<NativeCleanupOutcome> {
|
||||
const denied = () => new Error("native_cleanup_maintenance_unproven");
|
||||
const leaseOwner = `native-cleanup:${randomUUID()}`;
|
||||
let reservedScope: string | null = null;
|
||||
@@ -3062,7 +3080,7 @@ export async function reconcileRetainedNativeSessionCleanup(
|
||||
executingRunnerdSessionScopes.delete(reservedScope);
|
||||
}
|
||||
};
|
||||
let claim: {
|
||||
type CleanupClaim = {
|
||||
execution: NativeExecutionInput;
|
||||
run: typeof heartbeatRuns.$inferSelect;
|
||||
quarantine: string;
|
||||
@@ -3091,7 +3109,12 @@ export async function reconcileRetainedNativeSessionCleanup(
|
||||
itemId: string;
|
||||
};
|
||||
history: Array<Record<string, unknown>>;
|
||||
} | null = null;
|
||||
};
|
||||
type DeferredCleanup = Extract<NativeCleanupOutcome, { status: "deferred" }>;
|
||||
const defer = (waitFor: Exclude<DeferredCleanup["waitFor"], "lease_expiry">): DeferredCleanup => ({
|
||||
runId: input.runId, status: "deferred", waitFor,
|
||||
});
|
||||
let claim: CleanupClaim | DeferredCleanup | null = null;
|
||||
try {
|
||||
claim = await db.transaction(async (tx) => {
|
||||
const row = await tx
|
||||
@@ -3117,10 +3140,12 @@ export async function reconcileRetainedNativeSessionCleanup(
|
||||
!run.runnerInstanceId ||
|
||||
!["succeeded", "failed"].includes(run.status) ||
|
||||
!run.finishedAt ||
|
||||
!cleanupProcessAbsent(run.processPid) ||
|
||||
process.platform === "win32" ||
|
||||
!Number.isSafeInteger(run.processPid) || Number(run.processPid) <= 0 ||
|
||||
run.processGroupId !== run.processPid
|
||||
)
|
||||
return null;
|
||||
if (!cleanupProcessAbsent(run.processPid)) return defer("process_exit");
|
||||
const execution = parseNativeExecutionInput(
|
||||
record(run.runnerProfileJson).nativeExecutionInput,
|
||||
);
|
||||
@@ -3149,7 +3174,7 @@ export async function reconcileRetainedNativeSessionCleanup(
|
||||
initializingSessionToolAuthorities.has(scope) ||
|
||||
warmNativeSessions.has(scope)
|
||||
)
|
||||
return null;
|
||||
return defer("session_release");
|
||||
executingRunnerdSessionScopes.set(scope, leaseOwner);
|
||||
reservedScope = scope;
|
||||
const coordinator = await tx
|
||||
@@ -3172,14 +3197,14 @@ export async function reconcileRetainedNativeSessionCleanup(
|
||||
!coordinator.assessmentId ||
|
||||
!coordinator.decisionId ||
|
||||
coordinator.nextAttemptAt ||
|
||||
(coordinator.leaseOwner &&
|
||||
coordinator.leaseExpiresAt &&
|
||||
coordinator.leaseExpiresAt > new Date()) ||
|
||||
coordinator.recoveryHistory.some(
|
||||
(event) => event.kind === "native_cleanup_runner_epoch",
|
||||
)
|
||||
)
|
||||
return null;
|
||||
if (coordinator.leaseOwner && coordinator.leaseExpiresAt && coordinator.leaseExpiresAt > new Date()) {
|
||||
return { runId: input.runId, status: "deferred" as const, waitFor: "lease_expiry" as const, retryAt: coordinator.leaseExpiresAt };
|
||||
}
|
||||
const result = await tx
|
||||
.select()
|
||||
.from(nativeRunResults)
|
||||
@@ -3206,7 +3231,7 @@ export async function reconcileRetainedNativeSessionCleanup(
|
||||
),
|
||||
)
|
||||
.limit(1);
|
||||
if (environment.length) return null;
|
||||
if (environment.length) return defer("environment_release");
|
||||
const root = scopedRunnerdStateRoot(execution);
|
||||
const maintenanceHistory = coordinator.recoveryHistory.filter(
|
||||
(event) => event.kind === "native_cleanup_maintenance",
|
||||
@@ -3225,7 +3250,7 @@ export async function reconcileRetainedNativeSessionCleanup(
|
||||
let emptyRoot: ReturnType<typeof cleanupCanonicalVacancy> = null;
|
||||
let sourceDirectory: string;
|
||||
let quarantine: string;
|
||||
let sourceArchive: NonNullable<typeof claim>["sourceArchive"] = null;
|
||||
let sourceArchive: CleanupClaim["sourceArchive"] = null;
|
||||
const rootExists = !!lstatSync(root, { throwIfNoEntry: false });
|
||||
if (rootExists && !isSafeNativeStateDirectory(root)) return null;
|
||||
if (archiveHistory.length) {
|
||||
@@ -3352,7 +3377,7 @@ export async function reconcileRetainedNativeSessionCleanup(
|
||||
providerEvent.sourceKind !== "runner" ||
|
||||
providerEvent.turnId !== identity.turnId ||
|
||||
providerEvent.itemId !== identity.itemId ||
|
||||
!cleanupProcessAbsent(provider.processId) ||
|
||||
!Number.isSafeInteger(provider.processId) || Number(provider.processId) <= 0 ||
|
||||
typeof provider.providerSessionId !== "string" ||
|
||||
source.provider.threadId !== provider.providerSessionId ||
|
||||
providerEvent.runId !== run.id ||
|
||||
@@ -3361,6 +3386,7 @@ export async function reconcileRetainedNativeSessionCleanup(
|
||||
typeof providerEvent.sourceEventId !== "string"
|
||||
)
|
||||
return null;
|
||||
if (!cleanupProcessAbsent(provider.processId)) return defer("process_exit");
|
||||
const persisted = await tx
|
||||
.select()
|
||||
.from(heartbeatRunEvents)
|
||||
@@ -3449,13 +3475,8 @@ export async function reconcileRetainedNativeSessionCleanup(
|
||||
);
|
||||
if (attemptedNames.length !== 1) return null;
|
||||
const directory = resolve(runnerdStateBase(), attemptedNames[0]!);
|
||||
if (
|
||||
!retainedRunnerdMaintenanceIsIdle(directory) ||
|
||||
lstatSync(resolve(directory, CLEANUP_ACTIVATION_FILE), {
|
||||
throwIfNoEntry: false,
|
||||
})
|
||||
)
|
||||
return null;
|
||||
if (lstatSync(resolve(directory, CLEANUP_ACTIVATION_FILE), { throwIfNoEntry: false })) return null;
|
||||
if (!retainedRunnerdMaintenanceIsIdle(directory)) return defer("maintenance_settlement");
|
||||
const attempted = cleanupStateSnapshot(directory);
|
||||
const receipts = await tx
|
||||
.select()
|
||||
@@ -3557,14 +3578,27 @@ export async function reconcileRetainedNativeSessionCleanup(
|
||||
history,
|
||||
};
|
||||
});
|
||||
} catch {
|
||||
} catch (error) {
|
||||
releaseScope();
|
||||
return { status: "not_eligible", runId: input.runId };
|
||||
|
||||
|
||||
|
||||
const code = record(error).code;
|
||||
if (error instanceof SyntaxError || code === "native_execution_input_invalid"
|
||||
|| code === "ENOENT" || code === "ENOTDIR" || code === "ELOOP"
|
||||
|| (error instanceof Error && ["native_cleanup_maintenance_unproven", "native_state_file_changed"].includes(error.message))) {
|
||||
return { status: "not_eligible", runId: input.runId };
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
if (!claim) {
|
||||
releaseScope();
|
||||
return { status: "not_eligible", runId: input.runId };
|
||||
}
|
||||
if ("status" in claim) {
|
||||
releaseScope();
|
||||
return claim;
|
||||
}
|
||||
const owned = claim;
|
||||
let stagingDirectory: string | null = null;
|
||||
let maintenanceStarted = !owned.sourceArchive;
|
||||
@@ -6527,9 +6561,9 @@ async function releaseWarmNativeSession(
|
||||
entry.busy = true;
|
||||
try {
|
||||
await closeWarmNativeSession(entry, entry.closeOnReleaseReason);
|
||||
if (warmNativeSessions.get(sessionId) === entry) warmNativeSessions.delete(sessionId);
|
||||
forgetWarmNativeSession(sessionId, entry);
|
||||
} catch (error) {
|
||||
if (!entry.instructionWorkingCopy?.runId) warmNativeSessions.delete(sessionId);
|
||||
if (!entry.instructionWorkingCopy?.runId) forgetWarmNativeSession(sessionId, entry);
|
||||
if (requireCleanClose) throw error;
|
||||
} finally { entry.busy = false; }
|
||||
return;
|
||||
@@ -6555,8 +6589,8 @@ function armWarmNativeSessionIdleTimer(
|
||||
current.busy = true;
|
||||
current.closeOnReleaseReason = "warm native session idle timeout";
|
||||
const closing = closeWarmNativeSession(current, current.closeOnReleaseReason)
|
||||
.then(() => { if (warmNativeSessions.get(sessionId) === current) warmNativeSessions.delete(sessionId); })
|
||||
.catch(() => { if (!current.instructionWorkingCopy?.runId && warmNativeSessions.get(sessionId) === current) warmNativeSessions.delete(sessionId); })
|
||||
.then(() => { forgetWarmNativeSession(sessionId, current); })
|
||||
.catch(() => { if (!current.instructionWorkingCopy?.runId) forgetWarmNativeSession(sessionId, current); })
|
||||
.finally(() => {
|
||||
current.busy = false;
|
||||
if (closingWarmNativeSessions.get(sessionId) === closing) closingWarmNativeSessions.delete(sessionId);
|
||||
@@ -8003,6 +8037,9 @@ export async function executePaperclipNativeSession(input: {
|
||||
})
|
||||
.catch(() => undefined);
|
||||
}
|
||||
releaseRetainedNativeSessionCleanup(input.db, {
|
||||
companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -8921,7 +8958,7 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
entry.closeOnReleaseReason = "warm native session configuration changed";
|
||||
try {
|
||||
await closeWarmNativeSession(entry, entry.closeOnReleaseReason, input.execution.binding.runId);
|
||||
warmNativeSessions.delete(warmSessionId);
|
||||
forgetWarmNativeSession(warmSessionId, entry);
|
||||
} finally { entry.busy = false; }
|
||||
if (preparedCopy) throw new Error("native_instruction_preparation_configuration_changed");
|
||||
persistedWarmSession = loadWarmNativeCheckpoint(
|
||||
@@ -8938,7 +8975,7 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
entry.closeOnReleaseReason = "warm instruction copy changed during preparation";
|
||||
try {
|
||||
await closeWarmNativeSession(entry, entry.closeOnReleaseReason);
|
||||
warmNativeSessions.delete(warmSessionId);
|
||||
forgetWarmNativeSession(warmSessionId, entry);
|
||||
} finally { entry.busy = false; }
|
||||
throw new Error("native_instruction_preparation_copy_changed");
|
||||
}
|
||||
@@ -9345,6 +9382,9 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
existing.issueId = input.execution.binding.issueId;
|
||||
} else
|
||||
warmNativeSessions.set(warmSessionId, {
|
||||
onReleased: () => releaseRetainedNativeSessionCleanup(input.db, {
|
||||
companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId,
|
||||
}),
|
||||
pluginTaskId: usesPluginTask && input.runnerExecutionTarget?.kind === "remote" && input.runnerExecutionTarget.transport === "sandbox"
|
||||
? input.runnerExecutionTarget.sandboxLeaseAcquisition?.providerLeaseId : undefined,
|
||||
runId: input.execution.binding.runId,
|
||||
@@ -9392,7 +9432,7 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
if (existing.idleTimer !== null) {
|
||||
clearTimeout(existing.idleTimer);
|
||||
}
|
||||
warmNativeSessions.delete(warmSessionId);
|
||||
forgetWarmNativeSession(warmSessionId, existing);
|
||||
}
|
||||
}
|
||||
if (session) {
|
||||
@@ -11852,6 +11892,9 @@ export async function createRunnerdBackend(input: {
|
||||
);
|
||||
} finally {
|
||||
initializingSessionToolAuthorities.delete(sessionScopeId);
|
||||
releaseRetainedNativeSessionCleanup(input.db, {
|
||||
companyId: input.execution.binding.companyId, agentId: input.execution.binding.agentId,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyRetainedNativeSessionCleanup } from "../retained-native-session-cleanup-notifications.js";
|
||||
import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq, inArray, ne, sql } from "drizzle-orm";
|
||||
@@ -107,20 +108,23 @@ export async function settleNativeWorkspaceExportResume(db: Db, lease: Lease, op
|
||||
const receipt = remoteTerminationReceipt(lease, options.receipt);
|
||||
const stopped = receipt?.state === "stopped";
|
||||
const now = new Date();
|
||||
const [row] = await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({
|
||||
status: stopped ? options.status ?? "released" : "pending_cleanup", cleanupStatus: stopped ? "success" : "failed",
|
||||
releasedAt: now, lastUsedAt: now, updatedAt: now,
|
||||
metadata: sql`(${environmentLeases.metadata} - 'remoteExecutionTermination'
|
||||
${stopped ? sql`- 'nativeWorkspaceExportResume'` : sql``}) || ${JSON.stringify({
|
||||
...(stopped ? { remoteExecutionTermination: receipt } : {}),
|
||||
pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0,
|
||||
})}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId),
|
||||
eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.provider, lease.provider!),
|
||||
eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), eq(environmentLeases.status, "pending_cleanup"),
|
||||
sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.attemptId}`,
|
||||
sql`${environmentLeases.metadata}->'nativeWorkspaceExportResume'->>'requestId' = ${marker.requestId}`,
|
||||
sql`${environmentLeases.metadata}->>'pluginId' = ${marker.pluginId}`,
|
||||
)).returning());
|
||||
const [row] = await writeEnvironmentCleanup(db, async tx => {
|
||||
if (stopped) await notifyRetainedNativeSessionCleanup(tx, { companyId: lease.companyId, runId: lease.heartbeatRunId! });
|
||||
return tx.update(environmentLeases).set({
|
||||
status: stopped ? options.status ?? "released" : "pending_cleanup", cleanupStatus: stopped ? "success" : "failed",
|
||||
releasedAt: now, lastUsedAt: now, updatedAt: now,
|
||||
metadata: sql`(${environmentLeases.metadata} - 'remoteExecutionTermination'
|
||||
${stopped ? sql`- 'nativeWorkspaceExportResume'` : sql``}) || ${JSON.stringify({
|
||||
...(stopped ? { remoteExecutionTermination: receipt } : {}),
|
||||
pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0,
|
||||
})}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId),
|
||||
eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.provider, lease.provider!),
|
||||
eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), eq(environmentLeases.status, "pending_cleanup"),
|
||||
sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.attemptId}`,
|
||||
sql`${environmentLeases.metadata}->'nativeWorkspaceExportResume'->>'requestId' = ${marker.requestId}`,
|
||||
sql`${environmentLeases.metadata}->>'pluginId' = ${marker.pluginId}`,
|
||||
)).returning();
|
||||
});
|
||||
return row ?? null;
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
|
||||
import { onRetainedNativeSessionCleanupRelease, type RetainedNativeSessionCleanupScope } from "../retained-native-session-cleanup-notifications.js";
|
||||
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
|
||||
import { findRetainedNativeSessionCleanupCandidates, pendingRetainedNativeSessionCleanup, runRetainedNativeSessionCleanup,
|
||||
type RetainedNativeSessionCleanup } from "./retained-native-session-cleanup.js";
|
||||
|
||||
type Entry = { id: string; companyId: string; issueId: string; agentId: string; dueAt: number };
|
||||
|
||||
|
||||
const PROCESS_EXIT_PROBE_MS = 30_000;
|
||||
|
||||
export function retainedNativeSessionCleanupWork(db: Db, cleanup: RetainedNativeSessionCleanup) {
|
||||
async function load(scope?: RetainedNativeSessionCleanupScope, signal?: AbortSignal, canRun = () => true): Promise<Entry[]> {
|
||||
const entries: Entry[] = [];
|
||||
let after: string | undefined;
|
||||
while (!signal?.aborted && canRun()) {
|
||||
const rows = await findRetainedNativeSessionCleanupCandidates(db, { scope, after, limit: 100, includeLeased: true });
|
||||
for (const row of rows) entries.push({ id: row.runId, companyId: row.companyId, issueId: row.issueId, agentId: row.agentId,
|
||||
dueAt: row.leaseOwner ? row.leaseExpiresAt?.getTime() ?? 0 : 0 });
|
||||
if (rows.length < 100) return entries;
|
||||
after = rows[rows.length - 1].runId;
|
||||
}
|
||||
throw new Error("Retained native cleanup discovery stopped");
|
||||
}
|
||||
async function check(entry: Entry, signal?: AbortSignal): Promise<Entry | null> {
|
||||
if (signal?.aborted) throw new Error("Retained native cleanup stopped");
|
||||
let pending = pendingRetainedNativeSessionCleanup(db, entry.id);
|
||||
if (!pending) {
|
||||
const [current] = await findRetainedNativeSessionCleanupCandidates(db, {
|
||||
scope: { companyId: entry.companyId, runId: entry.id }, limit: 1, includeLeased: true,
|
||||
});
|
||||
if (!current) return null;
|
||||
if (current.leaseOwner && current.leaseExpiresAt && current.leaseExpiresAt.getTime() > Date.now()) {
|
||||
return { ...entry, dueAt: current.leaseExpiresAt.getTime() };
|
||||
}
|
||||
if (signal?.aborted) throw new Error("Retained native cleanup stopped");
|
||||
pending = runRetainedNativeSessionCleanup(db, { companyId: entry.companyId, runId: entry.id }, cleanup);
|
||||
}
|
||||
const result = await pending;
|
||||
if (result.status !== "deferred") return null;
|
||||
if (result.waitFor === "lease_expiry") return { ...entry, dueAt: result.retryAt.getTime() };
|
||||
if (result.waitFor === "process_exit") return { ...entry, dueAt: Date.now() + PROCESS_EXIT_PROBE_MS };
|
||||
|
||||
|
||||
return null;
|
||||
}
|
||||
function register(coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, canRun: () => boolean) {
|
||||
const worker = registerScopedDeadlineWork(coordinator, {
|
||||
queue: DELIVERY_QUEUES.retainedNativeSessionCleanup, batchSize: 1,
|
||||
decodeScope: (hint: string): RetainedNativeSessionCleanupScope | undefined => hint === "*" ? undefined : JSON.parse(hint),
|
||||
matches: (entry: Entry, scope?: RetainedNativeSessionCleanupScope) => !scope || entry.companyId === scope.companyId
|
||||
&& (!scope.runId || entry.id === scope.runId) && (!scope.issueId || entry.issueId === scope.issueId) && (!scope.agentId || entry.agentId === scope.agentId),
|
||||
load: (scope, signal) => load(scope, signal, canRun),
|
||||
check: (entry, signal) => {
|
||||
if (!canRun()) throw new Error("Retained native cleanup paused");
|
||||
return check(entry, signal);
|
||||
},
|
||||
}, canRun);
|
||||
const unsubscribe = onRetainedNativeSessionCleanupRelease(db, scope => worker.invalidate(JSON.stringify(scope)));
|
||||
return { ...worker, stop: unsubscribe };
|
||||
}
|
||||
return { load, check, register };
|
||||
}
|
||||
@@ -0,0 +1,200 @@
|
||||
import { and, asc, eq, gt, inArray, isNotNull, isNull, lte, or, sql } from "drizzle-orm";
|
||||
import { heartbeatRuns, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
|
||||
import { nativeRunnerOwnershipNotHeldCondition } from "./native-runner-ownership.js";
|
||||
|
||||
import type { RetainedNativeSessionCleanupScope } from "../retained-native-session-cleanup-notifications.js";
|
||||
export type { RetainedNativeSessionCleanupScope } from "../retained-native-session-cleanup-notifications.js";
|
||||
|
||||
|
||||
|
||||
|
||||
export function findRetainedNativeSessionCleanupCandidates(db: Db, options: {
|
||||
scope?: RetainedNativeSessionCleanupScope;
|
||||
after?: string;
|
||||
limit: number;
|
||||
includeLeased?: boolean;
|
||||
}) {
|
||||
return db
|
||||
.select({
|
||||
runId: heartbeatRuns.id,
|
||||
companyId: heartbeatRuns.companyId,
|
||||
issueId: nativeRunFinalizations.issueId,
|
||||
agentId: heartbeatRuns.agentId,
|
||||
leaseOwner: nativeRunFinalizations.leaseOwner,
|
||||
leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.innerJoin(
|
||||
nativeRunFinalizations,
|
||||
and(
|
||||
eq(nativeRunFinalizations.runId, heartbeatRuns.id),
|
||||
eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
|
||||
eq(nativeRunFinalizations.issueId, heartbeatRuns.nativeIssueId),
|
||||
),
|
||||
)
|
||||
.innerJoin(
|
||||
nativeRunResults,
|
||||
and(
|
||||
eq(nativeRunResults.id, nativeRunFinalizations.resultId),
|
||||
eq(nativeRunResults.runId, heartbeatRuns.id),
|
||||
eq(nativeRunResults.companyId, heartbeatRuns.companyId),
|
||||
eq(nativeRunResults.issueId, heartbeatRuns.nativeIssueId),
|
||||
),
|
||||
)
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRuns.runtimeMode, "native"),
|
||||
inArray(heartbeatRuns.status, ["succeeded", "failed"]),
|
||||
isNotNull(heartbeatRuns.finishedAt),
|
||||
nativeRunnerOwnershipNotHeldCondition(),
|
||||
eq(nativeRunFinalizations.phase, "committed"),
|
||||
eq(nativeRunResults.schemaStatus, "accepted"),
|
||||
isNotNull(nativeRunFinalizations.assessmentId),
|
||||
isNotNull(nativeRunFinalizations.decisionId),
|
||||
isNull(nativeRunFinalizations.nextAttemptAt),
|
||||
options.includeLeased ? undefined : or(
|
||||
isNull(nativeRunFinalizations.leaseOwner),
|
||||
isNull(nativeRunFinalizations.leaseExpiresAt),
|
||||
lte(nativeRunFinalizations.leaseExpiresAt, sql`now()`),
|
||||
),
|
||||
|
||||
|
||||
sql`coalesce(${heartbeatRuns.errorCode}, ${heartbeatRuns.resultJson}->'recoveredExecutionFailure'->>'errorCode') in ('adapter_failed', 'provider_transport_failed')`,
|
||||
sql`coalesce(${heartbeatRuns.error}, ${heartbeatRuns.resultJson}->'recoveredExecutionFailure'->>'error') = 'provider_transport_failed: runner did not durably suspend before checkpoint'`,
|
||||
sql`not (${nativeRunFinalizations.recoveryHistory} @> '[{"kind":"native_cleanup_runner_epoch"}]'::jsonb)`,
|
||||
sql`not (${nativeRunFinalizations.recoveryHistory} @> '[{"kind":"native_cleanup_source_archive","phase":"operator_required"}]'::jsonb)`,
|
||||
|
||||
|
||||
|
||||
|
||||
sql`(
|
||||
select coalesce(
|
||||
jsonb_array_length(history.entries) = 0 or (
|
||||
jsonb_array_length(history.entries) = 2
|
||||
and history.entries->0->>'version' = '1'
|
||||
and history.entries->1->>'version' = '1'
|
||||
and history.entries->0->>'phase' = 'started'
|
||||
and history.entries->1->>'phase' = 'operator_required'
|
||||
and history.entries->1->>'code' = 'native_cleanup_maintenance_unproven'
|
||||
and history.entries->0->>'requestId' like 'native-cleanup:%'
|
||||
and history.entries->0->>'requestId' = history.entries->1->>'requestId'
|
||||
and history.entries->0->>'sourceFingerprint' ~ '^[a-f0-9]{64}$'
|
||||
), false
|
||||
)
|
||||
from (
|
||||
select coalesce(jsonb_agg(entry.value order by entry.ordinal), '[]'::jsonb) as entries
|
||||
from jsonb_array_elements(${nativeRunFinalizations.recoveryHistory})
|
||||
with ordinality as entry(value, ordinal)
|
||||
where entry.value->>'kind' = 'native_cleanup_maintenance'
|
||||
) history
|
||||
)`,
|
||||
options.scope ? and(
|
||||
eq(heartbeatRuns.companyId, options.scope.companyId),
|
||||
options.scope.runId ? eq(heartbeatRuns.id, options.scope.runId) : undefined,
|
||||
options.scope.issueId ? eq(nativeRunFinalizations.issueId, options.scope.issueId) : undefined,
|
||||
options.scope.agentId ? eq(heartbeatRuns.agentId, options.scope.agentId) : undefined,
|
||||
) : undefined,
|
||||
options.after ? gt(heartbeatRuns.id, options.after) : undefined,
|
||||
),
|
||||
)
|
||||
.orderBy(asc(heartbeatRuns.id))
|
||||
.limit(options.limit);
|
||||
}
|
||||
|
||||
export type NativeCleanupOutcome = {
|
||||
runId: string;
|
||||
status: "settled" | "not_eligible" | "operator_required";
|
||||
} | {
|
||||
runId: string;
|
||||
status: "deferred";
|
||||
waitFor: "process_exit" | "session_release" | "environment_release" | "maintenance_settlement";
|
||||
} | {
|
||||
runId: string;
|
||||
status: "deferred";
|
||||
waitFor: "lease_expiry";
|
||||
retryAt: Date;
|
||||
};
|
||||
export type RetainedNativeSessionCleanup = (input: { companyId: string; runId: string }) => Promise<NativeCleanupOutcome>;
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
const operations = new WeakMap<Db, {
|
||||
tail: Promise<void>;
|
||||
runs: Map<string, Promise<NativeCleanupOutcome>>;
|
||||
}>();
|
||||
export function runRetainedNativeSessionCleanup(db: Db, input: { companyId: string; runId: string }, cleanup: RetainedNativeSessionCleanup) {
|
||||
let owned = operations.get(db);
|
||||
if (!owned) { owned = { tail: Promise.resolve(), runs: new Map() }; operations.set(db, owned); }
|
||||
const existing = owned.runs.get(input.runId);
|
||||
if (existing) return existing;
|
||||
const pending = owned.tail.then(() => cleanup(input)).finally(() => {
|
||||
if (owned.runs.get(input.runId) === pending) owned.runs.delete(input.runId);
|
||||
if (!owned.runs.size && operations.get(db) === owned) operations.delete(db);
|
||||
});
|
||||
owned.runs.set(input.runId, pending);
|
||||
owned.tail = pending.then(() => undefined, () => undefined);
|
||||
return pending;
|
||||
}
|
||||
export function pendingRetainedNativeSessionCleanup(db: Db, runId: string) {
|
||||
return operations.get(db)?.runs.get(runId);
|
||||
}
|
||||
|
||||
type NativeCleanupSweep = {
|
||||
cursor: string | null;
|
||||
pending: Promise<NativeCleanupOutcome[]> | null;
|
||||
};
|
||||
const nativeCleanupSweeps = new WeakMap<Db, NativeCleanupSweep>();
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
export function reconcileRetainedNativeSessionCleanups(
|
||||
db: Db,
|
||||
options: {
|
||||
cleanup: RetainedNativeSessionCleanup;
|
||||
onError?: (error: unknown, runId: string) => void;
|
||||
limit?: number;
|
||||
},
|
||||
): Promise<NativeCleanupOutcome[]> {
|
||||
let sweep = nativeCleanupSweeps.get(db);
|
||||
if (!sweep) {
|
||||
sweep = { cursor: null, pending: null };
|
||||
nativeCleanupSweeps.set(db, sweep);
|
||||
}
|
||||
if (sweep.pending) return sweep.pending;
|
||||
const owned = sweep;
|
||||
const limit = Number.isInteger(options.limit)
|
||||
? Math.max(1, Math.min(5, options.limit!))
|
||||
: 1;
|
||||
const operation = async () => {
|
||||
const selectCandidates = (cursor: string | null) =>
|
||||
findRetainedNativeSessionCleanupCandidates(db, { after: cursor ?? undefined, limit });
|
||||
let candidates = await selectCandidates(owned.cursor);
|
||||
if (candidates.length === 0 && owned.cursor !== null) {
|
||||
owned.cursor = null;
|
||||
candidates = await selectCandidates(null);
|
||||
}
|
||||
const outcomes: NativeCleanupOutcome[] = [];
|
||||
for (const candidate of candidates) {
|
||||
owned.cursor = candidate.runId;
|
||||
try {
|
||||
outcomes.push(await runRetainedNativeSessionCleanup(db, { companyId: candidate.companyId, runId: candidate.runId }, options.cleanup));
|
||||
} catch (error) {
|
||||
options.onError?.(error, candidate.runId);
|
||||
}
|
||||
}
|
||||
return outcomes;
|
||||
};
|
||||
|
||||
|
||||
const pending = Promise.resolve()
|
||||
.then(operation)
|
||||
.finally(() => {
|
||||
if (owned.pending === pending) owned.pending = null;
|
||||
});
|
||||
owned.pending = pending;
|
||||
return pending;
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export type RetainedNativeSessionCleanupScope = {
|
||||
companyId: string;
|
||||
runId?: string;
|
||||
issueId?: string;
|
||||
agentId?: string;
|
||||
};
|
||||
|
||||
export function notifyRetainedNativeSessionCleanup(tx: object, scope: RetainedNativeSessionCleanupScope) {
|
||||
return notifyDeliveryWork(tx, DELIVERY_QUEUES.retainedNativeSessionCleanup, JSON.stringify({ companyId: scope.companyId, runId: scope.runId, issueId: scope.issueId, agentId: scope.agentId }));
|
||||
}
|
||||
|
||||
|
||||
|
||||
const listeners = new WeakMap<object, Set<(scope: RetainedNativeSessionCleanupScope) => void>>();
|
||||
export function onRetainedNativeSessionCleanupRelease(db: object, listener: (scope: RetainedNativeSessionCleanupScope) => void) {
|
||||
let owned = listeners.get(db);
|
||||
if (!owned) { owned = new Set(); listeners.set(db, owned); }
|
||||
owned.add(listener);
|
||||
return () => {
|
||||
owned.delete(listener);
|
||||
if (!owned.size && listeners.get(db) === owned) listeners.delete(db);
|
||||
};
|
||||
}
|
||||
export function releaseRetainedNativeSessionCleanup(db: object, scope: RetainedNativeSessionCleanupScope) {
|
||||
for (const listener of listeners.get(db) ?? []) listener(scope);
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { notifyRetainedNativeSessionCleanup } from "./retained-native-session-cleanup-notifications.js";
|
||||
import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq, sql, type SQL } from "drizzle-orm";
|
||||
@@ -126,28 +127,31 @@ export async function settleStopOnlyCleanup(db: Db, lease: Lease, options: { att
|
||||
const receipt = remoteTerminationReceipt(lease, options.receipt), stopped = receipt?.state === "stopped";
|
||||
const now = new Date();
|
||||
const removeIntent: SQL = stopped ? sql`- 'sandboxStopAndRetain'` : sql``;
|
||||
const [updated] = await writeEnvironmentCleanup(db, tx => tx.update(environmentLeases).set({
|
||||
status: stopped ? "released" : "pending_cleanup", cleanupStatus: stopped ? "success" : "failed",
|
||||
failureReason: stopped ? null : "sandbox_stop_pending",
|
||||
releasedAt: now, lastUsedAt: now, updatedAt: now,
|
||||
metadata: sql`(${environmentLeases.metadata} - 'remoteExecutionTermination' ${removeIntent}) || ${JSON.stringify({
|
||||
...(stopped ? { remoteExecutionTermination: receipt, sandboxStopAndRetainReceipt: {
|
||||
schema: "paperclip.sandbox-stop-and-retain-receipt.v1", requestId: intent.requestId,
|
||||
companyId: lease.companyId, runId: lease.heartbeatRunId, leaseId: lease.id,
|
||||
provider: lease.provider, providerLeaseId: lease.providerLeaseId,
|
||||
...(intent.pluginId ? { pluginId: intent.pluginId, method: "environmentStopLease" }
|
||||
: { builtinProvider: lease.provider, method: "builtin.stopLease" }),
|
||||
confirmedAt: receipt.confirmedAt,
|
||||
} } : {}),
|
||||
pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0,
|
||||
})}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId),
|
||||
eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.provider, lease.provider!),
|
||||
eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), eq(environmentLeases.status, "pending_cleanup"),
|
||||
sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.attemptId}`,
|
||||
sql`${environmentLeases.metadata}->'sandboxStopAndRetain' = ${JSON.stringify(lease.metadata?.[SANDBOX_STOP_AND_RETAIN_KEY])}::jsonb`,
|
||||
sql`${environmentLeases.metadata}->>'pluginId' is not distinct from ${intent.pluginId}`,
|
||||
)).returning());
|
||||
const [updated] = await writeEnvironmentCleanup(db, async tx => {
|
||||
if (stopped) await notifyRetainedNativeSessionCleanup(tx, { companyId: lease.companyId, runId: lease.heartbeatRunId! });
|
||||
return tx.update(environmentLeases).set({
|
||||
status: stopped ? "released" : "pending_cleanup", cleanupStatus: stopped ? "success" : "failed",
|
||||
failureReason: stopped ? null : "sandbox_stop_pending",
|
||||
releasedAt: now, lastUsedAt: now, updatedAt: now,
|
||||
metadata: sql`(${environmentLeases.metadata} - 'remoteExecutionTermination' ${removeIntent}) || ${JSON.stringify({
|
||||
...(stopped ? { remoteExecutionTermination: receipt, sandboxStopAndRetainReceipt: {
|
||||
schema: "paperclip.sandbox-stop-and-retain-receipt.v1", requestId: intent.requestId,
|
||||
companyId: lease.companyId, runId: lease.heartbeatRunId, leaseId: lease.id,
|
||||
provider: lease.provider, providerLeaseId: lease.providerLeaseId,
|
||||
...(intent.pluginId ? { pluginId: intent.pluginId, method: "environmentStopLease" }
|
||||
: { builtinProvider: lease.provider, method: "builtin.stopLease" }),
|
||||
confirmedAt: receipt.confirmedAt,
|
||||
} } : {}),
|
||||
pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0,
|
||||
})}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId),
|
||||
eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.provider, lease.provider!),
|
||||
eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), eq(environmentLeases.status, "pending_cleanup"),
|
||||
sql`${environmentLeases.metadata}->>'pendingCleanupAttemptId' = ${options.attemptId}`,
|
||||
sql`${environmentLeases.metadata}->'sandboxStopAndRetain' = ${JSON.stringify(lease.metadata?.[SANDBOX_STOP_AND_RETAIN_KEY])}::jsonb`,
|
||||
sql`${environmentLeases.metadata}->>'pluginId' is not distinct from ${intent.pluginId}`,
|
||||
)).returning();
|
||||
});
|
||||
return updated ?? null;
|
||||
}
|
||||
|
||||
@@ -159,19 +163,22 @@ export async function settleAbsentSandbox(db: Db, lease: typeof environmentLease
|
||||
const termination = remoteTerminationReceipt(lease, { providerLeaseId: proof.providerLeaseId, state: "destroyed" });
|
||||
if (!termination) return false;
|
||||
const now = new Date();
|
||||
const changed = await db.update(environmentLeases).set({
|
||||
status: "released", cleanupStatus: "success", failureReason: "workspace_source_unavailable", releasedAt: now, updatedAt: now,
|
||||
metadata: sql`(${environmentLeases.metadata} - 'sandboxStopAndRetain' - 'nativeWorkspaceExportResume' - 'sandboxStopAndRetainReceipt') || ${JSON.stringify({
|
||||
remoteExecutionTermination: termination, sandboxAbsence: evidence,
|
||||
pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0,
|
||||
})}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId),
|
||||
eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.providerLeaseId, proof.providerLeaseId),
|
||||
eq(environmentLeases.status, lease.status),
|
||||
sql`exists (select 1 from ${heartbeatRuns} where ${heartbeatRuns.id} = ${lease.heartbeatRunId}
|
||||
and ${heartbeatRuns.companyId} = ${lease.companyId}
|
||||
and ${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceFinalizationOwner,token}' = ${proof.ownerToken})`,
|
||||
sql`${environmentLeases.metadata} is not distinct from ${lease.metadata === null ? null : JSON.stringify(lease.metadata)}::jsonb`,
|
||||
)).returning({ id: environmentLeases.id });
|
||||
const changed = await db.transaction(async tx => {
|
||||
await notifyRetainedNativeSessionCleanup(tx, { companyId: lease.companyId, runId: lease.heartbeatRunId! });
|
||||
return tx.update(environmentLeases).set({
|
||||
status: "released", cleanupStatus: "success", failureReason: "workspace_source_unavailable", releasedAt: now, updatedAt: now,
|
||||
metadata: sql`(${environmentLeases.metadata} - 'sandboxStopAndRetain' - 'nativeWorkspaceExportResume' - 'sandboxStopAndRetainReceipt') || ${JSON.stringify({
|
||||
remoteExecutionTermination: termination, sandboxAbsence: evidence,
|
||||
pendingCleanupInFlight: false, pendingCleanupLeaseExpiresAtMs: 0,
|
||||
})}::jsonb`,
|
||||
}).where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId),
|
||||
eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId!), eq(environmentLeases.providerLeaseId, proof.providerLeaseId),
|
||||
eq(environmentLeases.status, lease.status),
|
||||
sql`exists (select 1 from ${heartbeatRuns} where ${heartbeatRuns.id} = ${lease.heartbeatRunId}
|
||||
and ${heartbeatRuns.companyId} = ${lease.companyId}
|
||||
and ${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceFinalizationOwner,token}' = ${proof.ownerToken})`,
|
||||
sql`${environmentLeases.metadata} is not distinct from ${lease.metadata === null ? null : JSON.stringify(lease.metadata)}::jsonb`,
|
||||
)).returning({ id: environmentLeases.id });
|
||||
});
|
||||
return changed.length === 1;
|
||||
}
|
||||
Reference in new issue
Block a user