mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-02 08:28:48 +01:00
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/205/changes/paperclip/205 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents receive selected company skills before each turn. > - Imported skills can need files from the whole source repository. > - The current path stores those files in Postgres and reads large JSON payloads during preparation. > - This pull request stores verified packages in the configured file or object store and keeps preparation queries small. > - It also converts old rows automatically and restores missing files from the exact saved commit. > - The result reduces database traffic while preserving skill versions and repository dependencies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Company skill import, version storage, and preparation before agent turns. **Current behavior** Repository snapshot bodies live in JSONB. Skill preparation can read those bodies again even when the files already exist on disk. Historical packages and native assets also need a bounded cleanup path. **Proposed behavior** Store package archives and manifests through the existing storage service. Keep version identities, receipts, hashes, and recovery metadata in Postgres. Convert legacy rows after the stored package passes read-back verification. A missing package can use legacy bytes or its exact Git commit under current repository authorization. No separate data-migration command is required. **Reason and benefit** The warm path sends no repository bodies through SQL and reads no object bytes. Cleanup runs through the existing deadline coordinator. Current versions, active runs, explicit pins, and authored content retain their protection. **Breaking changes** The normal application schema upgrade is required. Single-version and skill-file responses keep their existing shape. Version lists, current-version summaries, and test-run version summaries now contain file names and kinds without file bodies. Diff and restore fetch selected versions through the existing single-version endpoint. Older server builds must not write after conversion unless they support the new package format. The storage override and rollback limits are documented in `doc/skill-package-storage.md`. **Additional context** Related work: #15731 preserves repository dependencies; #17923 changes version-cache scheduling; #18080 changes delivery to remote Codex homes. This change owns package persistence, acquisition, and expiry. It does not replace those delivery or scheduler changes. ## What Changed - Add company-scoped archive receipts, exact-commit recovery recipes, migration deadlines, and a durable object-deletion queue. - Add bounded archive upload, read-back verification, extraction, concurrent request coalescing, and verified local caches. - Convert old snapshots and versions automatically. Keep legacy bytes when verification fails. - Recover missing files through current source authorization and the original commit. Validate repository identity, file hashes, modes, and the aggregate hash. - Expire unused recoverable Git packages and native assets after 30 days. Preserve active pins and the durable copy of authored or unproven content. - Keep warm preparation queries metadata-only. Bound file digest caches and avoid repeated body hashing when file identities are unchanged. - Extend the existing storage contract with cancellation. Register package work with the existing coordinator. Optional conversion retries retain the original bytes and may wait for ingress without blocking sleep. - Add regression tests, operating guidance, a deep trace, benchmark receipts, and an optional maintenance command. ## Verification ### Final master integration and review repair — October 29, 2026 Current head: `a6065424249e763dfeac37161f1b52a9f25bd9b7`. Commit `f8bbdd3dbd` resolves the conflicts with master `76a796e964`; the final two commits repair the Apex deletion findings below. - Keep both package-maintenance and version-retention notifications in their write transactions. - Keep the shared run-pin predicate and master's controller-lease deadline metadata. Package expiry reads the updated protected-version result. - Preserve every master migration entry. Generate migration **0415 (`0415_zippy_zemo.sql`)** after master's 0414. Its skill SQL is byte-for-byte unchanged from the previously tested idempotent migration. Retain only the five newest schema snapshots. Migration numbering and safety checks pass. - Repair failed file/folder deletion: preserve a complete temporary subtree until transaction commit, restore on upload or SQL failure, and restore callback failures before releasing the mutation lock. Both PostgreSQL regressions reproduced missing files before the fix and now pass. They cover binary data, executable permissions, empty folders, unlisted files, unchanged database/version rows, successful retry after upload-claim expiry, and historical reads. - Reconcile lost commit replies under the same advisory and row locks. Check the saved version and source identity before restoring. Keep committed deletions intact and preserve backups when the database outcome is unknown or a newer edit exists. Four real-PostgreSQL outcome tests pass; the prior code failed three of them. - Full workspace typecheck and production build pass on the final commit. All **204** focused PostgreSQL package/cache/retention/authoring and filesystem-boundary tests pass, with none skipped. - Apex review **18 passes 5/5**, with zero new comments on the exact final head. All **34** review conversations are resolved. - [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/40914116508) completes successfully on attempt 2. The head has **55 successful checks**, two neutral reports, two skipped checks, and a successful Snyk status. No checks are pending or failing. All eight browser shards pass. - Runner Vitest 2/2 timed out once in the unchanged ACPX controller-replacement fixture. The same job passed on the preceding commit. One unchanged CI retry passes all five ACPX recovery cases, **2,253** Runner package cases and **126** native server integration cases; their standard one and five skips remain. The initial timeout's root cause is not established. Local diagnosis also hit the fixture's eight-second session.open timeout with an authenticated Runner; no local Runner pass is claimed. - Both full local `pnpm test:run` attempts were stopped when Apex identified deletion defects and source repair became necessary; neither is claimed as a passing full local run. The final **204** focused tests, full workspace typecheck, and production build pass. Full-suite acceptance uses the final-head CI jobs. The superseded merge-head CI included runners terminated by shutdown signals; each source repair started a new full run. - Auto-merge is enabled for squash merge. The final head is conflict-free and all checks pass. GitHub's active master ruleset still requires a code-owner approval for the dependency files. The existing review requests are with `devinfoley`, `nickyleach`, and `forgottendev`; no policy bypass was used. - The earlier hosted stress and full native receipts below remain tied to `a2e597d800`; they are not relabeled as a deployment of this final integration. The earlier local PostgreSQL startup failure remains recorded below; PostgreSQL ran successfully for this integration verification. ### Full native skill execution — October 29, 2026 The full native-agent acceptance gap is closed on candidate `a2e597d800eb989d565780e7aa947d5368676444`. Two qualifying native turns used OpenAI API / `gpt-5.4-mini` in real Daytona sandboxes, on the same disposable hosted TEST tenant as the stress test. A follow-up on the same task consumed a newly saved skill version. The task text asked for the assigned skill workflow. It did not contain the proof nonce or expected receipt. The assigned authored skill contained a Python verifier and a random package-local nonce. The agent read the actual installed skill files and executed that verifier against the imported, automatically converted 1,360-file repository package. It checked the repository file count and reference/binary bytes, and directly executed the repository's executable probe. Independent verification matched the emitted JSON to the nonce and expected byte hashes. The agent persisted the same receipt as its task response; native finalization committed, workspace finalization succeeded, and task state became Done. | Qualifying turn | End-to-end run time | Recorded skills.prepare | Repository files | Binary hex | Shell receipt SHA-256 | |---|---:|---:|---:|---|---| | Initial task | 47.881 s | 1822 ms | 1360 | `00ff07800a` | `a1deb30dc095f2306c7aa9cc5ec396da12c68961cf5b809f4d7c20f7fc68493b` | | Corrected follow-up, new nonce/version | 37.974 s | 377 ms | 1360 | `00ff07800a` | `dbcdc33fa1376b4edc136feae1cdf770560a73292467e93619efdda20e4282a7` | Both successful shell receipts reported `probe: STRESS18082` and `assetSha256: e8ea7a2f15f7bb3b332a46adbef848465884c8996595aa234748befd544d1f3a`. Their proof nonces, selected version IDs and receipt hashes differed, and both ran on the same task. These are observed timings, not latency SLOs; they include real provider work. The service stress-test timings remain a separate measurement boundary. Earlier failures remain in the evidence: - The original five-minute Daytona allocation timeout never reached model execution. After normal owned-stack restore, its durable cleanup record was `expired` / `success`. The cause of that original allocation timeout remains unknown; a later success establishes recovery only. - Initial continuation setup used unsupported generic agent-update fields and returned HTTP 409 before dispatch. Setup was corrected to use the supported skill-assignment and versioned instruction commands. - The intervening completed native turn read repository bytes and ran the probe, but omitted the verifier JSON under the older test agent's overly-specific instruction file. Its strict receipt validator failed; it is not counted as a qualifying turn. Correcting the test agent's instruction file and sending an explicit task-scoped wake produced the qualifying follow-up above. - The pinned application's immutable rate card omits `gpt-5.4-mini`, so completed usage triggered the normal block-on-unpriced budget policy. Supported append-only accounting corrections estimated the three completed runs using the [official OpenAI model rate card](https://developers.openai.com/api/docs/models/gpt-5.4-mini): $0.75 input, $0.075 cached input, $4.50 output per million tokens, assuming standard processing. Estimates totaled 12.482655 cents. Original usage and missing-price records are preserved; the $1 hard stop and unpriced-usage guard stayed active. - Model shell attempts included a missing `xxd` command and incorrect guessed file paths. Those failed calls and subsequent successful actual verifier calls remain visible in complete event streams. No zero-tool-error claim is made. No product source changed during this continuation. The previously measured contention, cold-start and memory limits remain. The later merge resolution is covered by the separate integration checks above; these hosted receipts qualify their recorded candidate. ### Hosted stress test — October 29, 2026 **Hosted acceptance:** full native-agent execution and a changed-skill follow-up pass on the recorded hosted candidate. The final integration section above records the subsequent conflict resolution and its separate checks. Tested baseline `929e840595c8933ca485ac7a2e99413e1e11ca64` and this exact candidate sequentially on one disposable TEST tenant, with external Neon PostgreSQL and actual S3. The fixture contained **1,360 files / 22.4 MB / ten skills sharing one repository**. Measurements cover deployed skill-preparation and native-context services in an adjacent diagnostic process, not a complete agent turn. The container had **24 vCPU / 24 GB**. | Warm workload | Median before → after | p95 before → after | DB returned value bytes/preparation before → after | |---|---:|---:|---:| | One skill, serial | 1,060 → 125 ms | 1,303 → 155 ms | 22,610,389 → 1,771 | | One skill, four concurrent | 4,415 → 297 ms | 5,143 → 328 ms | 22,610,389 → 1,771 | | Ten skills, serial | 5,723 → 535 ms | 6,057 → 600 ms | 224,739,030 → 10,671 | | Ten skills, four concurrent | 24,410 → 1,986 ms | 24,833 → 2,221 ms | 224,739,030 → 10,671 | - Candidate warm preparation performed **zero S3 GETs**. Ordinary-pool queries fell from 16 to 9 for one skill and 61 to 9 for ten, excluding dedicated lock connections. DB numbers measure returned values before Drizzle decoding, not wire bytes. Baseline/candidate sample counts were 30/100, 32/100, 5/50 and 8/48 respectively; the small baseline ten-skill samples are comparative evidence, not SLO estimates. - **Application-UID soak:** 800 ten-skill preparations over 10.3 minutes, four workers paced by 2.5 seconds; **zero failures**, median **559 ms**, p95 **775 ms**, p99 **910 ms**, zero warm S3 reads. All 726 accompanying health/company-list requests returned HTTP 200. Their end-to-end p95 was 569/502 ms, including network latency. The probe overlaps most of the soak. Soak DB-response capture was limited to its first 60 seconds. - **Important remaining costs:** an unpaced 16-way ten-skill burst had p95 **8.32 seconds** because company-level preparation serializes. Cold preparation took approximately **1.1–2.5 seconds** for one skill and **7.4–10.0 seconds** for ten. Ten-skill diagnostic RSS remained about **1.17 GiB** (baseline 1.18 GiB), and the soak peaked at **1.35 GiB**. A concurrent whole-container sample was 3.45 GB. This does not establish suitability for a small-memory host or eliminate per-entrypoint native asset duplication. - **Upgrade without a manual data migration passed:** legacy snapshot JSON shrank from 22,465,691 bytes to `[]`, with a stored package and zero pending conversions. - **Recovery passed:** sixteen concurrent cold requests coalesced into two S3 GETs; full bytes/binary/executable modes matched; same-size corruption repaired; an injected storage failure preserved warm access and failed cold access safely. A disabled public-Git source denied recovery, and after the actual five-minute backoff its exact SHA reconstructed all 229 files. Cross-company descriptor access was denied. Private-Git credential revocation was not tested. - **Retention passed:** reclaimed 27,754,496 allocated cache bytes, deleted an unreferenced package row and both objects, preserved aged authored history, and expired/reacquired reconstructible Git content. A version pinned by the failed native run was correctly retained. Only owned fixture timestamps were aged. - **Browser UI passed:** authored skill creation, saved edit surviving reload, version history, and restoration of earlier content. - **Initial native attempt (superseded by the qualifying turns above):** one real run failed at Daytona allocation after five minutes (`setup_failed`), before any model execution. Provider cleanup also reported failure. That attempt does not count as a passing native turn. Its receipt and all evidence remain preserved; its cleanup later recorded success after normal restoration. - Initial A/B commands ran as SSH root. Diagnostic-owned lock/cache permissions caused one setup HTTP 500 and the first UID-1000 soak to fail; exact fixture paths were repaired. The final soak, retention and recovery ran as application UID 1000. A fresh UID-1000 one-skill batch also passed (30 samples, median 116 ms, p95 321 ms, 1,771 DB value bytes/preparation). Earlier harness-export, SSH, npm provenance, backoff and UI-wait failures remain recorded alongside their successful retries. - Private evidence includes raw timings, assertions, HTTP probes, screenshots and failure/cleanup receipts. Initial failed-campaign cleanup suspended and retained the owned TEST tenant and released the shared QA account lease at **15:22:41 UTC**. The tenant was subsequently resumed under its existing ownership for the native execution qualification above. Final cleanup completed at **15:53:09 UTC**: the account lease is released and the tenant is suspended/retained under the original October 30, 12:24:52 UTC retention deadline. Its separate stack lease remains held; no active or pending-cleanup environment lease remained. The original retention window is deliberately preserved even after successful continuation. - Earlier hosted-tested head: `a2e597d800eb989d565780e7aa947d5368676444`, integrated with master `929e840595`. The earlier merge conflicts were resolved at this head. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/40884087629) passed; all checks are complete with 55 successes, two neutral reports, two skips, and no failures. [Apex review 15](https://github.com/paperclipai/paperclip/pull/18082#issuecomment-6480257258) passed 5/5 with zero findings on this exact head. All review conversations are resolved. - Ten real-filesystem boundary cases and server typecheck pass locally for the security fix. Coverage includes cache roots, descendants, individual files, symlink aliases, recursive ancestors, a future concurrent cache fill, and allowed sibling directories. [Current-head private-project integration tests](https://github.com/paperclipai/paperclip/actions/runs/40884087629/job/123483306405) passed all 29 cases, including authorized and unauthorized imports of a cached private package. Full typecheck, build, server suites, all eight browser shards, and Runner checks passed in current-head CI. Local database tests could not run because PostgreSQL could not initialize; this limitation is recorded below. - The earlier generated migration preserved the newest five snapshots. The final integration renumbers its unchanged SQL to 0415. Migration safety/numbering checks pass. Its SQL is byte-for-byte identical to the previously verified idempotent skill migration; repeated application on this latest schema was blocked by the same local database startup failure. - The broader verification before the final migration-only rebase passed 1,025 package, lifecycle, remote-launch, native-session, and asset-retention tests, 52 focused UI tests, full workspace typecheck/build, token gates, and map checks. The linked complete CI run passed before the latest focused fixes. - A disposable MinIO instance passed multipart upload, verified read-back, and cold restoration of an 18 MB archive. - Twenty warm samples with a synthetic 1,349-file, 22.8 MB repository retained five metadata queries and zero object reads. Median preparation was 256 ms for one selected skill and 691 ms for ten, with 5 ms emulated network delay each way. These are local measurements, not production latency guarantees. - No production instance was changed. Earlier verification failures and their disposition are retained below. <details> <summary>Earlier failures, review findings, fixes, and rerun evidence</summary> - The first CI pass found a stale inventory-span expectation and a missing repository-read span. Both are fixed; all three observability tests pass. One browser test copied page text into its YAML parser after reload; its first unchanged-head retry passed, but the race recurred after rebasing. The test now waits for the renamed URL and completed reload before copying the editor, retaining its original assertions. The earlier full local suite did not complete cleanly: it reported setup timeouts and assertions, while isolated relevant reruns passed. The complete current-head CI run closes that broad verification gap. No production instance was changed. - The first Apex review scored 2/5 and identified six issues. The fixes protect authored history even when it shares an in-flight Git upload, recheck upload expiry during orphan deletion, bound failed-upload cleanup, retain per-version kind/encoding metadata, add sanitized failure diagnostics, and correct the migration guide. The final focused run passes all 140 tests; the preceding eight-file run passed all 149 tests, including local storage and S3 diagnostics. Server typecheck and migration safety/numbering checks pass after the fixes. The second Apex review scored 1/5 and found unbounded maintenance storage calls and mismatched package/Git batch limits. Those are fixed with a 30-second cancellable cleanup deadline, shared 100 MiB/10,000-file scanner and storage limits, and bounded Git read batches. The proposed manual lockfile edit was rejected with evidence of the enforced CI-owned lockfile policy. All 19 storage/recovery unit tests and server typechecking pass. The new database-backed cleanup/conversion tests were locally skipped because the host exhausted its PostgreSQL semaphore capacity (confirmed directly by initdb). That gap is closed by current-head CI: [package-store tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109300889) pass all 18 cases and [package-worker tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109301157) pass all 7, with no skips in either suite. The corrected YAML browser shard passes. A separate planning-fixture test failed its cleanup DELETE with an activity-log foreign key violation after its behavior assertions passed; its one unchanged-head retry passed. The third Apex review scored 1/5 and identified a one-slot pool deadlock, projected path limits, cleanup holding the preparation lock during archive reads, and a publication/expiry race. These are fixed with transaction-local reads, prefix-aware bounds, isolated verification, and shared package row locks with a publication grace. The separate Paperclip review also found stale historical inspection, eager history downloads, and conversion retries blocking sleep. Those are fixed. Local PostgreSQL capacity recovered: the new database regressions ran, including the one-slot fork, paused storage during a warm turn, and a locked publication during expiry. The expanded run passed 144/145 tests; its remaining assertion still expected file bodies from the version list and was updated to fetch the selected version. The corrected company-skill suite passes all 81 tests, and the final package/projection run passes all 27. All 52 focused UI tests pass after updating their query-provider fixtures; the first UI run exposed a fixture remount, and the follow-up exposed an editing syntax error, both corrected. Full workspace typecheck and build, token gates, and map checks pass. The sandbox initially blocked the build tool’s local IPC socket; the authorized build rerun passed. Master then claimed migration 0401. The branch is rebased onf8226b0515, the unchanged SQL is regenerated as 0402, and repeated application passed twice on the combined schema. Post-rebase workspace typecheck passes. All 1,025 package, lifecycle, remote-launch, native-session, and asset-retention tests pass on the combined head. All CI checks passed for head 9d80a3b52d. The fourth Apex review scored 3/5 and found a save/edit lock-order cycle, a retry retaining an uncommitted source ID in its recovery recipe, and a summary read racing conversion. Commit b7d9237c6a fixes all three. The new concurrency and retry tests pass within all 61 package/source tests, and server typecheck passes. Master then claimed migration 0402. The branch is now rebased on65fa1cd617and the unchanged SQL is regenerated as 0403. All 68 package, source, and worker regressions, server typecheck, migration safety/numbering, and repeated migration application pass on current head62553671a0. Subsequent verification and review outcomes are recorded below. - The fifth Apex review scored 4/5 and found cached version reads unnecessarily depending on object storage and conversion reads missing the coordinator cancellation signal. Commit1eb90d80befixes both: cached reads verify bytes without the preparation lock; manifest and archive reads carry cancellation through conversion. The first focused run passed 70/71 tests, including all three new cases, but the existing diagnostic test observed no warning. Its fixture now sets an explicitly expired migration deadline; the final worker suite passes all nine cases. Server typecheck passes. - The sixth Apex review scored 4/5 with one finding: concurrent cached inventories were unbounded. Commit4b7589e14cgives cached inventories and previews a two-reader/32-waiter limit independent of cold storage. The first expanded pressure test observed a third reader before the preview path had the same bound; both paths now share the local limit. All 73 focused cases pass, followed by all 29 package cases after the final overload adjustment; server typecheck passes. The separate Paperclip review found no must-fix or should-fix issues. Its optional suggestion to persist durability verification was considered: cleanup intentionally verifies current external bytes before discarding the last local authored copy, and the missing-object regression requires that check. Shared-package eviction happens on cold fills with at most two attempts and a ten-second budget. - The seventh Apex review reached 5/5 with one remaining P2: timeout-bearing manifest reads bypassed request coalescing. The security review also found repeated archive acquisition after a cold administrative read. Commit79a80a39d2shares manifest loads with per-caller cancellation and cancels the underlying work when the last caller leaves; coalesces identical cold version/file requests; and publishes verified restored packages into the shared cache. Publication uses a short nonblocking lock, while eviction verification runs outside it. The first expanded run passed 73/74 tests and exposed a missing originating DB for cache publication inside read-only inspection; transaction readers now pass that owner explicitly, keeping metadata on the transaction. The corrected run passed 75/75. A final eviction test initially expected the shared package to remain after only its separate runtime projection was reused; it now explicitly accesses the shared package during verification. All 32 final package tests pass, including independent caller cancellation, one acquisition across concurrent/repeated reads, and warm preparation during stalled eviction verification. Server typecheck passes. - The eighth Apex review scored 4/5 with one finding: decoded cold inventories outlived the storage semaphore while waiting for cleanup. Commit9d7921d1ecadds a separate two-response/32-waiter lifetime bound held through cleanup, preserving the independent warm-read queue and forwarding cancellation through eviction verification. The new regression pauses cleanup after two responses decode, verifies a third archive stays queued, and completes warm preparation before releasing cleanup. All 77 package/source/worker cases and server typecheck pass. On the preceding head, serialized server shard 6/9 was interrupted with “The operation was canceled” after 57 seconds of tests; the log contains no failing assertion. Fresh latest-head CI reruns that shard along with the other gates. - The ninth Apex review scored 4/5 with one P1: cold responses could own both permits while waiting on a one-slot ordinary database pool held by a queued transaction reader. Commita94f0e409freads caller-scoped metadata before admission and isolates all acquisition/cleanup SQL on a private connection. The new real-PostgreSQL regression reproduces the deadlock against the previous head and passes with the fix; all 78 focused cases and server typecheck pass. The previous CI run completed with 54 successful checks, including both formerly interrupted/stuck serialized shards. - The tenth Apex review scored 4/5 with one P1: preloading legacy bodies before admission could retain many full repositories in queued requests. Commit276e353a37removes that preload and also bounds legacy manifest derivation/fallback, using private connections inside the I/O slot. All 80 focused tests pass; the final synchronized package suite passes all 36 tests and server typecheck passes. - Apex review 11 returned 5/5 with zero comments on276e353a37. While it ran, master8bc051245fadded migration 0403, creating a metadata conflict. Merge commite1783764d9integrates master and generates 0404_clumsy_leo.sql; its skill SQL is byte-for-byte identical to the previously verified migration. The combined schema passes migration safety/numbering, repeated SQL application twice, all 80 focused PostgreSQL tests, full workspace typecheck, and full build. The precedinga94f0e409fCI run was superseded and cancelled while its final runner job was active; no old-head success is claimed for that run. - Final acceptance: Apex review 12 returned 5/5 with zero comments one1783764d9. All 28 review threads are resolved. The final head has 55 successful checks, two neutral reports, two skipped checks, and no pending or failing checks; the PR workflow completed successfully. Superseded run 40789121902 was cancelled to free its workflow slot for the final head; normal cancellation did not finish promptly, so force cancellation completed it. - October 29 conflict resolution (867c00673b): master929e840595added installed-selection reuse and bundled relocation, routine cleanup notifications, a hardened YAML copy helper, and migrations through 0413. The merge retains those behaviors together with package read scoping and the public-selection fast path; restricted selections still reauthorize and the full row read rechecks scope. Generated migration 0414 retains the existing SQL unchanged. Workspace typecheck/build and migration checks pass. Local database verification is unavailable: initdb reports semget / “could not create semaphores: No space left on device”; all 168 cases skipped, and repeated migration application failed at startup. Docker returned HTTP 500 during container creation, so no Docker test pass is claimed. Current-head CI must close this gap. - Apex review 14 scored 3/5 with one P1: a local import could copy cached private package files into a company-wide skill. Commita2e597d800rejects cache trees and ancestors through a canonical filesystem boundary, including aliases and ancestors checked before a concurrent fill. Ten local filesystem tests and server typecheck pass. The existing private-project integration case now rejects package-cache imports for both authorized and unauthorized users. The preceding merge commit’s complete CI run passed, and logs confirm all 168 locally skipped cases executed successfully: package store 36, sources 35, package worker 9, runtime selection 4, observability 3, authoring 81. - Apex review 15 returned 5/5 with zero new findings ona2e597d800. The separate Paperclip review found no must-fix or should-fix issues. Its optional suggestions were considered: unexpected filesystem cleanup failures remain visible, the durable marker cannot replace checking that bytes still exist before deleting a last local authored copy, and administrative history batching remains outside the pre-turn optimization. The cache-path literal centralization suggestion is also optional; the current producer and guard use the same directory, covered by the new integration regression. All review conversations are resolved. - Earlier October 29 CI/Apex acceptance:a2e597d800was mergeable at that check, Apex review 15 passed 5/5 with zero findings, and all review conversations were resolved. The hosted verification above records the later acceptance limits and renewed conflicts. Workflow 40884087629 completed successfully, with 55 successful checks, two neutral reports, two skips, and no pending or failed checks. The project-skill-sources job log confirms all 29 cases ran and passed, closing the latest integration gap. The preceding merge workflow also confirmed all 168 focused database cases ran and passed. No local PostgreSQL or Docker pass is claimed. </details> ## Risks - First use after eviction can require storage or Git access. Revoked grants, removed sources, and unreachable commits can prevent reconstruction. Recovery never substitutes a newer commit. - Historical packages without a proven recipe keep durable bytes. Cleanup favors retention when a run pin cannot be interpreted safely. - New uploads and conversion use bounded work and persisted retries. Large initial conversions still add temporary storage and database load. - Optional cleanup can wait until the next server wake. Supported notifications assume one owning server process. - Per-entrypoint native tree duplication and cold preparation cost remain. The final ten-skill cold sample took 47.6 seconds on the shared host. - Generic authored edits can still hold an existing edit transaction while a package uploads. - Explicit file/folder deletion uses temporary disk space for rollback. An uncertain commit with a changed skill or unavailable database retains that backup and logs its location for recovery. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and tests. The exact served model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1011 lines
57 KiB
TypeScript
1011 lines
57 KiB
TypeScript
import { beginIdleTrackedWork, idleWorkSnapshot } from "../services/task-admission.js";
|
|
import { randomUUID } from "node:crypto";
|
|
import { eq, sql } from "drizzle-orm";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import {
|
|
agentInstructionWorkingCopies, agents, agentApiKeys, agentWakeupRequests, budgetPolicies, companies, companySecretProposals, createDb,
|
|
adapterAuthSessions, environments, environmentLeases, executionWorkspaces,
|
|
feedbackExports, feedbackVotes, companySkills, companySkillVersions, companySkillRepositorySnapshots,
|
|
heartbeatRuns, issues, issueWatchdogs, issueThreadInteractions, issueQuestionResponseDeliveries,
|
|
projects, projectWorkspaces, routines, routineTriggers, toolApplications, toolConnections, workspaceOperations, executionWorkspaceRuntimeLeases, type Db,
|
|
} from "@paperclipai/db";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
|
|
import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js";
|
|
|
|
const now = Date.UTC(2020, 0, 1);
|
|
const held = (): IdleSleepDrainStatus => ({
|
|
draining: true, startedAt: new Date(now - 1000), expiresAt: new Date(now + 60_000),
|
|
activeRuns: 0, pendingWakes: 0,
|
|
});
|
|
const unknown = { version: 1, backgroundWork: "unknown" };
|
|
const present = { version: 1, backgroundWork: "present" };
|
|
const none = { version: 1, backgroundWork: "none" };
|
|
const ownerId = "d0b833f4-4098-42de-8420-1907f3aa4895";
|
|
const owned = () => ({ ...held(), ownerId });
|
|
const emptyLocal = async () => "none" as const;
|
|
|
|
describe("idle backup checkpoint", () => {
|
|
const database = (blocked = false) => {
|
|
const transaction = vi.fn(async (run: (tx: unknown) => Promise<unknown>) => run({ execute: async () => [{ blocked }] }));
|
|
return { db: { transaction } as unknown as Db, transaction };
|
|
};
|
|
const backup = (work: () => Promise<boolean> = async () => true) => vi.fn(async () => {
|
|
const finish = beginIdleTrackedWork();
|
|
try { return await work(); } finally { finish(); }
|
|
});
|
|
it("backs up after durable inspection and rechecks local work before authorizing sleep", async () => {
|
|
const { db, transaction } = database();
|
|
const local = vi.fn(emptyLocal);
|
|
const checkpoint = backup(async () => {
|
|
expect(transaction).toHaveBeenCalledOnce();
|
|
expect(idleWorkSnapshot().active).toBe(1);
|
|
return true;
|
|
});
|
|
expect(await readIdleSleepSafety(db, owned, () => now, ownerId, local, undefined, checkpoint)).toEqual(none);
|
|
expect(checkpoint).toHaveBeenCalledOnce();
|
|
expect(local).toHaveBeenCalledTimes(2);
|
|
});
|
|
it.each(["present", "unknown"] as const)("avoids DB probes and backups for known local %s work", async state => {
|
|
const { db, transaction } = database();
|
|
const checkpoint = backup();
|
|
const plugins = vi.fn();
|
|
expect(await readIdleSleepSafety(db, owned, () => now, ownerId, async () => state, plugins, checkpoint))
|
|
.toEqual({ version: 1, backgroundWork: state });
|
|
expect(transaction).not.toHaveBeenCalled();
|
|
expect(plugins).not.toHaveBeenCalled();
|
|
expect(checkpoint).not.toHaveBeenCalled();
|
|
});
|
|
it("does not start a backup for durable work or an unowned hold", async () => {
|
|
const checkpoint = backup();
|
|
expect(await readIdleSleepSafety(database(true).db, owned, () => now, ownerId, emptyLocal, undefined, checkpoint)).toEqual(present);
|
|
expect(await readIdleSleepSafety(database().db, owned, () => now, "stale", emptyLocal, undefined, checkpoint)).toEqual(unknown);
|
|
expect(checkpoint).not.toHaveBeenCalled();
|
|
});
|
|
it.each(["failed", "threw", "untracked", "concurrent", "expired", "replaced"])("keeps the instance awake when the backup is %s", async kind => {
|
|
let clock = now;
|
|
let status = owned();
|
|
const work = async () => {
|
|
if (kind === "threw") throw new Error("private archive path");
|
|
if (kind === "concurrent") { const finish = beginIdleTrackedWork(); finish(); }
|
|
if (kind === "expired") clock += 60_000;
|
|
if (kind === "replaced") status = { ...status, ownerId: "replacement" };
|
|
return kind !== "failed";
|
|
};
|
|
const checkpoint = kind === "untracked" ? work : backup(work);
|
|
expect(await readIdleSleepSafety(database().db, () => status, () => clock, ownerId, emptyLocal, undefined, checkpoint)).toEqual(unknown);
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
});
|
|
it("retains actual backup work after the owning hold expires", async () => {
|
|
let finishBackup!: () => void;
|
|
const pending = new Promise<void>(resolve => { finishBackup = resolve; });
|
|
const checkpoint = backup(async () => { await pending; return true; });
|
|
let clock = now;
|
|
const scan = readIdleSleepSafety(database().db, owned, () => clock, ownerId, emptyLocal, undefined, checkpoint);
|
|
await vi.waitFor(() => expect(checkpoint).toHaveBeenCalledOnce());
|
|
clock += 60_000;
|
|
expect(idleWorkSnapshot().active).toBe(1);
|
|
expect(await readIdleSleepSafety(database().db, owned, () => clock, ownerId, emptyLocal)).toEqual(unknown);
|
|
finishBackup();
|
|
expect(await scan).toEqual(unknown);
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
});
|
|
});
|
|
|
|
|
|
describe("idle sleep safety failure boundaries", () => {
|
|
it.each([
|
|
{ draining: false }, { activeRuns: 1 }, { pendingWakes: 1 },
|
|
{ startedAt: null }, { expiresAt: new Date(now) },
|
|
])("does not query durable work without a quiet admission hold: %j", async (change) => {
|
|
const transaction = vi.fn();
|
|
expect(await readIdleSleepSafety({ transaction } as unknown as Db, () => ({ ...held(), ...change }), () => now))
|
|
.toEqual(unknown);
|
|
expect(transaction).not.toHaveBeenCalled();
|
|
});
|
|
|
|
it("fails closed when the database is unavailable", async () => {
|
|
const db = { transaction: vi.fn().mockRejectedValue(new Error("private connection detail")) } as unknown as Db;
|
|
expect(await readIdleSleepSafety(db, held, () => now, undefined, emptyLocal)).toEqual(unknown);
|
|
});
|
|
|
|
it.each([
|
|
{ draining: false }, { startedAt: new Date(now) }, { expiresAt: new Date(now) },
|
|
{ activeRuns: 1 }, { pendingWakes: 1 },
|
|
])("rejects a report when the hold changes during the durable scan: %j", async (change) => {
|
|
const getStatus = vi.fn().mockReturnValueOnce(held()).mockReturnValue({ ...held(), ...change });
|
|
const execute = vi.fn().mockResolvedValue([{ blocked: true }]);
|
|
const transaction = vi.fn(async (fn: (tx: unknown) => Promise<unknown>) => fn({ execute }));
|
|
expect(await readIdleSleepSafety({ transaction } as unknown as Db, getStatus, () => now, undefined, emptyLocal)).toEqual(unknown);
|
|
expect(transaction).toHaveBeenCalledWith(expect.any(Function), { isolationLevel: "repeatable read", accessMode: "read only" });
|
|
});
|
|
|
|
it("reports persisted work when the admission hold remains unchanged", async () => {
|
|
const execute = vi.fn().mockResolvedValue([{ blocked: true }]);
|
|
const transaction = vi.fn(async (fn: (tx: unknown) => Promise<unknown>) => fn({ execute }));
|
|
expect(await readIdleSleepSafety({ transaction } as unknown as Db, held, () => now, undefined, emptyLocal)).toEqual(present);
|
|
});
|
|
});
|
|
|
|
describe("idle sleep admission and local work", () => {
|
|
const emptyDb = () => ({ transaction: async (fn: (tx: unknown) => Promise<unknown>) => fn({ execute: async () => [{ blocked: false }] }) }) as unknown as Db;
|
|
it("requires the exact current idle owner and a bounded expiry", async () => {
|
|
expect(await readIdleSleepSafety(emptyDb(), held, () => now, undefined, emptyLocal)).toEqual(unknown);
|
|
expect(await readIdleSleepSafety(emptyDb(), owned, () => now, "old-owner", emptyLocal)).toEqual(unknown);
|
|
expect(await readIdleSleepSafety(emptyDb(), () => ({ ...owned(), expiresAt: null }), () => now, ownerId, emptyLocal)).toEqual(unknown);
|
|
expect(await readIdleSleepSafety(emptyDb(), owned, () => now, ownerId, emptyLocal)).toEqual(none);
|
|
});
|
|
it.each(["present", "unknown"] as const)("retains local work reported as %s", async state => {
|
|
expect(await readIdleSleepSafety(emptyDb(), owned, () => now, ownerId, async () => state))
|
|
.toEqual({ version: 1, backgroundWork: state });
|
|
});
|
|
it("refuses sleep while accepted work remains in flight", async () => {
|
|
const done = beginIdleTrackedWork();
|
|
try { expect(await readIdleSleepSafety(emptyDb(), owned, () => now, ownerId, emptyLocal)).toEqual(unknown); }
|
|
finally { done(); }
|
|
expect(await readIdleSleepSafety(emptyDb(), owned, () => now, ownerId, emptyLocal)).toEqual(none);
|
|
});
|
|
it("invalidates a scan even when concurrent work finishes before the final check", async () => {
|
|
const inspect = async () => { const done = beginIdleTrackedWork(); done(); return "none" as const; };
|
|
expect(await readIdleSleepSafety(emptyDb(), owned, () => now, ownerId, inspect)).toEqual(unknown);
|
|
});
|
|
it("rechecks owner identity after disk inspection, including same-millisecond replacement", async () => {
|
|
let status = owned();
|
|
const inspect = async () => { status = { ...status, ownerId: "replacement" }; return "none" as const; };
|
|
expect(await readIdleSleepSafety(emptyDb(), () => status, () => now, ownerId, inspect)).toEqual(unknown);
|
|
});
|
|
it("rechecks expiry after disk inspection and hides inspection errors", async () => {
|
|
let clock = now;
|
|
expect(await readIdleSleepSafety(emptyDb(), owned, () => clock, ownerId, async () => {
|
|
clock += 60_000; return "none";
|
|
})).toEqual(unknown);
|
|
expect(await readIdleSleepSafety(emptyDb(), owned, () => now, ownerId, async () => { throw new Error("private spool path"); })).toEqual(unknown);
|
|
});
|
|
});
|
|
|
|
const support = await getEmbeddedPostgresTestSupport();
|
|
if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${support.reason}`);
|
|
(support.supported ? describe : describe.skip)("idle sleep durable work", () => {
|
|
let db: ReturnType<typeof createDb>;
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
|
|
beforeAll(async () => {
|
|
database = await startEmbeddedPostgresTestDatabase("idle-sleep-safety-");
|
|
db = createDb(database.connectionString);
|
|
}, 60_000);
|
|
afterEach(async () => { await db.execute(sql`TRUNCATE companies, plugins, environments CASCADE`); });
|
|
afterAll(async () => { await database?.cleanup(); });
|
|
|
|
const read = () => readIdleSleepSafety(db, owned, () => now, ownerId, emptyLocal);
|
|
const readScheduled = () => readIdleSleepSafety(db, owned, () => now, ownerId, emptyLocal, undefined, undefined, true);
|
|
async function seed() {
|
|
const companyId = randomUUID();
|
|
const agentId = randomUUID();
|
|
const runId = randomUUID();
|
|
await db.insert(companies).values({ id: companyId, name: "Idle test", issuePrefix: `T${companyId.slice(0, 6).toUpperCase()}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "On-demand agent", role: "engineer", status: "idle", adapterType: "process" });
|
|
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "on_demand", status: "succeeded" });
|
|
return { companyId, agentId, runId };
|
|
}
|
|
|
|
it.each(["true", "false"])("lets optional conversion retries sleep with package storage %s", async enabled => {
|
|
const previous = process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED;
|
|
process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = enabled;
|
|
try {
|
|
const { companyId } = await seed();
|
|
const [skill] = await db.insert(companySkills).values({ companyId, key: "legacy", slug: "legacy", name: "Legacy", markdown: "original" }).returning();
|
|
await db.insert(companySkillVersions).values({ companyId, companySkillId: skill!.id, revisionNumber: 1,
|
|
fileInventory: [{ path: "SKILL.md", kind: "skill", content: "original" }], migrationAt: new Date(Date.now() + 60 * 60_000) });
|
|
await db.insert(companySkillRepositorySnapshots).values({ companyId, contentHash: "a".repeat(64),
|
|
fileInventory: [{ path: "SKILL.md", kind: "skill", content: "original" }], migrationAt: new Date(0) });
|
|
expect(await read()).toEqual(none);
|
|
} finally {
|
|
if (previous === undefined) delete process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED;
|
|
else process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = previous;
|
|
}
|
|
});
|
|
|
|
it("retains committed budget stop intent until its exact generation is delivered", async () => {
|
|
const f = await seed();
|
|
const [policy] = await db.insert(budgetPolicies).values({ companyId: f.companyId, scopeType: "company", scopeId: f.companyId,
|
|
windowKind: "calendar_month_utc", enforcementVersion: 2, enforcementDeliveredVersion: 1 }).returning();
|
|
expect(await read()).toEqual(present);
|
|
await db.update(budgetPolicies).set({ enforcementDeliveredVersion: 2 }).where(eq(budgetPolicies.id, policy.id));
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it("retains company hold projection and lifecycle operations through admission drain", async () => {
|
|
const f = await seed();
|
|
await db.update(companies).set({ status: "paused" }).where(eq(companies.id, f.companyId));
|
|
expect(await read()).toEqual(present);
|
|
await db.update(agents).set({ status: "paused", lifecycleState: "pausing", lifecycleHolds: ["company_paused"] }).where(eq(agents.id, f.agentId));
|
|
expect(await read()).toEqual(present);
|
|
await db.update(agents).set({ lifecycleState: "paused" }).where(eq(agents.id, f.agentId));
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it("retains a committed budget Stop after terminal status until delivery is acknowledged", async () => {
|
|
const f = await seed();
|
|
const intent = { companyId: f.companyId, scopeType: "agent", scopeId: f.agentId, policyId: randomUUID(), version: 1 };
|
|
await db.update(heartbeatRuns).set({ status: "cancelled", resultJson: { startupCancellation: { budgetEnforcement: intent } } })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
expect(await read()).toEqual(present);
|
|
await db.update(heartbeatRuns).set({ resultJson: { startupCancellation: { budgetEnforcement: { ...intent, deliveredAt: new Date(now).toISOString() } } } })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
async function feedbackHistory(status: string, change: Partial<typeof feedbackExports.$inferInsert> = {}) {
|
|
const f = await seed();
|
|
const [issue] = await db.insert(issues).values({ companyId: f.companyId, title: "Finished fixture", status: "done" }).returning();
|
|
const [vote] = await db.insert(feedbackVotes).values({ companyId: f.companyId, issueId: issue.id,
|
|
targetType: "issue", targetId: issue.id, authorUserId: "fixture-user", vote: "up" }).returning();
|
|
await db.insert(feedbackExports).values({ companyId: f.companyId, feedbackVoteId: vote.id,
|
|
issueId: issue.id, authorUserId: "fixture-user", targetType: "issue", targetId: issue.id,
|
|
vote: "up", status, targetSummary: {}, ...change });
|
|
}
|
|
|
|
it("permits saved private feedback without sharing or deleting its history", async () => {
|
|
await feedbackHistory("local_only");
|
|
expect(await read()).toEqual(none);
|
|
expect(await db.select({ status: feedbackExports.status }).from(feedbackExports)).toEqual([{ status: "local_only" }]);
|
|
});
|
|
|
|
it("permits an acknowledged feedback upload and preserves its receipt", async () => {
|
|
await feedbackHistory("sent", { exportedAt: new Date(now), failureReason: null });
|
|
expect(await read()).toEqual(none);
|
|
expect(await db.select({ status: feedbackExports.status, exportedAt: feedbackExports.exportedAt }).from(feedbackExports))
|
|
.toEqual([{ status: "sent", exportedAt: new Date(now) }]);
|
|
});
|
|
|
|
it.each(["pending", "failed", "unfamiliar"])("keeps %s feedback work awake even with an old completion stamp", async status => {
|
|
await feedbackHistory(status, { exportedAt: new Date(now), failureReason: null });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it.each([
|
|
{ exportedAt: null, failureReason: null },
|
|
{ exportedAt: new Date(now), failureReason: "unconfirmed delivery" },
|
|
])("keeps inconsistent sent feedback awake: %j", async change => {
|
|
await feedbackHistory("sent", change);
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
async function scheduledRoutine(at = now + 600_000) {
|
|
const fixture = await seed();
|
|
const [routine] = await db.insert(routines).values({ companyId: fixture.companyId,
|
|
title: "Scheduled fixture", status: "active", assigneeAgentId: fixture.agentId }).returning();
|
|
const [trigger] = await db.insert(routineTriggers).values({ companyId: fixture.companyId,
|
|
routineId: routine!.id, kind: "schedule", cronExpression: "*/15 * * * *", timezone: "UTC",
|
|
nextRunAt: new Date(at), enabled: true, archived: false, setupPending: false }).returning();
|
|
return { ...fixture, routine: routine!, trigger: trigger! };
|
|
}
|
|
|
|
it("scheduled routines require the opt-in report and retain their durable definitions", async () => {
|
|
const f = await scheduledRoutine();
|
|
expect(await read()).toEqual(present);
|
|
const result = await readScheduled();
|
|
expect(result).toMatchObject({ version: 2, backgroundWork: "none", scheduledWake: {
|
|
kind: "routines", version: 1, nextWakeAt: new Date(now + 600_000).toISOString(),
|
|
} });
|
|
expect(result.scheduledWake?.generation).toMatch(/^[a-f0-9]{64}$/);
|
|
expect(await db.select().from(routineTriggers).where(eq(routineTriggers.id, f.trigger.id)))
|
|
.toMatchObject([{ nextRunAt: f.trigger.nextRunAt, cronExpression: f.trigger.cronExpression }]);
|
|
expect((await readScheduled()).scheduledWake).toEqual(result.scheduledWake);
|
|
});
|
|
it("scheduled routines cover all companies and invalidate a moved-earlier alarm", async () => {
|
|
await scheduledRoutine(now + 900_000);
|
|
const f = await scheduledRoutine(now + 600_000);
|
|
const first = await readScheduled();
|
|
expect(first.scheduledWake?.nextWakeAt).toBe(new Date(now + 600_000).toISOString());
|
|
await db.update(routineTriggers).set({ nextRunAt: new Date(now + 300_000) }).where(eq(routineTriggers.id, f.trigger.id));
|
|
const edited = await readScheduled();
|
|
expect(edited.scheduledWake?.nextWakeAt).toBe(new Date(now + 300_000).toISOString());
|
|
expect(edited.scheduledWake?.generation).not.toBe(first.scheduledWake?.generation);
|
|
});
|
|
it.each([
|
|
{ kind: "webhook" }, { setupPending: true }, { nextRunAt: null },
|
|
{ nextRunAt: new Date(now) }, { cronExpression: "bad" }, { timezone: "Unknown/Zone" },
|
|
])("scheduled routines retain unsupported or due work: %j", async change => {
|
|
const f = await scheduledRoutine();
|
|
await db.update(routineTriggers).set(change).where(eq(routineTriggers.id, f.trigger.id));
|
|
expect(await readScheduled()).toEqual(present);
|
|
});
|
|
it("scheduled routines cannot exempt external API credentials or an unowned hold", async () => {
|
|
const f = await scheduledRoutine();
|
|
expect(await readIdleSleepSafety(db, owned, () => now, "stale", emptyLocal, undefined, undefined, true)).toEqual(unknown);
|
|
await db.insert(agentApiKeys).values({ companyId: f.companyId, agentId: f.agentId,
|
|
name: "Fixture external automation", keyHash: "fixture-hash" });
|
|
expect(await readScheduled()).toEqual(present);
|
|
});
|
|
it("scheduled routines ignore disabled triggers and fail closed on unknown routine states", async () => {
|
|
const f = await scheduledRoutine();
|
|
await db.update(routineTriggers).set({ enabled: false }).where(eq(routineTriggers.id, f.trigger.id));
|
|
expect(await readScheduled()).toEqual(none);
|
|
await db.update(routines).set({ status: "unknown" }).where(eq(routines.id, f.routine.id));
|
|
expect(await readScheduled()).toEqual(present);
|
|
});
|
|
it("scheduled routines reject a trigger with a different company", async () => {
|
|
const f = await scheduledRoutine();
|
|
const other = await seed();
|
|
await db.update(routineTriggers).set({ companyId: other.companyId }).where(eq(routineTriggers.id, f.trigger.id));
|
|
expect(await readScheduled()).toEqual(present);
|
|
});
|
|
it.each(["checkpoint", "local inspection"])("keeps a routine awake when it becomes due during %s", async stage => {
|
|
await scheduledRoutine(now + 1000);
|
|
let clock = now;
|
|
let inspections = 0;
|
|
const inspect = async () => {
|
|
if (++inspections === 2 && stage === "local inspection") clock += 1000;
|
|
return "none" as const;
|
|
};
|
|
const checkpoint = stage === "checkpoint" ? async () => {
|
|
const done = beginIdleTrackedWork();
|
|
clock += 1000;
|
|
done();
|
|
return true;
|
|
} : undefined;
|
|
expect(await readIdleSleepSafety(db, owned, () => clock, ownerId, inspect, undefined, checkpoint, true)).toEqual(present);
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
});
|
|
|
|
it.each(["succeeded", "failed", "cancelled", "timed_out", "interrupted"])(
|
|
"permits a finished %s run with a consumed retry date without changing history", async status => {
|
|
const { runId } = await seed();
|
|
const scheduledRetryAt = new Date(now - 1000);
|
|
await db.update(heartbeatRuns).set({ status, finishedAt: new Date(now), scheduledRetryAt }).where(eq(heartbeatRuns.id, runId));
|
|
expect(await read()).toEqual(none);
|
|
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))
|
|
.toMatchObject([{ status, scheduledRetryAt }]);
|
|
},
|
|
);
|
|
|
|
it.each([
|
|
{ status: "scheduled_retry", finishedAt: new Date(now) },
|
|
{ status: "queued", finishedAt: new Date(now) },
|
|
{ status: "running", finishedAt: new Date(now) },
|
|
{ status: "failed", finishedAt: null },
|
|
{ status: "failed", finishedAt: new Date(now), costAccountingPending: true },
|
|
{ status: "failed", finishedAt: new Date(now - 2000) },
|
|
])("retains retry work without terminal consumption proof: %j", async change => {
|
|
const { runId } = await seed();
|
|
await db.update(heartbeatRuns).set({ scheduledRetryAt: new Date(now - 1000), ...change }).where(eq(heartbeatRuns.id, runId));
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("keeps a live retry child awake after its parent's schedule is consumed", async () => {
|
|
const { companyId, agentId, runId } = await seed();
|
|
await db.update(heartbeatRuns).set({ finishedAt: new Date(now), scheduledRetryAt: new Date(now - 1000) }).where(eq(heartbeatRuns.id, runId));
|
|
await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "on_demand", status: "scheduled_retry", retryOfRunId: runId, scheduledRetryAt: new Date(now + 1000) });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
async function seedQuestionDelivery(change: Partial<typeof issueQuestionResponseDeliveries.$inferInsert> = {}) {
|
|
const seeded = await seed();
|
|
await db.update(heartbeatRuns).set({ finishedAt: new Date(now) }).where(eq(heartbeatRuns.id, seeded.runId));
|
|
const issueId = randomUUID();
|
|
const interactionId = randomUUID();
|
|
await db.insert(issues).values({ id: issueId, companyId: seeded.companyId, title: "Finished question", status: "done" });
|
|
await db.insert(issueThreadInteractions).values({
|
|
id: interactionId, companyId: seeded.companyId, issueId,
|
|
kind: "ask_user_questions", status: "answered", payload: {},
|
|
});
|
|
await db.insert(issueQuestionResponseDeliveries).values({
|
|
companyId: seeded.companyId, issueId, interactionId, sourceRunId: seeded.runId,
|
|
targetRunId: seeded.runId, correlationId: randomUUID(), payloadSha256: "fixture",
|
|
status: "fallback_queued", deliveryMode: "wake_fallback", acknowledgedAt: new Date(now), ...change,
|
|
});
|
|
return { ...seeded, issueId, interactionId };
|
|
}
|
|
|
|
async function seedQuestionWake(
|
|
delivery: Awaited<ReturnType<typeof seedQuestionDelivery>>,
|
|
change: Partial<typeof agentWakeupRequests.$inferInsert> = {},
|
|
) {
|
|
await db.insert(agentWakeupRequests).values({
|
|
companyId: delivery.companyId, agentId: delivery.agentId, source: "automation",
|
|
idempotencyKey: `question-response:${delivery.interactionId}`,
|
|
status: "completed", finishedAt: new Date(now), ...change,
|
|
});
|
|
}
|
|
|
|
it.each(["pending", "delivering"])("keeps a %s question delivery awake", async status => {
|
|
await seedQuestionDelivery({ status, acknowledgedAt: null });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it.each(["delivered", "fallback_queued"])("permits acknowledged %s history after its target run finishes", async status => {
|
|
await seedQuestionDelivery({ status });
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it.each(["delivered", "fallback_queued"])("keeps an unacknowledged %s question delivery awake", async status => {
|
|
await seedQuestionDelivery({ status, acknowledgedAt: null });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it.each([
|
|
["queued", { status: "queued", finishedAt: null }],
|
|
["running", { status: "running", finishedAt: null }],
|
|
["unfinished", { finishedAt: null }],
|
|
["retry", { scheduledRetryAt: new Date(now + 1000) }],
|
|
["accounting", { costAccountingPending: true }],
|
|
] as const)("keeps an acknowledged question delivery awake with %s target evidence", async (_name, change) => {
|
|
const { runId } = await seedQuestionDelivery();
|
|
await db.update(heartbeatRuns).set(change).where(eq(heartbeatRuns.id, runId));
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("does not accept a foreign company's finished question target", async () => {
|
|
const foreign = await seedQuestionDelivery({ status: "failed", targetRunId: null });
|
|
await seedQuestionDelivery({ targetRunId: foreign.runId });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("keeps delivered history with a missing target awake", async () => {
|
|
await seedQuestionDelivery({ status: "delivered", targetRunId: null });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("keeps fallback history without a durable wake receipt awake", async () => {
|
|
await seedQuestionDelivery({ targetRunId: null });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it.each(["completed", "failed", "cancelled", "skipped", "timed_out"])(
|
|
"permits fallback history after its exact durable wake is %s and finished", async status => {
|
|
const delivery = await seedQuestionDelivery({ targetRunId: null });
|
|
await seedQuestionWake(delivery, { status });
|
|
expect(await read()).toEqual(none);
|
|
},
|
|
);
|
|
|
|
it.each(["queued", "deferred_issue_execution", "coalesced"])("keeps an unresolved %s question wake awake", async status => {
|
|
const delivery = await seedQuestionDelivery({ targetRunId: null });
|
|
await seedQuestionWake(delivery, { status, finishedAt: null });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("keeps a terminal question wake without completion evidence awake", async () => {
|
|
const delivery = await seedQuestionDelivery({ targetRunId: null });
|
|
await seedQuestionWake(delivery, { finishedAt: null });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("permits a deferred question fallback after its coalesced continuation finishes", async () => {
|
|
const delivery = await seedQuestionDelivery({ targetRunId: null });
|
|
await seedQuestionWake(delivery, { status: "deferred_issue_execution", finishedAt: null });
|
|
expect(await read()).toEqual(present);
|
|
await db.update(agentWakeupRequests).set({
|
|
status: "coalesced", runId: delivery.runId, finishedAt: new Date(now),
|
|
}).where(eq(agentWakeupRequests.companyId, delivery.companyId));
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it.each(["missing_run", "wrong_company", "wrong_agent", "unfinished", "retry", "accounting"])(
|
|
"keeps a coalesced question fallback awake with %s linked-run evidence", async kind => {
|
|
const delivery = await seedQuestionDelivery({ targetRunId: null });
|
|
let runId = delivery.runId;
|
|
if (kind === "missing_run") runId = randomUUID();
|
|
if (kind === "wrong_company") runId = (await seedQuestionDelivery({ status: "failed", targetRunId: null })).runId;
|
|
if (kind === "wrong_agent") {
|
|
const otherAgent = randomUUID();
|
|
await db.insert(agents).values({ id: otherAgent, companyId: delivery.companyId, name: "Other", role: "engineer", status: "idle", adapterType: "process" });
|
|
await db.update(heartbeatRuns).set({ agentId: otherAgent }).where(eq(heartbeatRuns.id, runId));
|
|
}
|
|
if (kind === "unfinished") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, runId));
|
|
if (kind === "retry") await db.update(heartbeatRuns).set({ scheduledRetryAt: new Date(now + 1000) }).where(eq(heartbeatRuns.id, runId));
|
|
if (kind === "accounting") await db.update(heartbeatRuns).set({ costAccountingPending: true }).where(eq(heartbeatRuns.id, runId));
|
|
await seedQuestionWake(delivery, { status: "coalesced", runId });
|
|
expect(await read()).toEqual(present);
|
|
},
|
|
);
|
|
|
|
it("does not accept another interaction's or company's question wake receipt", async () => {
|
|
const delivery = await seedQuestionDelivery({ targetRunId: null });
|
|
await seedQuestionWake(delivery, { idempotencyKey: `question-response:${randomUUID()}` });
|
|
const foreign = await seedQuestionDelivery({ status: "failed", targetRunId: null });
|
|
await seedQuestionWake(delivery, { companyId: foreign.companyId, agentId: foreign.agentId });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("permits failed question-delivery history without a pending target", async () => {
|
|
await seedQuestionDelivery({ status: "failed", targetRunId: null, acknowledgedAt: null });
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it("still blocks other work after a question handoff finishes", async () => {
|
|
const { issueId } = await seedQuestionDelivery();
|
|
await db.update(issues).set({ status: "todo" }).where(eq(issues.id, issueId));
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("requires a current quiet owner even with completed question history", async () => {
|
|
await seedQuestionDelivery();
|
|
expect(await readIdleSleepSafety(db, owned, () => now, "stale-owner", emptyLocal)).toEqual(unknown);
|
|
expect(await readIdleSleepSafety(db, owned, () => now + 60_001, ownerId, emptyLocal)).toEqual(unknown);
|
|
});
|
|
|
|
it("keeps saved agent files awake until deferred cleanup completes", async () => {
|
|
const { companyId, agentId, runId } = await seed();
|
|
await db.insert(agentInstructionWorkingCopies).values({
|
|
runId, companyId, agentId, responsibleUserId: "test-user", entryFile: "AGENTS.md",
|
|
baseHash: "test-hash", localRoot: "/tmp/idle-copy", executionRoot: "/tmp/idle-copy",
|
|
location: "local", state: "saved", processStoppedAt: new Date(now),
|
|
receipt: { schema: "paperclip.agent-files.v1", cleanupPending: true },
|
|
});
|
|
expect(await read()).toEqual(present);
|
|
await db.update(agentInstructionWorkingCopies).set({ receipt: { schema: "paperclip.agent-files.v1" } })
|
|
.where(eq(agentInstructionWorkingCopies.runId, runId));
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it("authorizes an empty database and completed history under a quiet owned hold", async () => {
|
|
expect(await read()).toEqual(none);
|
|
const { companyId } = await seed();
|
|
await db.insert(issues).values({ companyId, title: "Finished work", status: "done" });
|
|
await db.insert(routines).values({ companyId, title: "Paused schedule", status: "paused" });
|
|
await db.insert(environmentLeases).values({ companyId, status: "released" });
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it("blocks a saved watchdog on a completed issue before its first review starts", async () => {
|
|
const { companyId, agentId } = await seed();
|
|
const [issue] = await db.insert(issues).values({ companyId, title: "Finished work", status: "done" }).returning();
|
|
expect(await read()).toEqual(none);
|
|
const [watchdog] = await db.insert(issueWatchdogs).values({
|
|
companyId, issueId: issue!.id, watchdogAgentId: agentId, status: "active",
|
|
}).returning();
|
|
expect(watchdog!.watchdogIssueId).toBeNull();
|
|
expect(watchdog!.lastTriggeredAt).toBeNull();
|
|
expect(await read()).toEqual(present);
|
|
await db.update(issueWatchdogs).set({ status: "disabled" }).where(eq(issueWatchdogs.id, watchdog!.id));
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it.each([
|
|
["queued run", { status: "queued" }],
|
|
["orphan running run", { status: "running" }],
|
|
["future retry", { scheduledRetryAt: new Date(now + 86_400_000) }],
|
|
["unsettled accounting", { costAccountingPending: true }],
|
|
] as const)("blocks %s even when process counters are zero", async (_name, change) => {
|
|
const { runId } = await seed();
|
|
await db.update(heartbeatRuns).set(change).where(eq(heartbeatRuns.id, runId));
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("reports an enabled heartbeat timer and permits sleep after it is disabled", async () => {
|
|
const { agentId } = await seed();
|
|
await db.update(agents).set({ runtimeConfig: { heartbeat: { enabled: true, intervalSec: 86_400 } } }).where(eq(agents.id, agentId));
|
|
expect(await read()).toEqual(present);
|
|
await db.update(agents).set({ runtimeConfig: { heartbeat: { enabled: false, intervalSec: 86_400 } } }).where(eq(agents.id, agentId));
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it("blocks a durable deferred wake", async () => {
|
|
const { companyId, agentId } = await seed();
|
|
await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "assignment", status: "deferred_issue_execution" });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("allows consumed coalesced wakes only after their linked run finishes", async () => {
|
|
const { companyId, agentId, runId } = await seed();
|
|
await db.update(heartbeatRuns).set({ status: "running" }).where(eq(heartbeatRuns.id, runId));
|
|
await db.insert(agentWakeupRequests).values({
|
|
companyId, agentId, source: "assignment", status: "coalesced", runId, finishedAt: new Date(now),
|
|
});
|
|
expect(await read()).toEqual(present);
|
|
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date(now) })
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it.each(["unfinished", "missing_run", "wrong_company", "wrong_agent", "unfinished_run", "retry", "accounting"])(
|
|
"keeps a coalesced wake awake with %s evidence", async kind => {
|
|
const { companyId, agentId, runId } = await seed();
|
|
await db.update(heartbeatRuns).set({ finishedAt: new Date(now) }).where(eq(heartbeatRuns.id, runId));
|
|
let linkedRunId = runId;
|
|
if (kind === "missing_run") linkedRunId = randomUUID();
|
|
if (kind === "wrong_company") {
|
|
linkedRunId = (await seed()).runId;
|
|
await db.update(heartbeatRuns).set({ finishedAt: new Date(now) }).where(eq(heartbeatRuns.id, linkedRunId));
|
|
}
|
|
if (kind === "wrong_agent") {
|
|
const otherAgentId = randomUUID();
|
|
await db.insert(agents).values({ id: otherAgentId, companyId, name: "Other agent", role: "engineer", status: "idle", adapterType: "process" });
|
|
await db.update(heartbeatRuns).set({ agentId: otherAgentId }).where(eq(heartbeatRuns.id, runId));
|
|
}
|
|
if (kind === "unfinished_run") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, runId));
|
|
if (kind === "retry") await db.update(heartbeatRuns).set({ scheduledRetryAt: new Date(now + 1000) }).where(eq(heartbeatRuns.id, runId));
|
|
if (kind === "accounting") await db.update(heartbeatRuns).set({ costAccountingPending: true }).where(eq(heartbeatRuns.id, runId));
|
|
await db.insert(agentWakeupRequests).values({
|
|
companyId, agentId, source: "assignment", status: "coalesced", runId: linkedRunId,
|
|
finishedAt: kind === "unfinished" ? null : new Date(now),
|
|
});
|
|
expect(await read()).toEqual(present);
|
|
},
|
|
);
|
|
|
|
it("blocks an active routine without waiting for its next due time", async () => {
|
|
const { companyId } = await seed();
|
|
await db.insert(routines).values({ companyId, title: "Tomorrow", status: "active" });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("blocks a remote resource pending cleanup", async () => {
|
|
const { companyId } = await seed();
|
|
await db.insert(environmentLeases).values({ companyId, status: "pending_cleanup" });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
async function seedAiConnection() {
|
|
const seeded = await seed();
|
|
const [application] = await db.insert(toolApplications).values({
|
|
companyId: seeded.companyId, name: "AI fixture", type: "rest_api",
|
|
}).returning();
|
|
const [connection] = await db.insert(toolConnections).values({
|
|
companyId: seeded.companyId, applicationId: application!.id,
|
|
name: "Saved AI login", uid: randomUUID(), connectionPurpose: "ai",
|
|
transport: "runtime_auth", status: "active", enabled: true,
|
|
}).returning();
|
|
return { ...seeded, connection: connection! };
|
|
}
|
|
|
|
it("allows saved AI credentials while preserving their connection", async () => {
|
|
const { connection } = await seedAiConnection();
|
|
expect(await read()).toEqual(none);
|
|
expect(await db.select().from(toolConnections).where(eq(toolConnections.id, connection.id)))
|
|
.toMatchObject([{ status: "active", enabled: true }]);
|
|
});
|
|
|
|
it.each(["draft", "active_login", "pending_run", "accepted_refresh"])(
|
|
"keeps AI connections awake for %s work", async kind => {
|
|
const { companyId, runId, connection } = await seedAiConnection();
|
|
if (kind === "draft") await db.update(toolConnections).set({ status: "draft" }).where(eq(toolConnections.id, connection.id));
|
|
if (kind === "pending_run") await db.update(heartbeatRuns).set({ status: "queued" }).where(eq(heartbeatRuns.id, runId));
|
|
if (kind === "active_login") {
|
|
const [environment] = await db.insert(environments).values({ name: "Active AI login" }).returning();
|
|
await db.insert(adapterAuthSessions).values({
|
|
companyId, environmentId: environment!.id, connectionId: connection.id,
|
|
adapterType: "codex_local", startedByUserId: "fixture-user",
|
|
publicSessionId: randomUUID(), status: "waiting_for_user",
|
|
});
|
|
}
|
|
const finish = kind === "accepted_refresh" ? beginIdleTrackedWork() : undefined;
|
|
try { expect(await read()).toEqual(kind === "accepted_refresh" ? unknown : present); }
|
|
finally { finish?.(); }
|
|
},
|
|
);
|
|
|
|
it("allows saved outbound MCP connections alongside inert AI credentials", async () => {
|
|
const { companyId, connection } = await seedAiConnection();
|
|
await db.insert(toolConnections).values({
|
|
companyId, applicationId: connection.applicationId, name: "Remote tools", uid: randomUUID(),
|
|
connectionPurpose: "tool", transport: "mcp_remote", status: "active", enabled: true,
|
|
});
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
async function seedFinishedFailedLease() {
|
|
const seeded = await seed();
|
|
await db.update(heartbeatRuns).set({ status: "failed", finishedAt: new Date(now) })
|
|
.where(eq(heartbeatRuns.id, seeded.runId));
|
|
const [environment] = await db.insert(environments).values({ name: "Failed turn environment", driver: "sandbox" }).returning();
|
|
const leaseId = randomUUID();
|
|
const receipt = {
|
|
schema: "paperclip.remote-termination.v1", companyId: seeded.companyId,
|
|
runId: seeded.runId, leaseId, provider: "plugin:demo:sandbox",
|
|
providerLeaseId: "fixture-resource", state: "destroyed", confirmedAt: new Date(now).toISOString(),
|
|
};
|
|
const [lease] = await db.insert(environmentLeases).values({
|
|
id: leaseId, companyId: seeded.companyId, environmentId: environment!.id,
|
|
heartbeatRunId: seeded.runId, status: "failed", leasePolicy: "ephemeral",
|
|
provider: receipt.provider, providerLeaseId: receipt.providerLeaseId,
|
|
acquiredAt: new Date(now - 1000), releasedAt: new Date(now), cleanupStatus: "success",
|
|
metadata: { remoteExecutionTermination: receipt },
|
|
}).returning();
|
|
return { ...seeded, environment: environment!, lease: lease!, receipt };
|
|
}
|
|
|
|
it.each(["destroyed", "stopped"])("allows failed lease history after exact provider %s proof", async state => {
|
|
const { lease, receipt } = await seedFinishedFailedLease();
|
|
await db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: { ...receipt, state } } })
|
|
.where(eq(environmentLeases.id, lease.id));
|
|
expect(await read()).toEqual(none);
|
|
expect(await db.select().from(environmentLeases).where(eq(environmentLeases.id, lease.id)))
|
|
.toMatchObject([{ status: "failed", cleanupStatus: "success" }]);
|
|
});
|
|
|
|
it("permits consumed retry history only with exact failed-lease termination proof", async () => {
|
|
const { lease, runId } = await seedFinishedFailedLease();
|
|
await db.update(heartbeatRuns).set({ scheduledRetryAt: new Date(now - 1000) }).where(eq(heartbeatRuns.id, runId));
|
|
expect(await read()).toEqual(none);
|
|
await db.update(environmentLeases).set({ metadata: {} }).where(eq(environmentLeases.id, lease.id));
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it.each([
|
|
"cleanup_failed", "cleanup_unknown", "unreleased", "retained_policy", "reusable_policy",
|
|
"active", "pending_cleanup", "missing_receipt", "wrong_company", "wrong_run", "wrong_lease",
|
|
"wrong_provider", "wrong_resource", "not_stopped", "missing_run", "unfinished_run",
|
|
"retry", "accounting", "cleanup_in_flight", "pending_export", "pending_retain",
|
|
])("retains failed leases with %s evidence", async kind => {
|
|
const { lease, receipt, runId } = await seedFinishedFailedLease();
|
|
const change: Partial<typeof environmentLeases.$inferInsert> = {};
|
|
if (kind === "cleanup_failed") change.cleanupStatus = "failed";
|
|
if (kind === "cleanup_unknown") change.cleanupStatus = null;
|
|
if (kind === "unreleased") change.releasedAt = null;
|
|
if (kind === "retained_policy") change.leasePolicy = "retain_on_failure";
|
|
if (kind === "reusable_policy") change.leasePolicy = "reuse_by_environment";
|
|
if (kind === "active" || kind === "pending_cleanup") change.status = kind;
|
|
if (kind === "missing_receipt") change.metadata = {};
|
|
const fields: Record<string, string> = { wrong_company: "companyId", wrong_run: "runId", wrong_lease: "leaseId", wrong_provider: "provider", wrong_resource: "providerLeaseId", not_stopped: "state" };
|
|
if (fields[kind]) change.metadata = { remoteExecutionTermination: { ...receipt, [fields[kind]]: "wrong" } };
|
|
if (kind === "missing_run") change.heartbeatRunId = null;
|
|
if (kind === "unfinished_run") await db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, runId));
|
|
if (kind === "retry") await db.update(heartbeatRuns).set({ scheduledRetryAt: new Date(now + 1000) }).where(eq(heartbeatRuns.id, runId));
|
|
if (kind === "accounting") await db.update(heartbeatRuns).set({ costAccountingPending: true }).where(eq(heartbeatRuns.id, runId));
|
|
if (kind === "cleanup_in_flight") change.metadata = { remoteExecutionTermination: receipt, pendingCleanupInFlight: true };
|
|
if (kind === "pending_export") change.metadata = { remoteExecutionTermination: receipt, nativeWorkspaceExportResume: {} };
|
|
if (kind === "pending_retain") change.metadata = { remoteExecutionTermination: receipt, sandboxStopAndRetain: {} };
|
|
if (Object.keys(change).length) await db.update(environmentLeases).set(change).where(eq(environmentLeases.id, lease.id));
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("uses the same failed-lease cleanup proof for finished login history", async () => {
|
|
const { companyId, environment, lease } = await seedFinishedFailedLease();
|
|
await db.insert(adapterAuthSessions).values({
|
|
companyId, environmentId: environment.id, adapterType: "codex_local", startedByUserId: "fixture-user",
|
|
publicSessionId: randomUUID(), providerLeaseId: lease.providerLeaseId,
|
|
status: "authenticated", finishedAt: new Date(now),
|
|
});
|
|
expect(await read()).toEqual(none);
|
|
await db.update(environmentLeases).set({ metadata: {} }).where(eq(environmentLeases.id, lease.id));
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("reports a login sandbox awaiting cleanup", async () => {
|
|
const { companyId } = await seed();
|
|
const [environment] = await db.insert(environments).values({ name: "Login fixture" }).returning();
|
|
await db.insert(adapterAuthSessions).values({
|
|
companyId, environmentId: environment!.id, adapterType: "codex_local",
|
|
startedByUserId: "fixture-user", publicSessionId: "fixture-session", status: "cleanup_pending",
|
|
});
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
async function seedFinishedLogin() {
|
|
const { companyId } = await seed();
|
|
const [environment] = await db.insert(environments).values({ name: "Finished login fixture" }).returning();
|
|
const [lease] = await db.insert(environmentLeases).values({
|
|
companyId, environmentId: environment!.id, providerLeaseId: "fixture-login-resource",
|
|
status: "released", cleanupStatus: "success", releasedAt: new Date(now),
|
|
}).returning();
|
|
const [session] = await db.insert(adapterAuthSessions).values({
|
|
companyId, environmentId: environment!.id, adapterType: "codex_local",
|
|
startedByUserId: "fixture-user", publicSessionId: "finished-fixture-session",
|
|
status: "authenticated", finishedAt: new Date(now), providerLeaseId: lease!.providerLeaseId,
|
|
}).returning();
|
|
return { companyId, environment: environment!, lease: lease!, session: session! };
|
|
}
|
|
|
|
it.each(["authenticated", "completed", "failed", "timed_out", "cancelled"] as const)(
|
|
"permits finished %s login history after confirmed provider cleanup", async status => {
|
|
const { session } = await seedFinishedLogin();
|
|
await db.update(adapterAuthSessions).set({ status }).where(eq(adapterAuthSessions.id, session.id));
|
|
expect(await read()).toEqual(none);
|
|
},
|
|
);
|
|
|
|
it.each(["canonical", "uppercase"])("accepts a %s cleanup reference to the internal lease id", async format => {
|
|
const { session, lease } = await seedFinishedLogin();
|
|
await db.update(adapterAuthSessions).set({ providerLeaseId: format === "uppercase" ? lease.id.toUpperCase() : lease.id })
|
|
.where(eq(adapterAuthSessions.id, session.id));
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it.each(["fixture-provider-id", "00000000-0000-0000-0000-00000000000z", randomUUID()])(
|
|
"accepts cleaned external provider reference %s without an unsafe UUID cast", async providerLeaseId => {
|
|
const { session, lease } = await seedFinishedLogin();
|
|
await db.update(environmentLeases).set({ providerLeaseId }).where(eq(environmentLeases.id, lease.id));
|
|
await db.update(adapterAuthSessions).set({ providerLeaseId }).where(eq(adapterAuthSessions.id, session.id));
|
|
expect(await read()).toEqual(none);
|
|
},
|
|
);
|
|
|
|
it("keeps a UUID-shaped external reference awake when an internal lease conflicts", async () => {
|
|
const { companyId, environment, session, lease } = await seedFinishedLogin();
|
|
const conflictingId = randomUUID();
|
|
await db.update(environmentLeases).set({ providerLeaseId: conflictingId }).where(eq(environmentLeases.id, lease.id));
|
|
await db.update(adapterAuthSessions).set({ providerLeaseId: conflictingId }).where(eq(adapterAuthSessions.id, session.id));
|
|
await db.insert(environmentLeases).values({
|
|
id: conflictingId, companyId, environmentId: environment.id,
|
|
status: "released", cleanupStatus: "failed", releasedAt: new Date(now),
|
|
});
|
|
expect(await read()).toEqual(present);
|
|
await db.update(environmentLeases).set({ cleanupStatus: "success" }).where(eq(environmentLeases.id, conflictingId));
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it("permits terminal login history without a provider resource", async () => {
|
|
const { session } = await seedFinishedLogin();
|
|
await db.update(adapterAuthSessions).set({ providerLeaseId: null }).where(eq(adapterAuthSessions.id, session.id));
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it.each(["cleanup_pending", "starting", "promoting", "awaiting_code", "submitting", "stored"] as const)(
|
|
"retains %s login work even with an old successful cleanup receipt", async status => {
|
|
const { session } = await seedFinishedLogin();
|
|
await db.update(adapterAuthSessions).set({ status }).where(eq(adapterAuthSessions.id, session.id));
|
|
expect(await read()).toEqual(present);
|
|
},
|
|
);
|
|
|
|
it.each(["unfinished", "promotion_claim", "missing_lease", "wrong_company", "wrong_environment", "cleanup_failed", "cleanup_unknown", "unreleased", "conflicting_lease"])(
|
|
"retains terminal login history with %s cleanup evidence", async kind => {
|
|
const { companyId, session, lease, environment } = await seedFinishedLogin();
|
|
if (kind === "unfinished") await db.update(adapterAuthSessions).set({ finishedAt: null }).where(eq(adapterAuthSessions.id, session.id));
|
|
if (kind === "promotion_claim") await db.update(adapterAuthSessions).set({ promotionExpiresAt: new Date(now + 1000) }).where(eq(adapterAuthSessions.id, session.id));
|
|
if (kind === "missing_lease") await db.delete(environmentLeases).where(eq(environmentLeases.id, lease.id));
|
|
if (kind === "wrong_company") await db.update(environmentLeases).set({ companyId: (await seed()).companyId }).where(eq(environmentLeases.id, lease.id));
|
|
if (kind === "wrong_environment") {
|
|
const [other] = await db.insert(environments).values({ name: "Other environment", driver: "sandbox" }).returning();
|
|
await db.update(environmentLeases).set({ environmentId: other!.id }).where(eq(environmentLeases.id, lease.id));
|
|
}
|
|
if (kind === "cleanup_failed") await db.update(environmentLeases).set({ cleanupStatus: "failed" }).where(eq(environmentLeases.id, lease.id));
|
|
if (kind === "cleanup_unknown") await db.update(environmentLeases).set({ cleanupStatus: null }).where(eq(environmentLeases.id, lease.id));
|
|
if (kind === "unreleased") await db.update(environmentLeases).set({ releasedAt: null }).where(eq(environmentLeases.id, lease.id));
|
|
if (kind === "conflicting_lease") await db.insert(environmentLeases).values({
|
|
companyId, environmentId: environment.id, providerLeaseId: lease.providerLeaseId,
|
|
status: "released", cleanupStatus: "failed", releasedAt: new Date(now),
|
|
});
|
|
expect(await read()).toEqual(present);
|
|
},
|
|
);
|
|
|
|
it("reports a pending secret proposal with future expiry", async () => {
|
|
const { companyId, agentId, runId } = await seed();
|
|
await db.insert(companySecretProposals).values({
|
|
companyId, kind: "secret", proposedName: "Fixture", proposedKey: "FIXTURE",
|
|
justification: "Test pending proposal", proposedByAgentId: agentId, originRunId: runId,
|
|
expiresAt: new Date(now + 86_400_000),
|
|
});
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
async function seedRetainedWorkspace(change: Partial<typeof executionWorkspaces.$inferInsert> = {}) {
|
|
const { companyId } = await seed();
|
|
const [project] = await db.insert(projects).values({ companyId, name: "Fixture project" }).returning();
|
|
const [workspace] = await db.insert(executionWorkspaces).values({
|
|
companyId, projectId: project!.id, name: "Fixture workspace", mode: "isolated_workspace",
|
|
strategyType: "git_worktree", status: "active", providerType: "git_worktree",
|
|
cwd: "/tmp/fixture-retained-worktree", providerRef: "/tmp/fixture-retained-worktree",
|
|
...change,
|
|
}).returning();
|
|
return workspace!;
|
|
}
|
|
|
|
it.each(["local_fs", "git_worktree"].flatMap(providerType =>
|
|
["active", "idle", "in_review", "archived"].map(status => ({ providerType, status }))))(
|
|
"permits a passive local directory without removing retained data: %j", async ({ providerType, status }) => {
|
|
const metadata = { config: {
|
|
environmentId: randomUUID(), provisionCommand: null, runtimeProvisionCommand: null,
|
|
teardownCommand: null, cleanupCommand: null, workspaceRuntime: null,
|
|
desiredState: null, serviceStates: null,
|
|
}, retainedFixture: true };
|
|
const strategyType = providerType === "local_fs" ? "project_primary" : "git_worktree";
|
|
const workspace = await seedRetainedWorkspace({ providerType, strategyType, status, metadata, cleanupEligibleAt: new Date(Date.UTC(2099, 0, 1)) });
|
|
expect(await read()).toEqual(none);
|
|
expect(await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace.id)))
|
|
.toMatchObject([{ status, metadata, cwd: workspace.cwd, cleanupEligibleAt: workspace.cleanupEligibleAt }]);
|
|
},
|
|
);
|
|
|
|
it.each([null, {}, { config: null }, { config: {} }, { config: { desiredState: "stopped", serviceStates: {} } }])(
|
|
"permits passive local configuration: %j", async metadata => {
|
|
await seedRetainedWorkspace({ metadata });
|
|
expect(await read()).toEqual(none);
|
|
},
|
|
);
|
|
|
|
it.each([
|
|
{ providerType: "cloud_sandbox" }, { providerType: "future_provider" },
|
|
{ strategyType: "adapter_managed" }, { strategyType: "future_strategy" },
|
|
{ mode: "future_mode" },
|
|
{ providerRef: "another-resource" }, { status: "cleanup_failed" }, { status: "future_state" },
|
|
{ cleanupEligibleAt: new Date(now) }, { metadata: { reopenPendingConsumption: true } },
|
|
{ metadata: { reopenPendingConsumption: "unknown" } },
|
|
{ metadata: { config: [] } }, { metadata: { config: "malformed" } },
|
|
{ metadata: { config: { workspaceRuntime: {} } } },
|
|
{ metadata: { config: { workspaceRuntime: { services: [] } } } },
|
|
{ metadata: { config: { desiredState: "running" } } },
|
|
{ metadata: { config: { desiredState: "manual" } } },
|
|
{ metadata: { config: { serviceStates: { web: "stopped" } } } },
|
|
{ metadata: { config: { provisionCommand: "echo fixture" } } },
|
|
{ metadata: { config: { runtimeProvisionCommand: "echo fixture" } } },
|
|
{ metadata: { config: { teardownCommand: "echo fixture" } } },
|
|
{ metadata: { config: { cleanupCommand: "echo fixture" } } },
|
|
{ metadata: { config: { futureWorker: true } } },
|
|
])("retains a directory with work or unfamiliar state: %j", async change => {
|
|
await seedRetainedWorkspace(change);
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("fails closed for malformed directory metadata", async () => {
|
|
const workspace = await seedRetainedWorkspace();
|
|
await db.execute(sql`UPDATE execution_workspaces SET metadata = '[]'::jsonb WHERE id = ${workspace.id}`);
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("retains a shared directory that can inherit project runtime configuration", async () => {
|
|
const workspace = await seedRetainedWorkspace();
|
|
const [projectWorkspace] = await db.insert(projectWorkspaces).values({
|
|
companyId: workspace.companyId, projectId: workspace.projectId, name: "Shared fixture",
|
|
metadata: { runtimeConfig: { workspaceRuntime: { services: [] } } },
|
|
}).returning();
|
|
await db.update(executionWorkspaces).set({ mode: "shared_workspace", projectWorkspaceId: projectWorkspace!.id })
|
|
.where(eq(executionWorkspaces.id, workspace.id));
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("retains a local directory with an unfinished operation", async () => {
|
|
const workspace = await seedRetainedWorkspace();
|
|
await db.insert(workspaceOperations).values({ companyId: workspace.companyId, executionWorkspaceId: workspace.id, phase: "cleanup", status: "running" });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("retains a local directory with a runtime owner", async () => {
|
|
const workspace = await seedRetainedWorkspace();
|
|
await db.insert(executionWorkspaceRuntimeLeases).values({
|
|
companyId: workspace.companyId, executionWorkspaceId: workspace.id,
|
|
ownerKey: "fixture-owner", lastAction: "use", expiresAt: new Date(Date.UTC(2099, 0, 1)),
|
|
});
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("blocks externally usable agent keys until revoked", async () => {
|
|
const { companyId, agentId } = await seed();
|
|
const [key] = await db.insert(agentApiKeys).values({ companyId, agentId, name: "Fixture", keyHash: "fixture-not-a-key" }).returning();
|
|
expect(await read()).toEqual(present);
|
|
await db.update(agentApiKeys).set({ revokedAt: new Date(now) }).where(eq(agentApiKeys.id, key!.id));
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it("checks every company in the instance", async () => {
|
|
await seed();
|
|
const otherId = randomUUID();
|
|
await db.insert(companies).values({ id: otherId, name: "Other company", issuePrefix: "OTHER" });
|
|
await db.insert(issues).values({ companyId: otherId, title: "Pending work", status: "todo" });
|
|
expect(await read()).toEqual(present);
|
|
});
|
|
|
|
it("blocks enabled plugins even without declared jobs or webhooks", async () => {
|
|
await db.execute(sql`INSERT INTO plugins (plugin_key, package_name, version, manifest_json)
|
|
VALUES ('demo.on-demand', 'demo-plugin', '1.0.0', '{}'::jsonb)`);
|
|
expect(await read()).toEqual(present);
|
|
await db.execute(sql`UPDATE plugins SET status = 'disabled'`);
|
|
expect(await read()).toEqual(none);
|
|
});
|
|
|
|
it("accepts only exact worker drain receipts and still checks durable work", async () => {
|
|
const [{ id }] = await db.execute<{ id: string }>(sql`INSERT INTO plugins (plugin_key, package_name, version, manifest_json)
|
|
VALUES ('demo.idle', 'demo-idle', '1.0.0', '{}'::jsonb) RETURNING id`);
|
|
const inspect = vi.fn(async () => ({ backgroundWork: "none" as const, pluginIds: [id!] }));
|
|
const scan = () => readIdleSleepSafety(db, owned, () => now, ownerId, emptyLocal, inspect);
|
|
expect(await scan()).toEqual(none);
|
|
expect(inspect).toHaveBeenCalledWith({ ownerId, expiresAt: owned().expiresAt!.getTime() });
|
|
await db.execute(sql`INSERT INTO plugins (plugin_key, package_name, version, manifest_json)
|
|
VALUES ('demo.unknown', 'demo-unknown', '1.0.0', '{}'::jsonb)`);
|
|
expect(await scan()).toEqual(present);
|
|
await db.execute(sql`UPDATE plugins SET status = 'disabled' WHERE plugin_key = 'demo.unknown'`);
|
|
const { companyId } = await seed();
|
|
await db.insert(issues).values({ companyId, title: "Still pending", status: "todo" });
|
|
expect(await scan()).toEqual(present);
|
|
});
|
|
|
|
it("fails closed when the installed schema is older than the report", async () => {
|
|
await db.execute(sql`ALTER TABLE heartbeat_runs RENAME COLUMN cost_accounting_pending TO hidden_pending`);
|
|
try { expect(await read()).toEqual(unknown); }
|
|
finally { await db.execute(sql`ALTER TABLE heartbeat_runs RENAME COLUMN hidden_pending TO cost_accounting_pending`); }
|
|
});
|
|
});
|