mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
21640b6d514a5569cd76b2d3a3bb9e13e2a623d2
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
ccc63ea458 |
fix(server): prove completed history before idle sleep (#18646)
Reviewed and verified at
|
||
|
Nicky LeachandPaperclip
|
a9b18e1cbe |
fix(runner): preserve cleanup ownership across startup failures (#18658)
## Thinking Path > - Paperclip controls which run owns an agent task. > - A plugin can continue accepted work after Core stops waiting. > - A late task-creation reply can therefore arrive after startup cancellation. > - Core must keep that work owned until its resource identity and cleanup outcome are known. > - This PR supplies the cleanup and recovery support before shared startup is enabled. > - The dependent activation PR can then reuse transport data within a bounded startup context. ## Linked Issues or Issue Description Refs #18405. This is the independently mergeable prerequisite for that stacked activation PR. **What existing behavior does this improve?** Cleanup of explicitly recorded startup work that outlives its caller. **Current behavior** Caller timeout, worker exit, or failed cleanup-intent persistence can leave work without a complete recovery path. **Proposed behavior** Retain the cleanup obligation, use existing Stop and recovery queues, and release ownership only after the exact task and accepted work have stopped. Ordinary calls keep their existing startup payloads and budgets. This PR does not create startup contexts or advertise transport reuse. **Reason and benefit** Separate the failure-handling contract from activation so each can be reviewed on its own. This support PR has **no standalone TTFT speedup**. ## What Changed - Add the startup cleanup marker, exact ownership checks, verified completion, and acknowledgement helpers. - Connect recorded cleanup obligations to existing Stop, deadline reconciliation, restart recovery, and retry scheduling. - Retain typed cleanup responsibility when a database write fails. Keep the nonterminal run from being treated as ordinary completed cleanup. - Observe late acquisition results and save resource identities before retiring them. Retain real host writes and received results across worker exit or persistence failure. - Queue abandoned replacement tasks through the existing creation journal and cleanup machinery. Require the replacement's own Stop proof. - Accept an explicit startup context in the internal bridge for lifetime tracking. Automatic context creation, marker production, capability advertisement, transport capture, and executor activation belong to #18405. - Add PostgreSQL and worker IPC regressions, including legacy payload assertions. ## Verification Initial split source `8fb482d8be9e70101b0cfdd0830f41cf5b8e3cf4` had tree `95508bd1859e991670cad8afd4de8d2167009ac5`. **190 distinct targeted tests pass**, along with server typechecking. The selected checks cover real PostgreSQL cleanup/ownership, worker IPC, the bridge and creation journal, ordinary/distributed Stop, restart and deadline recovery, idle-sleep behavior, and the unchanged legacy warm-task suite. Independent recovery and worker reviews found no blocking issue. The first correction, `b6342d42042601b376da38dc8e1dba9ca80e9539`, fixes the replacement cleanup finding with the existing startup-cleanup marker, scoped to contextual abandoned task creations. The replacement was never prepared, so preparation-stop proof is not required. No dispatch is fabricated and no normal-startup query is added. The actual queue/sweep/driver/receipt/cancellation regression fails before the fix and passes after it: void, running or foreign-task status retains ownership; exact-task stopped status permits completion even when Release returns void. **64 affected targeted cases and server typechecking pass**. Independent review found no blocker. Current source is `f53af72af76f263096c3c2e788506e6faef9474f`. It fixes both recovery findings from the prior head's Apex 3/5: orphan recovery now preserves pending or completed-but-not-cancelled cleanup through its existing final-write fence, and exact late identities can reach verified cleanup after reconciliation or restart. The regression selection went from four failing assertions to six passing cases. Affected selections cover 217 distinct cases; the exact final six-case regression and server typecheck pass. Independent reviewers found no blocker in either fix. Current-head standard review is 5/5 and all CI checks pass (54 successes, three expected skips, plus Snyk). Apex also completed at **5/5** on this exact head, with zero unresolved threads. The requested independent availability audit is complete on this head. It found an activation-dependent operability gap: an acquisition response that is never received can leave an unknown creation identity with no supported task inspection/reconciliation path, while cleanup keeps the run owner reserved. This does not occur in ordinary lower-only calls. The contract and recovery policy need follow-up before activation; the external review scores do not resolve that availability tradeoff. The split preserves the accepted implementation at `21db7b3851` after its clean merge with frozen master `946735d03a55c1fb4cc5434d6bf1d87742daa9df`. The initial dependent activation PR restored exact aggregate tree `f68c6895f87c7b93ae03e6324d955d2659a99dce`. Both subsequent cleanup corrections are integrated into upper head `27978f294a02da3992585c7f4daa73cd8fcf99f8`; its current aggregate tree is `a0b66c33f5e22745f034f45d24d213f5c210cf16`. Lower-only payload tests cover the absence of capability advertisement and transport projection. Only targeted local tests and typechecking are used. Full regression coverage runs in GitHub CI. No deployment or ROM qualification is part of this change. The lower audit covered connected worker, creation-journal, Stop, restart, deadline and orphan-recovery paths. A new isolated real-PostgreSQL characterization reproduced the unknown-ID hold through actual recovery, cleanup verification and run-control cancellation. It models a lost acquisition response; it is not a live provider or complete IPC test. No source edits were made by the audit. The upper/composition audit remains required after its review cycle. ## Risks - Recording a replacement result now commits with the existing cleanup notification. This conditional replacement path adds transaction/notification responses; common retained startup is unchanged. - This adds cleanup and recovery state handling. A missing receipt or permanently inaccessible old backend can keep ownership held indefinitely. Temporary outages retry; an ordinary retry is not evidence that a deleted or replaced backend stopped. - A caller timeout does not undo work already accepted by a plugin. Unknown resource creation remains unresolved until its identity/outcome is established. Current task creation has no supported inspect-by-operation or operator reconciliation API for a permanently lost identity; repeated Stop alone cannot resolve it. - The support code understands records produced by the dependent activation PR. It does not enable the shared startup policy by itself. - Existing legacy callers must retain their behavior; context-only automatic replacement cleanup remains explicitly scoped. ## Model Used OpenAI Codex, GPT-6 family. The exact model ID and context window are not exposed in this session. Tool-assisted implementation, targeted testing, and independent subagent review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, with available version and capability details - [x] I have checked ROADMAP.md and confirmed this is the requested split of existing work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue with the required fields - [x] I have not referenced internal issues or private links - [x] My branch name describes the change and contains no internal ticket ID - [x] Targeted local tests pass on this source - [x] I have added or updated relevant tests - [x] I have updated relevant documentation - [x] I have documented the risks - [x] All current-head GitHub CI gates are green - [x] Current-head standard Greptile and Apex are 5/5 with no unresolved findings - [x] I will address all review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
45a3a92e6d |
fix(environments): persist sandbox ownership before creation (#18357)
## Thinking Path > - Paperclip manages companies and the agents that do their work. > - Remote execution creates provider resources that need a durable owner. > - The server could create a sandbox before it saved its lease. > - A failed lease write could leave cleanup data only on the old web server disk. > - This change saves a company-scoped operation before creation and transfers ownership atomically. > - Recovery finds the original Daytona resource or records an explicit manual reconciliation outcome. ## Linked Issues or Issue Description **What happened?** A sandbox could exist without a lease after a database failure or company rebind. Failed cleanup recording could then create a local fallback file. Replacing that web server could lose the cleanup reference. **Expected behavior** A failed admission write makes no provider call. An interrupted creation retains a committed operation. Recovery must not create a second sandbox or destroy an attached lease. **Steps to reproduce** 1. Run the disposable PostgreSQL cases in `server/src/__tests__/sandbox-creation-operations.test.ts`. 2. Reject the lease insert after the controlled provider creates its resource. 3. Start a new receiver process with a different empty web disk. It must find the same resource and create one cleanup owner. Related ownership report: Refs #14672. This change addresses new allocations before lease persistence. It does not replace existing lease recovery. ## What Changed - Add an idempotent migration and a small creation-operation table. Retain original company, environment and run identities across deletion. - Commit creation intent before run, ad-hoc lease, and plugin task replacement calls. Fence duplicate requests and ambiguous lease commits. - Retain a known returned task resource ID in the existing operation before attachment. Preserve manual reconciliation and never delete an ambiguously attached resource. - Keep company and run authority checks and atomic lease transfer with the lease owner. - Reuse Daytona names, labels, read-only inspection and claimed cleanup. Keep provider payload handling in the provider. - Retain nonsecret credential identity hashes. Refuse recovery lookup under a changed credential value. - Record manual reconciliation for missing/deleted or explicitly inactive saved credentials, versions, and provider vaults. Keep temporary resolution failures retryable. Never reactivate a disabled credential. - Use transaction signals and concrete deadlines. Failed records back off independently. Manual reconciliation stays durable and quiet. - Remove new orphan-file fallback writes from the covered paths. Keep legacy file replay and its tests for cutover. - Replace obsolete fallback-producer assertions with normal-entrypoint durability tests. Keep cleanup, diagnostics and warm handoff coverage. ## Verification - [Independent current-head verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534744320) passed 190/190 combined tests, 40/40 incoming TestOps tests, full workspace typecheck, and build. PR head `958af986130f25e136065ad30d1fdd843c46255c` has tree `e6c3e5b02182d5941a70ccf5bf55638df3c2e28d`, identical to tested combined commit `b9d7a9c2e9b30f0153685fb24a5f70b56132201f`. Real PostgreSQL, compiled Rust Runner and controlled object providers were used; external-S3 integration remains outside this local proof. - Current head `958af986130f25e136065ad30d1fdd843c46255c` normally merges master `8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57`. The sole conflict was the maintenance test: a parameterized case now preserves both legacy-local `local_storage` and valid shared-S3 `provider_mismatch` checks, including the parent local receipt, exact version identity, missing-snapshot/no-seed refusal, and unchanged persisted row. The three master TestOps files remain intact. No production, schema, migration, or package changes were made after the fully qualified prior head `0b4d05505e`. Independent review approved the resolution; the full managed-skill-content file and Cloud guard controls pass 58/58 (105.11 seconds). [Required PR CI passed on this exact merge head, first attempt](https://github.com/paperclipai/paperclip/actions/runs/41136987412): 54 successful checks and three intended skips. The additional Paperclip review is neutral at 4/5; two external Superagent checks (Contributor trust and Security Scan) remain in progress with zero annotations and no assessment, so they are not passing scan evidence. Greptile reviewed exact head `958af986130f25e136065ad30d1fdd843c46255c` at 5/5; the final thread audit has zero unresolved threads. All completed evidence below retains its recorded revision scope. The additional Paperclip Review on `958af98613` is neutral at 4/5 with a Ship verdict and no must-fix or should-fix item. Its optional observation is that a generic create rejection remains pending/manual even when an operator believes allocation did not occur. This change preserves uncertainty because the generic create-error contract does not prove provider absence; no new error classification or provider-specific compensation was added. - [Independent final-source verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534328247) passed all 189 real-PostgreSQL/Runner and controlled-fixture cases, full workspace typecheck, and build. The integration commit `c8411a8528386f4824b340ec5fdaaed1efb9ea69` has the same source tree as final PR head `0b4d05505e` (tree `bda65efc14b4c904d0f589154db77bb160641c8c`). Controlled object-provider coverage passed; the separately guarded external-S3 suite was not run because the bounded local MinIO pull was unavailable. This includes the two final compatibility fixes. - The first final-head CI attempt exposed one E2E timing failure in `disconnect-write-recovery.spec.ts:181`: the title stayed unchanged through the 15-second session-recovery poll. The same original case passed in 13.8 seconds on immediately prior head `7ec06f77b7`, and all UI/session/E2E source is identical across those heads. The failure log and CI artifacts are retained; no cause is claimed. One failed-job-only retry is authorized on this evidence, without changing a timeout, assertion, or production behavior. [The single failed-job retry completed successfully](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2): all 35 E2E shard cases passed (6.6 minutes), including the original session-recovery case in 14.0 seconds. The initial failure remains disclosed and its cause remains unproven. - Final head `0b4d05505ee0aa72f2981cdff74761406960ec73` adds two independently reviewed integration fixes after the complete 7ec CI matrix exposed two deterministic failures. Server 9/12 failed legacy local-storage maintenance because the CLI reached the new Cloud server startup guard; server 5/12 found the Daytona package version did not match its bumped manifest. The failed run is retained at https://github.com/paperclipai/paperclip/actions/runs/41131204203. It is not passing evidence and was not blindly retried. - Maintenance now constructs the existing explicit storage dependency after loading the required persisted Cloud snapshot, and passes it to both maintenance owners. It can verify legacy local storage before provider cutover while normal Cloud server startup remains guarded. The original case reproduced red locally, then the complete managed-skill-content and managed-cloud-prerequisites files passed 57/57 (32.05 seconds), including missing-snapshot refusal, unchanged stored configuration, and existing server guard controls. Daytona package version now matches manifest 0.1.10; the complete bundled package identity file passed 14/14 (3.98 seconds), and explicit manifest/lifecycle controls passed 2/2 (328 intentionally filtered; 3.42 seconds). An initial workspace-filter command matched no project and is not counted as pass evidence. The final diff changes no sandbox runtime or schema behavior. [Full CI is green on this exact final head](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2): 56 successful checks, three intended skips, one additional incomplete/neutral Paperclip Review, and no pending or failed checks. The neutral review is not a passing assessment. [Greptile reviewed the exact final head at 5/5](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6529396969), with no new actionable issues and all review threads resolved. - Final combined head `7ec06f77b7c14a745c569d18401f16df3d26b626` includes master managed-skills migration 0417 and shared-storage prerequisites from `198218bf55` through normal merges. Sandbox creation moves to generated `0418_curved_lady_mastermind.sql`; its idempotent SQL is byte-for-byte unchanged. The regenerated 0418 snapshot contains both sandbox creation operations and the master skill mutation receipts, and points to the master 0417 snapshot. Master SQL and journal history remain intact. Exactly five snapshots remain, 0414 through 0418. The only sandbox-specific changes since the approved credential policy are migration metadata/number and the matching test/document references; the sandbox runtime files are unchanged. - [Independent final combined verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6533960026) passed on integration revision `e6f05b4cf0cd`, containing final sandbox head `7ec06f77b7` and actual master `198218bf55`: migration/declaration checks, all 158 real-PostgreSQL/Runner and controlled-fixture cases (251 seconds), full workspace typecheck (81 seconds), and build (35 seconds). - Final combined migration proof: fresh real PostgreSQL has both tables. A separate original-0417 sandbox table with a retained pending row survives two applications of the renamed 0418 SQL unchanged, with the expected primary/index and foreign-key structure. Migration numbering/safety checks pass. The normal runtime migration reapply case passes 1/1 (20 intentionally filtered cases; 46.69 seconds). The first temporary qualification harness stopped before any database assertion because its external `.ts` file selected CommonJS; the `.mts` harness corrected that setup and passed. Independent review approved the exact metadata and reference diff. Earlier full suite and live-provider results below retain their stated revision scope. - Credential-policy head `c606d473b03709f27577a55a1dfc0b0d3b4359fd` handles the remaining inactive-credential review warning. The existing boundary accepts only concrete secret-owner 422 codes and exact no-details disabled/coming-soon vault messages. Unknown 422, authorization errors, provider wrappers, and temporary database/provider failures still propagate. Independent source review approved this three-file correction. No generic resolver, schema, provider protocol, retry cap, or activation behavior changed. - Creation/cleanup qualification at `c606d473b0`: all 39 cases pass against real PostgreSQL (81.27 seconds). The new cases use actual disabled/archived saved secrets, a valid disabled version, and provider vaults created ready before normal API updates disable them or mark them coming soon. Each remains manual through restart/repeated scans, with zero provider lookup and no retry deadline. The independent plugin idle blocker remains intact. The temporary 503 control retains retry ownership and later recovers one cleanup owner. Before this correction, all five inactive-state cases remained pending; that failed evidence is retained. The prior 18-case known-ID task proof below belongs to `237144fcf6`, whose helper and task path are unchanged by this correction. - Known-ID head `237144fcf69a249c0e0310563428c0f55ed6d55c` also addresses the independent task-replacement reliability warning. The existing creation operation now saves only the known nonsecret returned resource ID before expiry validation and attachment. The write is scoped to operation/company/original environment and allows only a missing or matching ID. It changes no ownership state and copies no resolved configuration, credentials, or provider payload. Independent source review approved this four-file fix. - The complete real-PostgreSQL task suite passes 18/18 (9.52 seconds). Rejected attachments retain the ID through manual reconciliation and repeated scans. A real committed attachment followed by an injected lost response retains the same active lease and attached operation, with no release or second create. Same-ID retries work; conflicting IDs and foreign companies cannot change it. The initial three regression cases failed because the ID was absent, before the fix. - The shipped Daytona manifest does not declare task support, so it cannot enter this generic replacement path. The dispatcher requires both declared task support and an environmentTask worker method. Generic drivers without creation lookup still use manual reconciliation, with a known returned ID retained when the database write can commit. No automatic lookup capability or unconditional compensation was added. - Previous boundary-fix head `8ba77def151ea85eeae2db7693254db7f87673b0` fixes the fresh review finding about missing credentials. Only known secret-owner 404 results at the recovery secret-resolution boundary select the existing manual outcome. Provider inspection and transient database/provider errors remain retryable. Independent source review approved this change. - Secret-boundary focused acceptance at `8ba77def15`: all 34 cases pass against isolated real PostgreSQL. The complete creation suite passes 16/16 (100.75 seconds); the complete cleanup coordinator suite passes 18/18 (99.83 seconds). The new cases use normal saved-secret creation and deletion, confirm zero provider lookup, restart/repeated scans, no automatic deadline, preserved independent plugin idle blocking, and no additional idle blocker after the controlled worker supplies its quiet proof. A temporary 503 resolution failure preserves pending ownership and later recovers one cleanup owner. No additional live provider resource was created. - Failed attempts remain part of the evidence. The initial red run reproduced the pending-operation bug and also exposed a numeric-deadline test harness mistake, which was corrected. A later combined run reported an existing replacement-receiver failure and a new idle-fixture assertion failure; it was interrupted while the next test file remained in imports. An isolated diagnostic confirmed that the manual state was correct but the installed fixture plugin correctly blocked sleep without its independent quiet proof. The corrected test retains that blocker assertion before supplying the controlled worker witness. The unchanged replacement-receiver case passes in the final complete 16-case run; its earlier failure is not counted as a pass or assigned a proven cause. No timeout or production idle guard was relaxed. - Greptile reviewed previous combined head `7ec06f77b7c14a745c569d18401f16df3d26b626` at 5/5, with no unresolved review threads. Its CI produced the two concrete integration failures addressed above. The earlier CI and live evidence below remain historical evidence, not a claim that the current full gates have completed. - Master integration update: merged `dbff3747b8` and then latest master `76fde891cc` without rewriting branch history. Only `doc/DATABASE.md` conflicted; the sandbox ownership, object-backed run evidence, and cloud configuration sections all remain. The master merges changed no sandbox implementation, migration, or provider code. Migration numbering/safety checks pass and exactly five schema snapshots remain. At merge head `f5d23d4a59828835d01c150f83fa2645fb2d3bec`, the two focused creation/coordinator suites pass all 32 cases against isolated real PostgreSQL (64.57 seconds). Those merge-head checks precede the missing-credential fix below. The earlier live Daytona proof still covers the unchanged normal creation/lookup/teardown path; the new missing-credential branch is qualified with real PostgreSQL and controlled provider hooks. The optional Superagent Security Scan was inconclusive on prior head `7ec06f77b7`: its pinned PR merge-base checkout failed or timed out, with a neutral conclusion and zero annotations. An earlier scan could not start within 120 seconds; a bounded rerun request returned HTTP 404. These scanner failures are not passing security-scan evidence. On final head `0b4d05505e`, the additional Paperclip Review check is neutral: its assigned review execution did not complete and produced no validated assessment. Fresh Greptile 5/5 and independent source review are separate evidence. Previous-head required CI, independent review, and the live Daytona qualification below passed. - [Complete previous-head CI](https://github.com/paperclipai/paperclip/actions/runs/41105726052) passed at `0b5cbb763ab8faa25aa0ff1d8a405cdc0d3db409`: 53 successful checks and three intended skips, including typecheck, full tests, build and E2E. Greptile is 5/5 on this head and both original findings are resolved. - Initial head: 200 PostgreSQL runtime, creation and coordinator tests passed. The corrected head also passes all 32 creation and coordinator tests, including deletion before/during lookup, a failed lookup after deletion, restart, settled polling and actual idle-safety checks. Cases cover failed admission, failed lease writes, concurrent recovery, lost provider replies, ambiguous transfer retries, company rebind, environment deletion, stopped runs, lookup backoff, restart, drain and settled SQL quiet. - Separate receiver processes: passed. A controlled HTTP provider retains the resource while three independent receiver processes use two empty web disks. Recovery creates one cleanup owner and never repeats creation. - Fresh PostgreSQL migration and two additional applications: passed. The pending creation row survives unchanged. - Server TypeScript and SDK/shared build checks pass after the corrections. Migration numbering/safety checks passed. Exactly five schema snapshots remain. - Daytona adapter: all 330 tests passed, including stable-name lookup, ownership checks, deletion and the updated manifest version. An earlier run exposed the version expectation and it was corrected. Previous-head CI passed. - Live Daytona acceptance passed on `0b5cbb763a`: isolated real PostgreSQL, normal runtime acquisition through independent receiver processes, actual Daytona plugin/SDK 0.203.0, one disposable Ubuntu 24.04 sandbox (1 CPU / 1 GiB RAM / 1 GiB disk). A forced lease insert failure retained the committed operation; a fresh receiver with empty disk found the same sandbox, repeated recovery retained one cleanup owner, and the actual claimed cleanup sweep expired the lease and resolved the operation. Independent SDK lookup returned typed not-found; ordinary recorded-ID inspection returned destroyed. This successful attempt made one create, one recovery lookup, and one teardown. Two earlier failed harness/provider-confirmation attempts are disclosed in the [live acceptance record](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6530313054). The coordinator independently queried all three exact resource IDs through the SDK and confirmed typed not-found for each. - Complete CI-affected heartbeat/plugin warm-task suites passed 21/21 after test corrections. An unchanged workspace exposure case passed in isolation after the prior CI shard reported an unhandled readiness rejection. Complete previous-head CI and refreshed review passed. ## Risks - Deleted companies retain an explicit manual reconciliation outcome. Recovery does not retry missing credentials or insert a lease for a deleted company. - Only Daytona declares automatic creation lookup. Other uncertain providers retain `needs_reconciliation`. A missing creation name alone cannot prove that a late allocation will never appear. - Pending operations use the existing process-local signal and startup recovery model. This does not add cross-replica wake delivery or prove deployed sleep. - Legacy direct provider probes and custom-image setup use separate lifecycles. They are outside this replacement of the run-lease orphan-file fallback. The normal Test path that acquires a runtime lease is covered. - Existing orphan files still require the separate cutover inventory and settlement. This change does not delete them. - The schema change is additive. The new SQL can be reapplied. Historical SQL and journal entries are preserved; only the oldest generated snapshot is pruned. ## Model Used OpenAI `gpt-6.1-sol`, with tool use, code execution and repository review. The harness did not report the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge The required branch status checks (`ci / verify`, `ci / e2e`) and all test/build jobs passed on this head. The non-required Superagent Contributor trust and Security Scan checks are still running without findings; they are not claimed as passes. No required check or branch rule is overridden. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
4e27a82a42 |
fix(server): wake approval queue seeding from committed changes (#17510)
## Thinking Path > - Paperclip manages work and human decisions for AI-agent companies. > - Automatic decision queues store membership for pending attention sources. > - The old attention maintenance scan created that membership as a side effect. > - Moving approval retention out of the scan also removed approval queue discovery. > - Queue API consumers do not build the attention feed first, so they can miss pending PR approvals. > - This pull request gives approval queue discovery its own worker and commit signals. > - The worker becomes quiet after it saves every missing membership. ## Linked Issues or Issue Description Refs #17497. Stacked on #17508, following #17505. This is a correctness dependency for the approval retention migration in #17497. Do not treat that migration as complete without this queue-population path. A real PostgreSQL regression found no PR queue when a pending approval had a linked PR work product and the maintenance feed excluded approvals. Related: #13364 proposes pruning stale seeded memberships. This PR preserves existing membership retention and does not implement pruning. #15149 documents the decision-queue APIs. ## What Changed - Add an approval queue worker that selects missing PR-queue memberships from existing durable records. Bound each pass to 100 rows. Continue past failed rows and revisit them on retry. - Reuse the queue service's seed writer through a normalized source contract. Keep queue rules, uniqueness, and audit writes in that service. Keep feed readers compatible. - Wake from committed approval and link changes, issue/company visibility, PR work-product changes and imports, and queue-rule changes. Reuse the existing transaction boundaries. - Recheck approval status under a row lock. Serialize queue-rule changes with the seed writer. Preserve required recovery during drain and shutdown. - Register startup reconstruction under the existing maintenance switch. Leave other attention-source seeding with its current owner until its own migration. - Document the complete producer audit and stack dependency. Add a validated external PostgreSQL option to the existing queue route suite so it runs on hosts without embedded database support. ## Verification Current tested revision: `7948136ef8`. This PR targets master after #17508 merged as `8406047b4e`. Normal ancestry reconciliation preserved the complete tested integration tree, before the review repairs below. Only the original 25 seeding paths differ from master, and database schema/migration metadata is identical. - **56 cases pass in the three complete affected suites:** 28 approval seeding, 6 queue routes and 22 task-watchdog cases. Server typecheck and production build pass. - Three real PostgreSQL retry regressions failed then passed: new matching arrivals cannot postpone a recovered failure indefinitely; successful partial progress resets old failure penalties; an empty cursor rollover does not reset outage backoff. Exact one- and two-second retry assertions, eventual complete recovery and existing capped-backoff/company isolation tests pass. - Three genuinely empty external databases failed before setup repair because tables were missing. Each suite now applies migrations itself. All 55 cases passed on a second set of fresh databases; adding the final rollover guard then passed all 56 cases using those now-migrated fixtures. No production migration or table is added. - The initial expanded fresh run passed 54 cases but failed a cleanup expectation that omitted a required cursor rollover pass. The fixture now permits that explicit continuation while preserving exact deadline and complete-membership assertions. All failure records remain in `doc/idle-approval-queue-seeding.md`. - The prior integrated revision `4e725e2055` passed **522 cases in fifteen complete producer/consumer suites**, plus types/build. Its unchanged consumer code and historical deadlock, cross-company amplification, backoff, gateway integration and setup evidence remain documented; those counts are not relabeled as current acceptance. Commands: server Vitest for approval queue seeding, queue routes and task watchdogs; server typecheck/build. Fresh exact-head Greptile/Ponytail reviews and all CI checks remain required. No full local repository-suite, deployed suspension or cost claim is made. ## Risks - Signals cover the supported in-process service writers. Independent SQL writers and multiple server replicas need a separate notification contract. - The source predicate must stay aligned with unscoped attention maintenance. The first linked issue, company scope, pending status, and execution visibility are tested. A pending approval itself keeps its linked external conversation active. - Existing enabled seed rules can re-add manually removed matching memberships. This preserves the old scan's behavior. Completed or archived membership is not pruned. - Queue seeding and retention are separate owners. Other attention sources still need their own event-driven discovery before the aggregate scan can be retired. - This PR does not extend Cloud wake coverage, relax sleep admission, or claim measured host/database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5982246771 |
fix(server): schedule custom-image setup expiry from committed changes (#17443)
## Thinking Path > - Paperclip manages AI agent work and its execution environments. > - Interactive image setup sessions must expire even when users leave. > - The server currently scans these sessions on every maintenance tick. > - Empty scans keep the instance and database active. > - Existing session rows already record status and expiry. > - This pull request uses transaction signals and the shared scheduler for those rows. > - Empty and completed setup history no longer create recurring cleanup queries. ## Linked Issues or Issue Description **What happened?** The server checks custom-image setup expiry every heartbeat maintenance tick, even when no active session exists. **What did you expect?** Session writes should update the next cleanup deadline. Existing provider cancellation and cleanup ownership must remain intact. Empty state should stay quiet. **Steps to reproduce** Run an instance with heartbeat scheduling enabled and no custom-image setup sessions. Observe repeated setup-expiry reads. Refs #17145. This is the first PR in the recurring server sleep stack. Its base is master. The full requested scope and verification contract are recorded in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. Follow-up PRs target their predecessor and each own a separate reason for recurring work. ## What Changed - Carry the historical cleanup retry-observation fixture repair and its existing external test-database hook at the stack foundation. Runtime scheduling is unchanged. - Carry the previously verified login-cleanup fixture settlement correction at the stack foundation so every intermediate CI run receives it. This is test-only maintenance; custom-image expiry behavior is unchanged. - Add a custom-image expiry worker to the existing delivery coordinator. Remove its maintenance-tick scan. - Signal initial creation, provider refresh, cancellation/failure, and promotion inside their owning transaction. - Keep the active-status predicate, earliest saved expiry, and provider behavior in the custom-image service. Shared code gains only a queue topic. - Preserve startup recovery, bounded retries, concurrent write generations, and drain deferral. No tables or external alarm changes. - Add PostgreSQL lifecycle tests and repair this suite's database cleanup callback. A dedicated external test URL permits disposable PostgreSQL when the host cannot start another embedded cluster. ## Verification Current head: `3507c6be1e054fdd93353d5dcf77d4899b44518f`. This remains a single-duty PR stacked on master. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/3507c6be1e054fdd93353d5dcf77d4899b44518f/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md), [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/3507c6be1e054fdd93353d5dcf77d4899b44518f/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Signals cover supported service producers sharing the DB owner. Independent SQL and another process require the existing restart/reconciliation contract. - Failed provider cancellation retains the existing failed-session and historical-cleanup behavior. This PR changes scheduling, not teardown authority. - Active rows without an expiry preserve the existing behavior; no deadline is invented. Cloud sleep eligibility remains conservative. - The worker remains in the existing heartbeat-enabled startup path. It does not change disabled-scheduler behavior or prove deployed suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Review follow-up: Added a real producer commit during a paused deadline read. Updated startup assertions for coordinator ownership and removed added code comments. The three affected suites pass (102 tests). The separate login-cleanup suite also passes locally (16 tests); its prior CI consume test timed out, and the fresh complete CI run passed. External PostgreSQL tests now use the repository guard for disposable databases ending in `_test`. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
71c04c0bfb |
fix: stop idle environment cleanup polling (#17145)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent environments can leave pending teardown or unverified provider resources. > - Recovery must preserve those resources until ownership and workspace safety allow cleanup. > - Empty maintenance scans keep an idle instance and its database active. > - The existing rows, transaction signals, and scheduler can represent the actual recovery obligation. > - This pull request schedules pending cleanup and historical verification only while work remains. > - An empty or fully verified instance stops these scans without losing cleanup recovery. ## Linked Issues or Issue Description **What happened?** The server scanned pending environment cleanup on its maintenance tick. The run reaper also scanned it even when it recovered no runs or active leases. Historical resource verification scanned every five minutes, including when all history already had valid cleanup proof. **What did you expect?** Writes should wake cleanup. Pending rows, deferred orphan buffers, and unverified history should retain recovery deadlines. When none remain, these workers should make no periodic database or provider calls. **Steps to reproduce** Start a configured instance with no pending leases and fully verified history. Leave it idle. Observe repeated pending-cleanup and historical-verification reads from the maintenance paths. Related: #17106 supplies the preceding login-cleanup migration. #13523 improves lease-reaper query indexes. #16809 addresses reusable-sandbox recovery authority. This PR keeps those authority checks in their existing services. ## What Changed - Register two workers with the existing delivery coordinator and shared scheduler. Remove their unconditional maintenance-tick calls. Run the reaper's inline pending cleanup only after it recovers actual work. - Add commit-aware signals to the audited lease and custom-image writers. Existing transactions notify directly; other writers use one small transaction helper. No tables or migrations are added. - Derive pending cleanup deadlines from the existing row backoff and retry or claim metadata. Retain a historical retry only while an unverified candidate exists. - Register unwritten orphan buffers across runtimes that share the database handle. Buffer registration wakes recovery without a database connection. Load the existing spool at startup and retry failed or capacity-limited loads while needed. - Isolate spool failures per runtime. Attempt all buffers and persisted pending rows, then retain a retry for any failed spool. - Document producer coverage, idle behavior, and remaining blockers. Update the sleep inventory. - Repair the inherited CSS-recovery E2E test: the current build no longer emits task-page CSS. Exercise the actual lazy environment stylesheet, preserving real network failure, recovery, and bounded reload assertions. ## Verification October 24 integration update: merged current master and resolved the runtime/spool conflicts. Preserve serialized ingestion of records appended after startup. Plugin replacement failures now register deferred recovery with an idle hold; concurrent callers share one persistence attempt. The producer audit includes this new writer. - Focused cleanup selection: 270 tests now pass across five suites. The initial run lacked a built plugin SDK and executed no tests; building the SDK fixed setup. The new concurrent-flush assertion then reproduced duplicate rows; a single-flight guard fixed it and all 18 plugin integration tests passed on rerun. - Server typecheck passed. After review fixes, all 211 tests across the cleanup worker, real orphan spool, environment runtime, and plugin integration suites passed. These overlap the earlier focused selection. The rate-limit test initially expected a write after twenty idle minutes to wait; corrected it to assert that subsequent writes cannot accelerate the five-minute cadence. - Review fixes preserve retries for unreadable spool directories/records, keep historical provider checks off the startup wait while joining them at shutdown, and restore their independent five-minute minimum spacing. The complete [CI matrix](https://github.com/paperclipai/paperclip/actions/runs/40108905033) passed on final commit `79ae1589c3c7b924db4bedb337d0c7ed3d95020b`, including all E2E shards, server/workspace tests, runner checks, typecheck, build/public-install verification, and security checks. - CI found a stale E2E asset matcher: `issue-detail-*.css` is no longer emitted. Both corrected browser tests passed against a fresh production UI build and disposable local instance. Fresh Greptile review is 5/5 on that same final commit, with all review threads resolved. Previous implementation verification: - `pnpm -r typecheck` and `pnpm build` passed. - New PostgreSQL worker suite: 12 tests passed. Covers quiet configured state, verified history, writes, outer commit and rollback, retry deadlines, restart, drain, deferred orphan buffers, spool failure, pagination, provider uncertainty, shutdown, and the idle run-reaper path. - Worker and environment runtime suites after review fix: 183 tests passed. A new assertion reproduced buffer starvation before the fix. Updated transaction fault injection and isolated simulated-instance database handles. Server typecheck also passed after the fix. - Run-recovery suites: 58 tests passed. Pending-cleanup suite: 38 tests passed after updating the logging fixture to seed real orphan recovery; an idle reaper no longer invokes cleanup. - Stop and workspace-recovery integration selection: 268 tests passed; two live-provider tests skipped by their environment gate. This overlaps the worker suite and is not an additional unique-test count. - Started the full local `pnpm test:run`, then stopped that older duplicate run after CI found the fixture mismatch. The corrected 38-test suite passed locally. The full sharded [CI matrix](https://github.com/paperclipai/paperclip/actions/runs/39678174479) passed on final commit `fc0ff3c4554ca4190a8b0b81ec06ef7065b8ad2c`, including build, typecheck, runner, server, workspace, and E2E checks. This is not a claim that the local full run completed. ## Risks - Missing a future producer signal could delay discovery until restart. The producer audit is documented in `doc/idle-environment-cleanup.md`. Independent SQL writers and another server process are outside the existing process-local notification contract. - Historical candidates blocked by ownership or provider uncertainty keep a five-minute retry until they can be verified. Provider destruction, claim fencing, Stop retention, and workspace safety remain controlled by their existing services. - This does not remove orphan-active discovery or legacy shutdown settlement from run recovery. Active custom-image expiry is also still periodic. External wake coverage does not change. - Local tests prove these workers become quiet and recover. They do not establish deployed instance suspension or fleet cost savings. ## Model Used OpenAI Codex (GPT-6), with code execution and reasoning. The exact serving model ID and context size were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |