mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 21:01:54 +01:00
84aa989bf9d08fc005ac273c37f9e1d860a4db25
2302
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
84aa989bf9 |
Preserve native sessions through legacy workspace enrollment (#18943)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native session retains its workspace and provider session across turns. > - An ephemeral web server must restore those files without its predecessor's disk. > - Legacy enrollment changes the workspace authority and can change both workspace addresses. > - The retained provider backup checks those addresses as part of its identity. > - This pull request preserves the exact addresses in a verified migration receipt. > - The benefit is that the next turn can use the original session after enrollment. ## Linked Issues or Issue Description Refs #18932, merged as `612d9931a6`. This pull request builds on that all-Runner durability change. Related: #18918 fixes concurrent workspace preservation locks. This change covers identity preservation during offline enrollment. ## What Changed - The native workspace migration owner records the selected session bindings in the successful workspace migration operation. - Runtime selection retains the original logical address only for the exact agent, task and session. - Remote workspace preparation restores the original provider address. It refuses conflicting targets, forged receipts and occupied unowned roots. - The environment owner includes the migration operation in the reusable lease fingerprint. Warm policy stays warm and receives a fresh allocation through normal lease cleanup. - Exact migration evidence permits the existing verified provider backup to restore into that fresh warm allocation. - Preserve verified provider allocation files through a normalized, lease-bound digest receipt. Unknown files and altered markers remain refused; interrupted publication is retryable. - Retain legacy managed-instruction session fingerprints using the exact imported content hash and recorded paths, preserving all other configuration and credential checks. - Migrate existing local skill archive/manifest objects through upload claims, readback verification and receipt compare-and-swap; retain original objects and IDs. - Add explicit two-turn browser qualification for every released native Daytona profile, with warm and per-turn variants. The production maintenance entry runs offline. The old home is deleted only after a verified export and successor readiness. The oracle checks provider identity, native session, addresses, reference bytes, question and final document. ## Verification - Full repository typecheck passed at `b12ecd823e`. The production session fix passed 198 session/credential tests. Full repository build passed at `94bb917947`. The final Rust-only provider fix passed 57 ACPX provider tests and the exact Linux release build. - Workspace and transport checks passed: 117 transport/workspace tests, six final real PostgreSQL migration boundary cases, 178 environment tests, 539 Daytona tests (eight skips), and the native warm-backup admission control. - Local skill migration passed eight PostgreSQL/S3 tests (one existing opt-in skip). Failed readback retains the source. Successful readback still works after deletion of the old home. - Latest Product E2E typecheck and 38 focused harness tests pass. Earlier catalog/admission/report checks passed. Twelve unrelated support-test failures reproduce on unchanged base `1a39dea9aa`; these are not reported as passing. - Live Codex per-turn at `2d5bb9a6d1` retained the native and provider sessions, both workspace paths and exact reference bytes after deletion of the old home. It completed the task and delivered its saved document and original reference. Its original failure was an output-count grading defect. The corrected grader passes all 21 checks against retained, hashed evidence; the original result remains unchanged. - Live OpenCode per-turn and Codex warm at harness `94bb917947` pass all 21 product checks. Their launcher cleanup failed because a read-only diagnostic observer kept its disposable database open. The observer now closes before ordinary database cleanup; a no-provider reproduction confirms the fix. All owned sandboxes were independently verified absent, child keys revoked and buckets removed. Original failed campaign dispositions remain unchanged. - Claude at `94bb917947` kept its sessions and reference bytes but omitted the final clickable document link. That user-delivery failure remains recorded. The fixture now explicitly asks for the link and records migration evidence before checking navigation. The fresh Claude attempt at `b12ecd823e` passed both strict migration checks: exact provider/native session, both paths, original file bytes and independently downloaded attachment. Its full browser workflow failed again: it requested unnecessary confirmation, used the answer sentence as the reference, and omitted the clickable document link. This is retained as a failed user-workflow result, not a full pass. - Earlier live failures exposed and reproduced the skill-object, fresh-allocation marker and managed-instruction fingerprint bugs. The fixes retain original identity checks. No provider session reset or accounting bypass was used. - The real Grok continuation at `b12ecd823e` failed before provider execution because its immutable-instruction attach policy rejected relocated registered file copies. A no-provider regression reproduced the exact rejection. `b05ba3728a` selects the existing strict authenticated asset-path policy for Grok; content digests, unknown context fields, session/workspace identity and same-run mutations remain enforced. The regression and all 57 ACPX provider controls pass. The Grok browser migration now passes all 21 checks with the b05 Linux Runner, including provider identity, exact reference bytes and final user output; the launcher also reports successful cleanup. - `24fabf1d81` applies the same policy to released ACPX Codex after a second no-provider reproduction. All 58 ACPX provider controls and the final Linux release build pass. Its live browser migration passes all 21 checks on the final binary, including exact original session/file preservation and completed user output. - The full native-session executor suite passes all 1,121 tests after updating its existing lease database fixture for the migration-receipt join. The prior CI failure was this unsupported test-double method; the production guard still rejects ordinary unproven recovery. CI and Greptile 5/5 passed at `ca1e0d5fe6`. The first CI attempt lost several workers and had one rotated-recovery transport failure; its seven descriptor cases passed locally, and the single unchanged retry passed. The original failure remains recorded. Staging enrollment has not started. Initial offline enrollment and subsequent rolling availability are separate checks. - The final review follow-up adds real runtime-selection PostgreSQL cases for exact, different-session and absent receipts, plus executor-level warm restoration that copies actual backup files and verifies provider identity and the new lease stamp. Both complete affected suites pass (1,172 tests), and full repository typecheck passes. Removing either production migration connection independently makes its new positive case fail; both source files were restored byte-for-byte after these negative controls. - Final commit `0971d373a3` passes all 63 checks (57 successes, five expected skips, one neutral); Greptile is 5/5 and every review thread is resolved. ## Risks - An identity conflict blocks enrollment and keeps the original files. It does not silently reset the session. - A new migration generation retires a reusable lease through its existing owner. The original sandbox must finish normal cleanup before reuse proceeds. - Initial enrollment has an offline step. The separate rolling suites prove uninterrupted deploy behavior after enrollment. - This adds no schema migration and changes no provider credentials or accounting gate. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this session. The agent used reasoning, terminal tools, code edits, PostgreSQL/filesystem tests and isolated live browser qualification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
612d9931a6 |
Support per-turn Runner sessions on ephemeral web servers (#18932)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server controls native Runner sessions in remote sandboxes. > - An ephemeral web server must recover without its previous local filesystem. > - The existing admission rule accepts warm sessions. Managed Claude, Grok, and OpenCode sessions resolve to per-turn execution. > - Per-turn execution also needs durable provider-session bytes after the sandbox is retired. > - This pull request adds encrypted backup publication and removes the warm-only admission rule. > - Both lifecycle modes can use the existing native ownership and checkpoint contracts during a web deployment. ## Linked Issues or Issue Description Refs: #18518, #18529, #18826, #18835, #18841, #18864, #18887. **What existing behavior does this improve?** Native Runner execution on web servers with `PAPERCLIP_EPHEMERAL_SERVER=true`. **Current behavior** The admission rule requires a warm Runner lifecycle. Managed provider credentials can resolve an agent to per-turn execution, so enabling the ephemeral deployment profile blocks those turns. A database controller checkpoint does not preserve provider files after the old web home and per-turn sandbox are removed. **Proposed behavior** Admit native execution in either resolved lifecycle. Keep provider and credential choices unchanged. Publish encrypted provider state to shared storage before sandbox cleanup. Restore it from a fresh web home for the next turn. Use the existing controller lease and generation checks for active takeover. **Reason and benefit** Operators can remove the web server volume without excluding per-turn providers or forcing managed accounts into warm sessions. New native Daytona profiles automatically enter both rolling-deployment qualification lanes. **Breaking changes** The additive migration creates a backup publication ledger. Existing volume-backed local-disk installations retain their local backup path. S3 installations publish encrypted backups and require a stable secrets master key. Existing-volume enrollment must import surviving session data before detaching the volume. ## What Changed - Admit native Runner execution independently of its warm or per-turn lifecycle. Keep host-owned legacy execution excluded from ephemeral deployments. - Store private provider state as encrypted objects. Journal uploads before I/O, read them back, and fence publication against the current native owner. Keep the previous published backup when storage or ownership checks fail. - Restore exact provider bytes into a private cache. Require an exact durable backup before sandbox cleanup. Extend the existing bounded retention pass to retire stored objects without adding idle polling. - Import selected legacy controller and provider state in the existing offline maintenance process. Preserve queued work, task pointers, pending recovery, and original files. Prefer canonical database checkpoints. - Preserve already-failed legacy controllers with missing original artifact proof as encrypted archives. These archives do not become execution authority. Successful retained sessions still require proof. Report archival and executable imports separately. - Verify unmanaged Grok API keys through the existing xAI account validator. The live test exposed a readiness gap that kept remote Grok agents paused after platform checks. Preserve failed credentials and the separate runtime warning. - Ingest Grok whole-prompt usage at the owning ACP boundary. Bind the receipt to the exact outbound request/session; reject replay, conflicts, and incomplete authority. Separate turn cost from cumulative session cost in the sidecar schema and Rust normalization. Preserve managed billing identity and identify direct xAI API credentials. - Preserve published workspace directory names across cold allocations, including destroyed-allocation cleanup. Keep unpublished epoch/lease directories isolated and refuse to overwrite an occupied preparation. This fixes the cold-session identity regression exposed by #18922. - Increase bounded checkpoint-header inspection to 1 MiB. Retain the aggregate inspection limit. - Register all nine native Daytona profiles in per-turn and warm rolling-deployment suites. Delete the entire departing server home, exercise ordinary reads and writes, require active ownership transfer, then check a later turn on another fresh server. Permit only exact notification replay after reconnect. Ownership stays with the existing modules: Runner validates its protocol state; Core validates run scope and controller ownership; storage owns object I/O; the maintenance process owns offline conversion; the E2E wrapper owns process replacement and probes. No provider protocol is added to the shared admission rule. Grok receipt parsing stays in its provider module. The shared sidecar gains an explicit cost basis because its existing price field means session-cumulative spend, while Grok reports prompt totals. Existing consumers retain cumulative behavior when the new field is absent. ## Verification - Focused native admission, archive validation, publication fencing, cleanup, readiness, legacy import, and E2E observer tests pass. Real PostgreSQL cases verify that queued intent and historical recovery rows stay unchanged. - Full typecheck and build pass on the rebased branch. The review follow-up passes server typecheck, 29 focused PostgreSQL/archive/readiness tests, and module-boundary checks. Revision fb8d39a7a1 reached 55 successful CI checks and Greptile 5/5 with no unresolved threads. The Grok follow-up passes Runner/server typechecks, 238 ACP runtime/projection/usage tests, 16 sidecar-schema tests, the Rust cost regression, and Core billing regression. The |
||
|
Devin FoleyandPaperclip
|
1a39dea9aa |
fix(heartbeat): keep a routine claim queued while its slot is held (#18933)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A routine creates one task per tick. Only one open task of a routine can hold an execution lock (`issues_open_routine_execution_uq`). > - Since #18879, a failed native run that waits for a scheduled resume keeps its task's lock. Routine dispatch counts only queued, running or scheduled runs as live, so it can queue a newer task of the same routine meanwhile. > - When that newer run is claimed, stamping its execution lock violates the unique index. The raw database error is not a deferrable claim rejection, so it escapes the queue loop. The queue code notes that this can stop the agent's queue and startup recovery. > - This pull request turns that exact violation into a 409 claim rejection, so the run stays queued until the slot is free. > - The benefit is that a busy routine slot delays one run instead of stopping the queue or startup. ## Linked Issues or Issue Description Refs #18879 (keeps a failed run's lock while its resume is pending), #18916 (defers the same unique-index conflict in restart recovery's lock reclaim) and #18931 (open; keeps one run's restart-recovery failure from stopping startup recovery). This is the same index conflict on a different path: the heartbeat queue claim, not restart recovery. It does not change #18916 or #18931. Its doc paragraph is separate from the one #18931 edits. **What happened?** While auditing the lock paths from #18879 for this index, I found that a queued run of a newer routine task can be claimed while an older task of the same routine still holds the slot through a failed run with a pending resume. The claim's lock stamp then raises `duplicate key value violates unique constraint "issues_open_routine_execution_uq"`. `isDeferrableClaimRejection` accepts only `HttpError` 4xx, so the error escapes `startNextQueuedRunForAgent`. **Expected behavior** The claim is rejected as a conflict and rolls back. The run stays queued and is claimed after the older task releases the slot. Other errors behave as before. **Steps to reproduce** 1. Create two open tasks of the same routine (same origin and fingerprint). 2. Give the older task an execution lock held by a failed run. 3. Queue a run for the newer task and claim it. On `master` the claim throws the unique violation. The new queue test runs these steps against PostgreSQL. **Paperclip version or commit** `master` at `0dedbf23c35`. **Deployment mode** Any server with routines whose tasks run on native runners. ## What Changed - `bindClaimedIssueExecution` in `server/src/services/heartbeat/queue.ts`: if stamping the execution lock violates `issues_open_routine_execution_uq`, it throws `conflict("Another execution for this routine is already in progress")`. The claim transaction rolls back, and the queue loop keeps the run queued as it does for other 4xx claim rejections. Every other error is rethrown unchanged. The issue checkout route already maps this violation to the same 409. - `doc/execution-semantics.md`: one paragraph on the behavior. ## Verification - Failing first: `queue.test.ts` "keeps a claim queued while another task of its routine holds the execution slot". On `master` the claim raised the `issues_open_routine_execution_uq` violation. Now it rejects with 409, the run stays `queued`, neither task's lock changes, and nothing executes. - Failing first, review round 1: `heartbeat-queued-run-claim-isolation.test.ts` "keeps a routine run queued while another task holds the routine slot, then runs it once the slot is free" goes through `resumeQueuedRuns`. On `master` it rejected with the unique violation. Now the routine run stays queued with no error code while another queued run for the same agent executes. After the slot is released, the routine run executes exactly once (executions are counted per run, because the finished task can queue its own follow-up run). - `cd server && npx vitest run src/services/heartbeat/queue.test.ts src/__tests__/heartbeat-queued-run-claim-isolation.test.ts`: 70 passed. - `npx vitest run src/services/heartbeat/recovery.test.ts src/__tests__/routines*.test.ts`: 4 files, 205 passed. - `pnpm --filter @paperclipai/server typecheck`: pass. - Not verified: the queue loop end to end on a real instance. The loop's handling of a 4xx claim rejection is existing behavior. ## Risks - While the older task holds the slot, the newer run stays queued and the loop logs "queued heartbeat run claim was rejected" on each pass. It is claimed once the older task's resume finishes or its lock is released. - Only this one constraint is mapped. Other database errors still escape as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
652cd76026 |
fix(ui): expose Stop for failed native retries (#18899)
## Thinking Path > - Paperclip manages agents and their tasks. > - A native run can fail while it still owns a same-run retry. > - The server supports Stop for that pending execution. > - Run details show Cancel only for queued or running records. > - A failed pending retry therefore has no ordinary cleanup control. > - This pull request exposes the existing Stop eligibility and uses the existing cancellation endpoint. ## Linked Issues or Issue Description Refs #14743 and #14733 for correlated native Stop. Those changes cover backend cancellation and protocol qualification. They do not add the failed-run control to run details. Searches of current master and open PRs found no matching UI repair. **What happened?** A native run can show failed while its coordinator still owns a retryable failure with no accepted result. The backend can cancel that retry. Both run-detail implementations hide the cancellation button because the record is terminal. **Expected behavior** The selected run offers Stop when the server confirms that pending native retry. Unknown, legacy, terminal and accepted-result states do not gain this control. Existing queued and running cancellation stays available. **Steps to reproduce** Render either run-detail implementation for a failed native run with an eligible retry. The original UI has no Stop button. The original batched execution projection has no eligibility field. The regression tests exercise both real components and the actual PostgreSQL projection query. **Paperclip version or commit** The missing control and projection field were reproduced on |
||
|
DottaandPaperclip
|
0dedbf23c3 |
fix(chat): retain receiver ownership during runtime construction (#18930)
## Thinking Path > - Paperclip manages agents and their work. > - Chat receivers hold a durable lease before they accept messages. > - Runtime construction can outlast a lease renewal interval. > - Renewal previously required an attached runtime before construction had returned. > - That check could revoke the same receiver that was still starting. > - This change uses the exact pending initialization as proof during construction. > - The existing durable token and endpoint checks still fence every renewal. ## Linked Issues or Issue Description Related: #18899 exposed the existing receiver failure in its chat CI shard. This is a separate lifecycle repair. #18795 provides the relevant production and sleep fixture context. Searches found no overlapping open initialization repair. **What happened?** The receiver acquired its lease and started its renewal timer. The service then awaited credentials, command preparation and runtime construction before attaching the runtime to its context. A renewal during that interval treated the missing attachment as lost ownership. Configuration later failed with a Gateway ownership error, although its original durable token was still current. **Expected behavior** The exact pending initialization can retain its own lease while construction runs. Missing attachment alone must grant no ownership. A changed token, paused or removed endpoint, changed generation or credentials, and shutdown must still fence the receiver. **Steps to reproduce** 1. Use the real configure and PostgreSQL ownership producers with a 60-second lease and 50-millisecond renewal. 2. Hold the actual runtime construction call for 200 milliseconds. 3. Release construction. Untouched code returns the Gateway ownership error. 4. With the same construction delay and a five-second renewal control, configuration succeeds. **Paperclip version or commit** The causal regression fails on unchanged production source at `e7549ab613`. The original CI error was observed in the chat shard of #18899. That CI run did not record the precise setup and renewal order, so this local proof does not establish its historical scheduling. ## What Changed - Store the runtime context identity in the existing pending initialization entry. - Allow pre-attachment renewal only for that exact tracked initialization and current local ownership. - Recheck local ownership inside the existing renewal transaction. Preserve company, endpoint, generation, credential and original-token checks. - Before SDK listener initialization, wait for any older renewal, recheck local ownership, then obtain fresh durable proof. Preserve the existing checks after initialization. - Keep attached runtime identity checks, lease deadlines, renewal intervals, shutdown and listener retirement unchanged. - Add actual configure and database race controls. Document the shared Discord and Photon receiver contract. ## Verification - Untouched production red: the actual construction gate crosses a renewal and returns the same Gateway ownership error. The otherwise identical slower-renewal control passes. No lease, exception or provider outcome is injected in these positive cases. - A subsequent review found a second real race: final runtime removal did not prove that SDK initialization had never started. Four actual initialization-hook controls failed before the pre-initialize fence. The final matrix asserts zero hook calls for lost or changed authority, including a foreign-token control with no periodic renewal due. An attempted cross-company endpoint move is rejected by the existing database foreign key; the valid original initialization still succeeds. - A third actual-producer red holds completion of a real committed renewal, changes the durable token, and proves that joining that old promise could incorrectly accept configuration. The corrected startup check waits for that renewal and then obtains fresh proof; it cannot credit the earlier result alone. - Final relevant full Discord registration group, takeover/callback cases, buffered host-pause recovery, failed construction, credential-fenced pause and shared Photon consumers: 34 passed across two files; 1,840 other cases were deselected. Four executed cases cover Photon transport recovery, failed authentication, duplicate allocation ownership and checkpoint fencing. The tests use the real configure/database producers with fake provider transports and observe SDK initialization; they do not prove physical network behavior. - All server type layers, the module-boundary gate, and production server/vendored Runner build pass at exact final `38cf82ec114bd12c0aa7bbc125ed72b1ae941b4e`. The production build stamp records this exact commit. - Earlier fixture mistakes are preserved separately: a test initially changed every lease key, including the credential lease, and another attempted an invalid cross-company row move. Neither is counted as a production behavior red. The corrected controls target the Gateway key and assert the existing company constraint. - An initial isolated dependency graph resolved an older shared telemetry build. Correct workspace links restored the all-layer typecheck without changing telemetry source. - Full repository checks were not run locally for this narrow slice. Fresh current-head CI and substantive reviews remain required. No hosted task, provider work, fleet action or deployment was performed. ## Risks The ownership helper is shared by Discord and Photon. The selected tests cover both consumers. The pending entry grants only permission to renew the original token; it grants no message, callback or publication authority and adopts no listener. Missing or changed initialization identity fails closed. There is no schema change, historical backfill, new worker or additional idle scan. ## Model Used OpenAI `gpt-6.1-sol` in Codex, with reasoning, repository inspection, source editing, shell execution and test tools. The host did not report the context-window size. A separate coordinator reviews the source before delivery. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f211d44c00 |
fix(heartbeat): preserve queued coalesced comments at claim (#18913)
## Thinking Path > - Paperclip manages agents and their work. > - The wake queue records task comments before an agent starts. > - A queued run can accept another comment through coalescing. > - Claim rebuilds that run's input from its primary wake. > - Coalescing updated the run and its receipt but left the primary batch unchanged. > - This change saves the accepted batch atomically in the owning writer. > - The benefit is complete input at claim without repeating old work. ## Linked Issues or Issue Description Related: #18887 preserves offline queued turns. #13948 addresses equivalent automatic wake deduplication. This change repairs the accepted comment batch at the existing claim boundary. **What happened?** A queued run started with comment A. Admission accepted comment B, saved `[A, B]` in the run context, and finished B's coalesced receipt. The primary wake still held `[A]`. Actual claim rebuilt input from that wake and dropped B. **Expected behavior** Claim must retain every accepted live comment in order. Stop, changed source authority, deleted comments, and company boundaries must still apply. Already claimed wakes must retain their existing recovery source. **Steps to reproduce** 1. Queue a task run whose primary wake has an authoritative comment batch `[A]`. 2. Admit comment B through `createHeartbeatQueue.enqueueWakeup` and the real Postgres admission writer. 3. Confirm that admission accepts `[A, B]` and records B's coalesced receipt. 4. Claim the original queued run. Before this fix, the claimed input contains only A. **Paperclip version or commit** The untouched producer regression fails on `d7283e875772ca8c684dc86e471e6b48d543b7b1` with both process and native Runner/Codex adapter selection. Provider execution is disabled in the fixture. ## What Changed - Save the accepted comment batch in the unclaimed primary wake, run context, and coalesced receipt in one transaction. - Lock the primary wake before the run. Recheck the expected run status and same-company, same-agent and run bindings; apply the existing actor equality fence only to durable receipts before mutation. - Preserve incoming input as normal deferred work when the locked source changed. Preserve receipt identity and actor fields. - Preserve the original primary actor and trigger comment while updating its accepted batch. Each incoming receipt retains its actual actor. Keep claimed source payloads and legacy envelopes on their existing recovery paths. - Safely defer a same-named different agent without rewriting the original agent's run, wake or receipt. Missing and foreign-company targets remain rejected. Do not rebuild input from historical receipts. - Add actual process/native producer, real database rollback, Stop, deletion, foreign-source, and fallback controls. Update three direct writer fixtures for the required expected-status contract. ## Verification - The untouched process and native producer red on `d7283e875772ca8c684dc86e471e6b48d543b7b1` loses accepted comment B at actual claim. The initial incomplete mock is retained separately and excluded from behavioral proof. - Current process/native matrix: 22 passed, including mixed authors, system primary, same-named different agents, changed status, claimed sources, Stop, deletion and foreign-source boundaries. Queue, Postgres admission and use-case suites passed 234 cases before the source-preserving serializer follow-up; its 22 actual producer cases passed again afterward. - CI on the previous head found a real ordinary mixed-author workspace-copyback regression. The earlier 13-case selection omitted this test; that coverage gap is preserved. The first correction exposed a second real source-provenance/copyback failure. After preserving the primary trigger and actor while updating only its batch, the complete workspace recovery file passed all 188 cases, including ordinary mixed-author input and native copyback. - Real HTTP queued-comment consumption, discard/reorder and Stop boundaries: 101 passed before these review corrections. Existing chat and Slack consumers each passed their selected real writer case. - Previous-head CI also timed out in the CLI child-server startup fixture and a ZIP transfer case. The full untouched transfer file passed 17 cases locally. Startup remains unclassified; its intentionally reserved port is part of the fixture and is not diagnosed as the cause. No CI retry was issued. - An earlier queue run passed its 231 assertions but timed out during teardown; its preserved replay passed all 60 queue cases. This is distinct from the behavioral reds. - Exact `ab59d944895774d89f950248e356e93af00f4832` server type layers and production server/Runner build passed. The unchanged-default serializer file passed all 11 cases. Current-head CI and both substantive reviews remain required; previous grades and checks do not qualify this head. - No hosted task was retried or changed. Local tests do not establish any hosted deferred event's cause, provider execution, native acknowledgement, physical cleanup, or serving revision. ## Risks - Admission now performs bounded row reads and locks for each coalesced wake. It adds no idle scan or remote request. The existing task transaction owns the operation; lock order follows queued claim. - Claimed-wake recovery uses source identity. Its payload remains unchanged, and the existing actual native copyback tests cover that boundary. - A source that changes before the lock can defer the new input. This preserves work instead of consuming it into a stale run. No migration or historical backfill is required. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, shell execution, and test tools. The runtime does not report the context-window size. A separate OpenAI coordinator reviewed the source. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0884e73b46 |
test(e2e): verify saved assigned skill use (#18783)
## Thinking Path > - Paperclip lets people manage agent work and reusable company skills. > - A saved assigned skill must affect a task's actual execution. > - Existing QA coverage creates and edits skills. > - The daily target needs coverage for ordinary saved-skill use. > - The task receives its receipt data only through one saved skill. > - The admitted immutable version, native read and saved document establish the outcome. > - This PR adds one bounded journey for that gap. > - A shared lease test also needs a stable clock to assert its exact timeout. ## Linked Issues or Issue Description Refs: #18305. Supporting repair #18885 merged as `057898530c702ec14c1e14d163e96dfb045cd8c0`. It closes the recovery-conflict prerequisite. This branch does not duplicate that production patch. **What existing behavior does this improve?** Product E2E coverage for ordinary assigned company skill use on an existing deployment target. **Current behavior** The deployment journey catalog can create and edit a company skill. It has no separate case that requires a native task to load its assigned saved skill and use data available only inside that source. **Proposed behavior** Create one synthetic managed company skill through the public API. Assign its canonical key with the normal Latest selection. Create the task through the production browser composer. Require one native work run, the exact created immutable version in its admitted skill-version map, a successful native source read or load, and an agent-authored saved receipt. Reload and verify the result, revision and original source. ## What Changed - Add `saved-skill-use` with one native work run and seven required checks. The first check is `saved-skill-assignment-bound`. - Use ordinary `desiredSkills: [canonicalKey]` assignment. Require the returned stored assignment to have `versionId: null`. This works with Beta skills off and reads or changes no experimental setting. - Bind the creation receipt, pre-dispatch source read and version file inventory. Require a managed skill with the exact company key. Require the run's authoritative `skillSelectionSnapshot: true` and admitted version map to match the created immutable version. - Keep the random receipt reference and numeric inputs only in the saved source. The task prompt names the skill without supplying its answer. - Require exact company, task, agent, run, turn and runner bindings for a complete successful native source receipt. Accept the deployed OpenCode read wrapper only when both the saved assignment and native execution input bind its runtime directory to the created immutable skill version. Also accept the first-party assigned-skill read with a provider-independent canonical key/version binding from both the saved API assignment and admitted native input. Require the exact full source, checksum, byte count and complete page. Keep the materialized directory-name guard specific to the OpenCode filesystem path. Reject claims, shell output, foreign or incomplete reads and wrong skill loads. - Add a narrow Codex reader for the retained controller `semantic_tool.input` and `semantic_tool.result` pair. Require the actual native provider, runner, session, call, turn and item; the admitted immutable key/version; the read-only `read_assigned_skill` operation; and its successful active-task receipt. Recompute the input digest and the exact complete seven-field read-result digest from the saved source. Raw dynamic or MCP names grant no authority. Keep the builtin OpenCode/filesystem reader body unchanged. - Require authoritative native commit before qualifying the saved receipt. Independently check its value and same-run document revision authorship. Reload and verify the result and unchanged source. Retain those existing readback receipts in the private checkpoint for independent verification. - Keep the exact raw campaign ID in the journal, API name, frontmatter and saved name. A lost successful creation response remains recoverable by name. - Register discovery, source fingerprints, closed checks and existing feature associations. Update the case and fixture docs. Feature-map files are unchanged. - Stabilize one existing shared sandbox lease regression with a scoped `Date.now` spy and `finally` restoration. Keep its exact timeout, no-destroy, no-replacement and recorded lease assertions. Change no production service code. ## Verification - **Genuine QA staging qualification passed on this exact `2fb654f7be0bac3bfeb4ec6b7a2eb7a06b8de7e4` head**: [workflow41763421118](https://github.com/paperclipai/paperclip-cloud/actions/runs/41763421118), [TestOps launch2257/result4034](https://paperclip.testops.cloud/launch/2257/tree/4034?treeId=0), [sanitized report](https://pages.paperclip.ing/evals-qa-smoke-41763421118-1/). All seven required checks and all five phases passed, with exactly one configured OpenCode/DeepSeek native run succeeded and committed. Owned cleanup closed. A separate source-pinned replay verified25 original-evidence controls and16 rejection controls without altering originals. Automatic #testops PASS was independently read back. Both earlier failed QA grades remain unchanged. - Current-head CI `41758727042` attempt 2 passed with 48 successful jobs and one skip. The coordinator authorized one unchanged-source failed-jobs diagnostic. Start and end timestamps prove three fresh executions (Runner package, server shard 7 and aggregate verification), 45 retained successful executions and one skipped non-execution. The prepared-context test passed in 2,534 ms. All 28 repository rehome tests passed. The original failed attempt remains sealed. Its timeout causes are still unknown. No source, timeout or assertion changed, and no reliability fix is claimed. - Fresh exact-head Greptile review is 5/5 with zero unresolved threads. All 58 current check-runs and the status rollup are terminal and clean. The native Paperclip review ended neutral and incomplete, so it is not a validated assessment. The PR remains mergeable after the dependency case merged in #18901. The source remains frozen. A coordinator-only qualification draft is prepared with approval false. A genuine QA staging pass is still required before daily enrollment. - Current head `2fb654f7be0bac3bfeb4ec6b7a2eb7a06b8de7e4`: 269 focused tests across eight files passed. These include 42 Codex producer, converter and false-positive controls. The canonical E2E typecheck passed with its own declaration prerequisites. Credential-free discovery lists 24 journeys. Fresh exact-head Greptile review passed at 5/5 with zero unresolved threads. CI `41758727042` attempt 1 failed. It completed 45 successful jobs, one skip and three failed jobs. The primary failures were one existing OpenCode prepared-context test at its unchanged 5-second timeout and three existing repository rehome tests at their unchanged 15-second timeouts. The third failed job aggregates those results. The timeout causes are unknown. The original attempt remains sealed. A direct passing twin ran the prepared-context test in 2,533 ms. All package, script, CLI, UI, workflow and root inputs match between the actual tested merges. Three server files differ, including `task-retry.ts` inside the broader rehome import graph; no full-server equality or causal fix is claimed. The later single diagnostic pass is recorded above. The native Paperclip review ended neutral with an incomplete execution, so it is not a validated assessment. No live QA pass is claimed. - The valid review on `e1a474` found that Codex displays its first-party tool as dynamic. Source inspection confirmed that its raw name has no authority. The retained controller semantic input/result pair supplies the real operation authority. Core `09162fe893b464dd2ccdafd14bd3ef1611202a86` and current source have identical relevant envelope, digest, result-unwrapping and immutable-read implementation regions. - Three real-producer positive controls failed against the old grader. All 42 now pass. The tests call the actual immutable-file reader, production protocol digest builders, Codex display converter and API redaction. They validate the event schema and cover omitted defaults, the exact runtime alias and Unicode bytes. Missing, duplicate, reordered, failed, foreign, MCP, unbound, partial-page, wrong-digest, wrong-key/version and nonce-only evidence fails. Generic dynamic output fails even with copied full text and invented read-only markers. - The original builtin reader body and source-only fixture function are byte-identical to `e1a474`. All seven checks, the one-run limit, 500-cent guard, ordinary Latest assignment and Beta-off behavior remain unchanged. The independent verifier is pinned to this new head. Its synthetic semantic receipt passes 24 negative controls and its filesystem receipt passes 16. Older helpers and both original failed QA seals remain unchanged. - Earlier integrated head `e1a47438b1c7e51a5bb2896a4ec9b6ed6560318b`: 376 focused checks across eleven files passed, including Saved, settings and Chat overlap controls. All seven canonical server type layers and exact-head E2E types passed. All four lease controls passed. Credential-free discovery lists 24 catalog journeys. No new staging run has occurred. - Normal integrations consume actual master `fbb7055b9020e505cd1bf903b1b1c301ba8f3eef` and actual Chat squash `d7283e875772ca8c684dc86e471e6b48d543b7b1`. Preserve all five Saved case/parser files byte-for-byte and retain settings and Chat registrations. The only manual resolutions were two additive handler/type anchors. All seven checks, one-run limit and 500-cent guard remain unchanged. The independent verifier is pinned to this exact head. - Earlier dbdf CI passed after one authorized failed-jobs recheck of a confirmed Spot-terminated Build runner. Timestamp comparison proves two fresh executions (Build and verify), 46 retained successful executions and one skip. The original failed attempt remains sealed. Its native Paperclip Review completed neutral with an incomplete execution and is not a validated assessment. - New master entries made the earlier dbdf PR conflict. The frozen qualification guard correctly blocked dispatch before any pin or receipt. Preserve its unapproved draft as history; it did not authorize a QA run. This normal integration resolves the observed conflict. Fresh current-head CI/review and one genuine staging pass remain required. - Earlier merged head `dbdf1f3cbeaacc7cf9a2a457618035c945bd09f0`: 227 focused checks passed across seven files. All seven canonical server type layers passed. The E2E typecheck passed using the declarations built in this same checkout on this exact head. All four existing clock and lease controls passed. Credential-free discovery lists 22 catalog journeys, including both saved-skill use and task lifecycle. These are local checks, not a live staging qualification. - Normal merge of master `8771606fd87f69ed4057779e3dce3a9bca8088e6` retains the reviewed task-lifecycle case and its Cursor fixture repair from #18782. Resolve six shared journey registrations by preserving both cases. The five saved-skill case files remain byte-for-byte equal to reviewed `ff3814`. All seven required checks, one-run limit, 500-cent bound and ordinary Beta-off Latest assignment remain unchanged. No feature-map delta was added. - The independent sealed-run verifier is now pinned to this exact merged head. Synthetic semantic evidence passed with 24 negative controls. Synthetic filesystem evidence passed with 16 negative controls. Foreign document authorship and missing reopen/source readback still fail. The old `ff3814` verifier and both original failed QA seals remain unchanged. - Common generated outputs were copied only after all 4,225 package, script and root inputs matched, installed lock/TypeScript hashes matched, and all 4,955 source output paths and hashes passed before and after copy. No server declarations or editable source/module links were copied. The server declarations were rebuilt locally. A full frozen workspace install disabled lifecycle scripts and used the same locked packages. No Rust build ran locally. - Previous `ff3814` CI `41733864320` exposed a real wrapped PostgreSQL uniqueness error in a recovery server test. That production defect is independently fixed and tested in #18885. The later supporting PR merged separately. This branch does not duplicate its production patch or regrade the original failure. Preserve the original failed result and the separate Spot interruption. New full CI and a fresh exact-head 5/5 review remain required. - Previously, **225 tests passed across six files** in the focused evidence, flow, contract, catalog and report set after the semantic-read correction. The prior 181-test proof and all original failed attempts remain retained. This correction changes only the saved-skill evidence helper, flow argument, their tests and case documentation. The shared clock repair remains unchanged. - E2E TypeScript check passed on Node **24.22.2**. - Earlier credential-free discovery listed 21 cases and exposes the ordinary saved-skill journey without a Beta prerequisite. These local checks make no provider or model calls. - Source inspection of Core `09162fe893b464dd2ccdafd14bd3ef1611202a86` confirms key-only selection becomes a stored null Latest selection. Managed execution resolves its current immutable version and records the admitted map in `contextSnapshot`. The public run API preserves that map independently of optional result presentation data. - Seven invalid admitted-version fixtures wrongly qualified without the new guard. The regressions now reject them before the read or document can qualify. Beta-off and Beta-on ordinary request controls use the production sync validator. Wrong assignments, missing null selection, source drift, creation-version mismatch and another skill's complete native output cannot pass. - Existing native commit, null/compact result data, read provenance, nonce, saved-result, reload and lost-creation-response controls remain required. Their original red results are retained. - The earlier experimental version-pin variant remains unqualified. Its missing-prerequisite regression and the denied target-setting attempt remain separate failure evidence. This case qualifies ordinary assignment and the version actually admitted by the server. - Normal merge of master `5f1b59ba2c9adfb40a681bf419a8b7e7837d1125` includes the reviewed inbox snapshot fix #18798 and App/Shell access-fixture fix #18799. The merge has no conflicts. All 17 journey source, helper, registration and documentation files remain byte-identical to reviewed head `eab260955ff43eeacfdafc9b75dfc9b116f0eb79`. - Previous CI `41672686614` failed two UI test shards with obsolete user-ID fixtures; its aggregate verify job also failed. The merged access-fixture repair addresses those failures. This is a new source CI run; the failed run is retained. - Dependency output reuse follows exact path and byte checks for 10,977 tracked build inputs and 3,411 generated outputs. The first copy guard detected 12 bundles regenerated after its initial manifest and stopped before any destination mutation. The stable replacement manifest passed all checks before copying. Final focused tests, E2E typecheck and discovery passed after the verified copy. No editable source or module link was shared. - An earlier integration found stale generated Cursor Cloud declarations. Canonical dependency builds refreshed them, then typecheck passed. The original typecheck failure remains recorded. - The initial report support failure came from a blocked `tsx` IPC pipe before report execution. The unchanged test passed with normal local IPC permission. The original failure and diagnostic remain recorded. - `git diff --check` passes. Full workspace recursive typecheck, unit run and build remain assigned to hosted CI and coordinator verification. Full CI `41679720041` passed on the prior `f3f288a54177f2b9a7b225b1d1ab35440dfc597c` source. A clean 5/5 review passed on `0f445cf1c622afb25a1af92398784209df542696`. Fresh CI and review are required after the shared clock fixture repair. - The first real QA staging attempt on `f3f288a54177f2b9a7b225b1d1ab35440dfc597c` failed `saved-skill-native-read-attributed`. Assignment, immutable admitted version and cleanup passed. The test stopped before document, reload and source-after observations. The original failed attempt and its complete evidence remain sealed. - The original successful builtin `read` event contained the full saved procedure. OpenCode returned its file wrapper with an unknown normalized operation, null target and a hashed runtime skill name. The old parser rejected that shape. A synthetic regression failed before this correction. The corrected parser recognizes that exact retained read in memory, with no change to the original grade. Wrong paths, copied shell text, missing starts, mismatched calls, failed/truncated reads and wrong version/name bindings still fail. A later staging attempt is described below. Neither failed original was regraded. - Original CI `41703535552` failed one shared server test and its dependent verification job. The lease-resume test expected an RPC timeout of 31,234 ms, but the runtime correctly supplied 31,233 ms after one elapsed millisecond. The test and runtime service were byte-identical to the previous passing source. The failed run remains recorded. - A scoped advancing clock reproduced that exact 31,233-versus-31,234 failure locally. A stable clock around only the acquisition call now preserves the exact original assertion. The repaired test and three related sandbox lifecycle controls pass on disposable embedded Postgres. No production timeout, lease preservation, cleanup or replacement behavior changed. - The first local reproduction stopped before test execution because six same-package native Postgres library aliases were missing. Their exact existing versioned libraries matched the canonical dependency. Restoring only those internal relative aliases allowed the separate deterministic red and green measurements. This prerequisite failure remains recorded. - The second real staging attempt on `02582aadd23fc128ae29daa69482113cf0f95ad1` also failed `saved-skill-native-read-attributed`. Authentication, setup, assignment, source-only input, collection and cleanup passed. It completed one native run. Later document, reload and source-after checks were not executed. The complete original remains sealed as failed. - That attempt used the ordinary first-party `paperclip_read_assigned_skill` tool. Its complete native receipt contained the exact canonical skill key, created version, `SKILL.md` bytes, SHA-256, byte count and null next-page offset. Core verifies the pinned manifest and file checksum before returning that receipt. The old grader recognized only filesystem and native skill-load output. An actual-shape regression failed before this correction. The corrected parser recognizes the retained read in memory. Wrong keys, versions, source bytes, hashes, partial pages, semantic failures, copied shell output and foreign or unpaired native calls still fail. No model prompt, product source or original grade changed. A live pass on this corrected source remains required. - CI `41708636059` on the previous `02582` source passed at attempt 3. The first two interrupted attempts remain recorded. AWS evidence identified Spot capacity termination for the second Rust runner. The targeted retry executed Rust and aggregate verification; 46 successful jobs retained their exact prior execution timestamps and one job remained skipped. New metadata IDs do not establish new executions. Fresh full CI and review remain required for this source. - Earlier CI `41730743279` passed on `1ee86838ea528edd8c60fb9ddd858faf3c83e783`. A later review finding correctly showed that the semantic read still depended on an OpenCode-only directory-name helper. That finding remains recorded. Three local Codex, ACPX Cursor and Cursor Cloud flow controls reproduced the rejection before this correction. - The semantic path now binds the canonical key and immutable version through the common native runtime skill selection contract. Its checksum, complete source, page and native call attribution checks stay unchanged. Missing, duplicate, wrong-key and foreign-version API/native assignment controls still fail for all three provider configurations. The OpenCode filesystem wrapper still requires its exact provider and materialized directory name. These local controls test the existing canonical receipt shape; they do not claim live provider qualification or add a guessed provider encoding. Fresh CI and review are required after this correction. - This public PR does not enroll the case in the daily plan or change protected QA configuration. ## Risks The target must supply a disposable native agent with normal company skill-write access and cleanup. It must preserve the authoritative admitted version map and provide a complete source read or named native skill-load receipt. Missing evidence fails the case even when the receipt document has the right value. The OpenCode wrapper path requires matching assignment and native input records for the same immutable version; it does not accept a generic targetless read. The builtin first-party path requires the same version binding and exact full-source receipt. Codex must provide its authoritative controller pair and a digest that commits to that exact complete result. Partial pages, unknown operations and semantic failures do not qualify. Local flow controls simulate the target API and browser boundaries. They do not establish live Cloud qualification. Experimental assignment pins need separate qualification. The journey changes only its owned skill, assignment and task. It changes no shared agent, setting, model, connection or budget. ## Model Used OpenAI GPT-6 through Codex with reasoning and code execution. The runtime does not expose a more specific model identifier or context-window size. Local checks made no product-model calls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
27a825da2f |
feat(telemetry): track the server OS with the registered contract (#18897)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its first-party telemetry uses a generated event contract and typed helpers. > - The existing events do not identify the server's OS family. > - A proposed event is dropped until the generated contract registers it. > - This PR syncs the registered contract and emits the OS family when an enabled server telemetry client starts. > - The benefit is a way to compare Windows, Linux, and macOS installations while keeping machine details out of the event. ## Linked Issues or Issue Description **Problem or motivation** We cannot compare Paperclip installations by host OS. Counting task or install events does not provide this dimension. **Proposed solution** Send one `server.started` event with Node's raw `process.platform` as `os_platform` each time the server initializes its enabled telemetry client. Compare distinct install identifiers per platform, rather than restart event totals. **Alternatives considered** A proposal-only emitter would send no data. A separate later contract-sync PR would leave the feature incomplete. This PR includes the canonical generated SDK and promotes the emitter together. CPU architecture and OS versions are outside this feature's scope. **Roadmap alignment** This is a narrow telemetry addition to existing server startup, with no new user flow. The full contract copy also adopts the three existing connector proposals described in #13578; their schemas are already registered. The generated file stays an exact copy, so their helpers and tests must be promoted in the same change. ## What Changed - Export `trackServerStarted` and call it once when server telemetry initializes. Repeated initialization returns the same client and emits no duplicate event. - Copy the canonical generated telemetry SDK byte for byte from schema source commit `5369e6b`. Remove the startup proposal marker. Normal opt-out and CI suppression apply before initialization. - Promote the three connector events included in the canonical SDK. Derive enum types from the generated map and preserve raw values. Update their real-client transport and disabled-state tests. - Assign startup to the existing `operational_enum_count` retention class. Document OS privacy, container/WSL interpretation, and distinct-install counting. - Add real-client startup transport tests and server initialization tests, including file opt-out, environment opt-out, and CI suppression. - Keep startup running when telemetry state storage fails. Publish the client and start periodic flushing only after the startup event succeeds. On failure, clean up the client, disable telemetry for the process, and log a generic warning without the storage path or underlying error. ## Verification - `pnpm --filter @paperclipai/shared typecheck`: passed. - `pnpm --filter @paperclipai/shared build`: passed. - `pnpm --filter @paperclipai/shared exec vitest run`: 98 files, 1,055 tests passed. - `pnpm --filter @paperclipai/shared exec vitest run src/telemetry`: 6 files, 39 tests passed. - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/telemetry-startup.test.ts src/__tests__/shared-telemetry-events.test.ts`: 2 files, 19 tests passed. - Exact generated artifact comparison, `git diff --check`, and local secret/PII review passed. - Proposal inventory contains no startup or connector proposals. The repository extractor fails with TypeScript 7's missing compiler API; rerunning it with the installed TypeScript 5.9 compiler API passed. No extractor source change is included. - Initial local server typecheck attempts found missing workspace dependencies and a missing plugin SDK link in this fresh checkout. Dependencies and the local SDK link were repaired; `pnpm --filter @paperclipai/server typecheck` then passed. Full repository typecheck, tests, and build are covered by the PR CI rather than claimed as local passes. Greptile found that a telemetry state-file error could abort startup. New regression tests reproduced this on the previous head with both an injected permission error and the real state store blocked by a file. Both now pass, including no published client, no periodic flushing, no send, and no retry after failure. Shared telemetry tests and server typecheck were rerun after the fix. Full PR CI passed on `a41540b7899bc8a0aedbd78d72f8b69ad1a4b318`, including typecheck, build, all selected test shards, and the final verify gate. [Successful workflow attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41752039595/attempts/2). Attempt 1 was interrupted when the server (1/12) runner received a shutdown signal; its incomplete result was retained in the PR discussion. The same-commit shard retry and dependent gate passed. Greptile reviewed this exact head at 5/5 with all threads resolved. ## Risks This change starts sending one OS family value per server startup when telemetry is enabled. It sends no OS version, architecture, hostname, path, prompt, or machine identifier beyond the existing envelope. Containers and WSL report the runtime's OS. Event totals reflect restart frequency, so use distinct install identifiers for OS adoption analysis. The complete generated SDK also enables the previously registered connector proposals. Those events keep their existing approved payloads; no connection names, credentials, tool arguments, or results are added. Privacy review is requested for both newly active paths. Deploy the registered schema before merging this app change. Live production delivery has not been tested. No schema migration is needed. Revert this PR to stop the new startup emitter and restore the prior contract. ## Model Used - Original implementation: Anthropic Claude Opus 5.5, as recorded in the original commit. - Completion: OpenAI `gpt-6.1-sol`, xhigh reasoning, through Codex in T3 Code; shell, Git, and automated tests. Context window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the feature issue template - [x] The final change and description contain no private issue links or proposal markers - [x] The existing PR branch is preserved to retain its review history - [x] I have run the targeted tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
057898530c |
fix(server): handle wrapped recovery action conflicts (#18885)
## Thinking Path > - Paperclip coordinates durable agent work and recovery. > - Independent workers can find the same task needs recovery. > - The database permits one active recovery action for that task. > - A losing insert must read the action that already committed. > - Drizzle wraps the PostgreSQL unique violation in its cause. > - The current classifier reads only the outer error and misses that conflict. > - This PR recognizes the same allowed conflict through a bounded cause chain. > - Both workers can then return the one durable action through the existing retry path. ## Linked Issues or Issue Description Refs: #18783. **What happened?** The saved-disposition-debt recovery regression failed in CI `41733864320`. Two recovery workers collided on `issue_recovery_actions_active_source_uq`. The fixture captured an unhandled PostgreSQL `23505` error. A separate canceled shard in that run lost its AWS Spot instance; that interruption is a different failure. **Expected behavior** Two service instances that race to create the same active action must return that one action. Unrelated database failures must still propagate. Existing transaction and ownership rules must stay in force. **Steps to reproduce** Hold a PostgreSQL SHARE lock on the recovery-action table. Start the same source-scoped upsert on two independent service instances. Confirm both real INSERTs are waiting in `pg_stat_activity`, then release the lock. The original code persists one action but rejects one caller with a `DrizzleQueryError`. Its outer code is absent; its `PostgresError` cause has SQLSTATE `23505` and the expected active-source constraint. **Paperclip version or commit** Supporting branch starts at master `54782706e1d83d92295246e499627930df03662d`. It now contains a normal merge of master `8771606fd87f69ed4057779e3dce3a9bca8088e6`. The classifier correction stays unchanged. The PR diff against that master still contains only the recovery service and its tests. ## What Changed - Inspect up to eight error causes with cycle detection. - Require SQLSTATE `23505` and one of the same two recovery-action constraint names on the same error. Support the PostgreSQL driver's `constraint_name` field. - Reuse the existing source-scoped retry path and its unchanged three-retry limit. - Add a deterministic real-database collision regression and no-retry controls for unrelated uniqueness, other SQLSTATEs and cyclic errors. - Keep the original recovery regression, assertions and timeouts unchanged. Keep the saved-skill journey separate in #18783. ## Verification - Review follow-up: the real collision test now checks the intended retry merge. The surviving action has attempt count 2, active status, the default unlimited budget, the expected owner, supplied evidence and supplied attempt timestamp. Its wake and monitor fields remain null. Existing identity and convergence assertions run first. - A disposable counterfactual returned the winner unchanged after the lost insert. It passed the original barrier, one-row, no-rejection and same-ID assertions, then failed the new attempt-count check with 1 instead of 2. The counterfactual was not committed. The production service was restored byte-for-byte. The actual service passed all seven focused race, no-retry and sequential controls. The affected canonical test-tree-2 typecheck passed against verified unchanged local declarations. - Parent `4f8abfe4f2244649219ef6ced3534634193f0b53` full CI `41746314291` passed all 48 selected jobs with one intended skip. Its 5/5 summary also had two duplicate unresolved reliability comments about the missing state assertions. This test-only follow-up addresses both. Fresh full CI and current-head review are required again. - The deterministic real-database regression failed before the correction. Both inserts reached the lock and exactly one active action persisted. The losing error was a `DrizzleQueryError` with a `PostgresError` cause, SQLSTATE `23505` and constraint `issue_recovery_actions_active_source_uq`. - **11 focused checks passed across two files on the integrated head** (181.72 seconds). This includes the real-database collision, no-retry controls and all four unchanged recovery producer/dependency regressions, including the original failed real-clock saved-debt case. Both callers return the one persisted action. - Current-head canonical server typecheck passed on Node24.22.2 with all seven layers: below, cycle, above, tests and test-tree-0 through test-tree-2. Canonical shared, SDK, DB and Runner TypeScript prerequisites passed. The first filtered installs lacked CLI and UI imports required by the new test layers. Those module-resolution failures remain recorded as setup failures. A full frozen workspace install with lifecycle scripts disabled supplied the same locked dependencies. No source or type-configuration workaround was used. No Rust or broad workspace build ran locally; full repository typecheck, tests and build remain hosted-CI gates. - The actual supporting master baseline reproduced the same wrapped23505 failure before the correction. Earlier configuration-resolution, local IPC and native-library prerequisites remain separate setup failure evidence. No test failure was reclassified as a pass. - First supporting CI `41739489620` was canceled before its build/test jobs completed. Its later automatic full-CI run `41739747947` failed. The recovery server shard passed. The Runner failure was the Cursor fixture case repaired upstream in #18782, now included by the normal master merge. That CI also observed an attachment-browser timeout, a repository-rehome timeout and a lost-runner connection. Their underlying causes remain unknown. A later pass will establish recovery, not explain those earlier failures. These original results remain retained. - Current `fbd392b9b65dbcf4ac078825096037ed0d490a0b` full CI [41751706634 attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41751706634/attempts/2) passed: 48 successful jobs and one intended skip. Attempt 1 remains retained: ten primary jobs lost their bound Spot capacity and the verify job failed as a dependent. A single ordinary failed-job recheck produced eleven fresh successful executions (ten primaries and verify); timestamps prove 37 successful executions were retained. The skip is not counted as execution. Current Greptile is 5/5 with both addressed reliability threads resolved and no unresolved findings. All 57 current check runs are successful, neutral or skipped, and Snyk is successful. This is CI recovery; it does not explain the separately retained older attachment and repository timeouts. - The original CI failure and both failed saved-skill QA attempts remain unchanged. This PR does not establish a live QA pass or deploy a production change. ## Risks Conflict classification must not turn unrelated errors into retries. The new controls require those errors to propagate after one insert attempt. The traversal is bounded and stops at cycles. Transaction, company scope, ownership, notifications, retry count and database constraints are unchanged. No schema or public API change is required. ## Model Used OpenAI GPT-6 through Codex with reasoning and code execution. The runtime does not expose a more specific model ID or context-window size. These checks make no model or provider calls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Michael NguyenandPaperclip
|
58d1b21098 |
fix: separate fresh setup from archived personal connections (#18902)
## Thinking Path
> - Paperclip helps people manage AI agents and their service access.
> - Each personal connection belongs to one user.
> - Setup can recover an archived connection by its name and provider.
> - A fresh setup can therefore select another user's old identity and
fail its owner check.
> - This change creates a separate connection for fresh setup after
personal archival.
> - Explicit reconnect still requires the existing owner and preserves
its history.
## Linked Issues or Issue Description
**What happened?**
A fresh GitHub setup can return 403 after a personal connection with the
same name was archived. The request does not include a reconnect target.
Setup still selects the archived identity by name and provider.
**Expected behavior**
Fresh setup creates a separate personal identity. It leaves the archived
connection, credentials, grants, and activity unchanged. Explicit
reconnect rejects a different user.
**Steps to reproduce**
1. Create a personal GitHub connection in a test company.
2. Archive that connection and its application.
3. As another authenticated member, start fresh personal setup with the
same name and provider.
4. Observe the owner-check failure before this change.
**Paperclip version or commit**
Base:
|
||
|
DottaandPaperclip
|
e2dc2cc0be |
fix(native): retire idle shared-sandbox ingress owners before dispatch (#18893)
## Thinking Path > - Paperclip manages agents and their tasks. > - Native tasks can share one reusable sandbox workspace. > - A completed child can keep its provider process warm. > - A parent has its own logical session, but the sandbox has one ingress listener. > - Retirement by task identity leaves the child's listener in place. > - A new parent Runner then fails to bind that listener. > - This pull request retires a proven idle physical owner before the parent's distinct session starts. ## Linked Issues or Issue Description Refs #15353 and #17097 for listener configuration and warm transport rotation. Refs #18682 for owned process retirement. #18860 repairs a different resume-source failure. Searches found no existing repair for retained warm sessions on different tasks that share the same physical ingress allocation. **What happened?** A completed child and a later parent can use the same reusable sandbox. Warm-session retirement compares logical task ownership. It does not find the child's distinct task scope. The child keeps the fixed ingress listener, and the parent's new process fails with `runner_ingress_bind_conflict` / address already in use. **Expected behavior** A parent must start in its own logical session after the proven idle prior listener closes. Active, foreign or unverified owners must remain protected. **Steps to reproduce** Run the `retires the completed child listener` cases in `native-session-executor.test.ts`. The actual executor registers a warm child. PostgreSQL holds its committed, accounted result and clean expired lease. A distinct parent receives the same allocation and current workspace writer. The provider boundary uses real TCP listeners. The original executor reaches the second bind with the child still listening and fails with EADDRINUSE. The changed executor waits for the owned close, then starts the distinct parent session. **Paperclip version or commit** Reproduced on |
||
|
DottaandPaperclip
|
7b1c53f8f1 |
fix(recovery): continue verified preparation after lease interruption (#18892)
Use recorded system interruption and the existing execution, controller, stopped-lease, and saved-file proof instead of a shutdown-only error label. Preserve bounded retries, user Stops, task authority, and exactly one continuation. Keep the chat recovery fixture faithful to the preparation stage and terminal time written by the real lifecycle. Retain missing-field safety controls. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
82dd3bb702 |
Preserve unstarted queued turns during offline enrollment (#18887)
## Thinking Path > - Paperclip manages AI agents and their work. > - Existing-volume enrollment moves server files to durable object storage. > - Enrollment stops application deployments before it changes file authority. > - A task drain keeps unstarted turns queued in the database. > - The migration rejected every queued row, so a busy instance could never finish enrollment without discarding requests. > - This change preserves unstarted queue intent and continues to reject running or previously started executions. > - Users keep their pending turns and saved files through the conversion. ## Linked Issues or Issue Description **What happened?** A preserving enrollment found queued turns after task admission stopped. The workspace, instruction-file, and historical-log guards treated these rows as live writers. The drain also prevented the rows from starting, so waiting could not satisfy the migration guard. **Expected behavior** After every old deployment has stopped, maintenance can move files while unstarted queued turns remain unchanged. The replacement claims those turns through normal admission. Actual executions and retained execution ownership continue to block migration. **Steps to reproduce** 1. Queue a turn behind another task. 2. Start a task drain and let the current task finish. 3. Stop the application deployments and run offline enrollment. 4. Observe `active_execution_requires_drain` for the unstarted queued turn. **Paperclip version or commit** Reproduced on the migration contract through #18864. This branch starts at `ab2d2098a2764c511978d1c3e7a3fc5dfcef07f4`. Related: #18826, #18835, #18841, #18864. Searches for queued enrollment and queued migration found no duplicate fix. ## What Changed - Share one offline execution predicate across workspace, agent-file, and historical-log migration. - Reuse the existing pre-dispatch evidence predicate; queued turns with execution markers, saved checkpoints, source events, coordinators, or environment leases remain blockers. - Keep task bindings and normal live-migration restrictions unchanged. - Exercise normal replacement admission after deleting the old filesystem: the same queued run reads restored workspace bytes and instructions, completes its original wake, and cannot execute twice. - Document the offline boundary and queued-turn behavior. ## Verification - **111 tests passed** across the final 110-case migration suite and assembled queued-turn heartbeat proof, with real PostgreSQL and local S3. This includes workspace/log migration, queued ownership blockers, ordinary agent-file controls, and the actual bundled maintenance process. - The maintenance process preserves the queued row while archiving historical output. The new assembled heartbeat test creates a normal wake under drain, migrates without changing either row, deletes the original filesystem, and resumes through normal admission. It reads saved workspace bytes and the restored instruction bundle, persists one successful native result, completes the same wake, and verifies a second sweep does not duplicate execution. Only sandbox transport/provider execution are scripted; no live model or Daytona calls. - An earlier optional run-evidence worker integration invocation was skipped without its dedicated environment; it is outside the changed offline path. - Server typecheck, server build, and runtime bundle passed. - The first maintenance test exposed the separate agent-file guard; that guard now uses the same offline predicate and the test passes. - Initial local typecheck attempts lacked CLI dependency links in this isolated checkout. Restoring dependencies fixed setup without source changes. - Full repository checks run in CI. The first head lost spot runners in Build and serialized tests; one unchanged idle-deadline fixture crossed UTC midnight and failed its timestamp assertion. Final-head CI passed on run 41744678994 after retrying a spot-runner shutdown. Greptile reviewed the exact final head at 5/5; all inline findings are resolved. The separate Paperclip Review app remains queued. Hosted enrollment remains pending release qualification and compatibility verification for the existing provider configuration. ## Risks The offline predicate relies on Cloud stopping every old application deployment. It does not make online migration safe. Running executions always block. Requeued executions with retained ownership also block. No queued request, task binding, or recovery state is removed or rewritten. No database schema changes. ## Model Used OpenAI Codex, GPT-6 based, with reasoning and tool-assisted code execution. The exact deployed model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
ab2d2098a2 |
test: verify sleep producer readiness and CI contracts (#18795)
Reviewed and verified at
|
||
|
Nicky LeachandClaude Opus 5.5
|
11ac90e31f |
fix(build): typecheck server/src/__tests__ and fix every type error (#18739)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server has about 1,100 test and helper files in `server/src/__tests__` > - `server/tsconfig.json` excludes that directory, Vitest does not check types, and no script runs the tsconfig in that directory > - So a test can call a function with the wrong arguments and still pass, and no tool reports it > - This gap hid 2,168 real type errors, and some of them hid tests that did not test what they claim > - This pull request fixes all of these errors and adds the directory to the server layers from paperclipai/paperclip#18737 > - The benefit is that a type error in a server test now fails the typecheck, with no baseline and no backlog ## Linked Issues or Issue Description Refs paperclipai/paperclip#18737, which is merged and added the server layers. Refs paperclipai/paperclip#18738, which is merged and checks the test projects outside `pnpm -r`, so it is not a duplicate. Refs paperclipai/paperclip#18137, which records that this directory has no check. **What happened?** No tool checked the types of 1,074 of the 1,101 files in `server/src/__tests__`. I searched `package.json`, the workflows, and all tsconfig files. 27 files are in the program of another tsconfig, and these 27 have 0 errors. 11 of the 27 are in `helpers/maintenance/tsconfig.json`. No script ran that project when I measured this. paperclipai/paperclip#18738 now runs it. **Expected behavior** The server typecheck checks these files, and a type error fails the check. **Steps to reproduce** 1. On the base commit, run `tsc -p server/src/__tests__/tsconfig.json --noEmit`. It reports 6,536 errors and needs 12.3 GiB. 2. Add a type error to any file in `server/src/__tests__`. 3. Run `pnpm --filter @paperclipai/server typecheck`. It passes. **Paperclip version or commit** I measured the error breakdown below on commit `8dfd632452`, with TypeScript 7.0.2. The branch now includes master up to paperclipai/paperclip#18865. **Deployment mode** Not applicable. This issue applies to development and CI. No production source file in the server changes. ### Error breakdown Each row adds one change to the row above. | Step | Total errors | Change | Cause | | --- | --- | --- | --- | | Config as committed | 6,536 | | | | Add `src/types/*.d.ts` | 4,447 | -2,089 | Config. The Express request types were missing. | | Set `rootDir` to the repository root | 2,436 | -2,011 | Config. | | Set `allowImportingTsExtensions` | 2,181 | -255 | Config. Vitest accepts `.ts` imports. | | Add `ES2024.Promise` to `lib` | 2,138 | -43 | Config. Node 24 has `Promise.withResolvers`. | | Set `allowJs` for `.mjs` imports | 2,148 | +10 | Config. It removes 8 `TS7016` errors and shows new real errors. | | Set `declaration` to false | 2,142 | -6 | Config. Tests emit nothing. | | Count only files in `server/src` | 2,058 | -84 | Other packages: 53 Daytona, 29 Runner test, 2 UI. | So 4,478 of the 6,536 errors (69%) came from config or scope, and 2,058 are real type errors in 314 files. The 29 Runner test errors are also real, which gives 2,087 errors. Tests that master added after this measurement had 81 more errors, so this pull request fixes 2,168 errors. This table classifies the first 2,058 errors. | Kind of real error | Errors | | --- | --- | | Wrong argument or assignment type | 1,023 | | Stale or partial fixture objects | 525 | | Property that the type does not have | 317 | | Missing narrowing of null or unknown | 101 | | Implicit any | 60 | | Missing name or module | 16 | | Other | 16 | ### What the type errors hid Most errors were stale fixtures and loose mocks. These errors hid real test problems, and this pull request fixes them: - **A test that never ran.** `instrumentation.test.ts` required `sdk-trace-base`, which is not installed. So 3 span tests were skipped. They now run and pass. - **Assertions that could not fail.** - `issue-thread-interactions-service` used a company ID as an issue ID. - `status-card-evaluation-work` compared `[]` with `[]` in a rollback check. - `slack-conversation-lifecycle` read `item.issueId`, which does not exist. - `email-channels` used a permission key that does not exist. - `heartbeat-project-env` imported `LOW_TRUST_REVIEW_PRESET`, which is not exported. The value was undefined. - **Postgres processes that did not stop.** 11 test files called `stop` on the test database helper. That method does not exist, and the helper method is `cleanup`. - **Arguments that the service ignored.** - `methodKey` is now `connectionMethodKey` in the Daytona, Unikraft, 1Password, and Railway tests. - 108 calls passed `"owner-user"` to a parameter that expects an options object. - 9 chat wizard fixtures passed `managementContext` as `source`, so they ran `existing_app` and not `manifest`. - 2 tests passed a `{ userId }` actor that the function ignores. - Inserts into `environments` passed a `companyId` column that the table does not have. - **Problems in tests that master added during this work.** - `sandbox-creation-operations` read `database.connectionString` when an external test database was set. In that mode `database` is undefined, so the fixture would stop with a TypeError. It now uses the external URL. - `idle-completed-history` set a Dot operation to `dispatched`. That is a Muse status, and the Dot type does not have it. The fixture now uses `admitted`, which is a real in-flight Dot status. ### Possible product problems, not changed These may be real problems in production code or types. I did not change them, because this pull request does not change production behavior: - `decisions-service` `service.create` does not validate the lenient `cancel_issue_tree` input. - `createConnection` ignores `input.connectionPurpose`. - The `createConnection` route schema default stores `{}` for `transportConfig`. - `prepareManagedAiRuntime` accepted the invalid value `mode: "connection"`. - `createBinding` accepts invalid `projectionClass` strings outside class 3. - `ensureManagedProjectWorkspace` forwards an untyped `repoRef` by spread. - `routineVariableSchema.label` is optional, but `RoutineVariable.label` is required. - `ToolRunContext.projectId` may be too strict. - `withdrawInteraction` reads `issue.status`, which its type omits. - `issueService.list` and `reconcileNativeFinalizations` omit fields that they return at runtime. - `createGitHubExternalObjectProvider` handles `tokenProvider: null`, but its type forbids this value. ## What Changed - Fixed the 2,168 type errors. The diff changes 322 test and helper files. - Corrected `server/src/__tests__/tsconfig.json`. Editors and the typecheck now read the same options and report the same errors. - Added three `test-tree` layers to `server/scripts/typecheck-layers.mjs`. They read the server layers as declarations, and any error fails them. A hash of the path selects the shard, so a new file does not move other files. - Added `server/scripts/typecheck-stubs/daytona-sdk.d.ts`. Server tests reach Daytona plugin source, and the root install does not have `@daytonaio/sdk`. - Changed two imports in `ui/src/lib/issue-execution-policy.ts` to use `.js` extensions, because server tests import this file. - Made `declarationsConfig` and `--declarations-only` skip the `test-tree` layers, because these layers emit nothing. Other projects read only the source layers. Added a test for this. - Updated `doc/DEVELOPING.md`. ## Verification Run these commands: ```sh pnpm --filter @paperclipai/server typecheck pnpm exec vitest run server/scripts/typecheck-layers.test.mjs ``` Results on macOS (arm64), Node 24, 16 cores. Each check ran after the merge of paperclipai/paperclip#18738 unless the row says otherwise: | Check | Commit | Result | | --- | --- | --- | | Server typecheck, `PAPERCLIP_TYPECHECK_LOW_MEMORY=1`, all 7 layers | `6e45a08573` | Exit 0, 0 errors, 62 seconds, peak 3.52 GiB | | `node scripts/typecheck-test-projects.mjs --low-memory` | `a3306bd13e` | Exit 0, 0 errors, 42 seconds, peak 3.25 GiB | | The same script with `--reuse-declarations` | `1b878aabc0` | Exit 0, 0 errors | | `typecheck-layers.test.mjs` and `live-recovery-work.test.ts` | `01db5634e4` | 90 passed. These include real compiler runs for a pass, a type error, and an invalid tsconfig. | | `tsconfig.declarations.json` after a full server typecheck | `0fb1e4fb0a` | It references only `below`, `cycle`, and `above`. No generated config names a `tsconfig.layers.json` file. | | Owned runtime test in `idle-completed-history` | `e8f2f448ac` | Passed, with the fix that master later replaced | | `idle-incoming-wake`, `plugin-idle-manager`, and the changed `chat-channels` test | `6e45a08573` | 30 passed | | Runner `native-execution.test.ts` | `eab4dcc1f4` | 114 passed | | The 19 server test files that changed since `eab4dcc1f4`, and `typecheck-layers.test.mjs` | `259737b700` | 4,275 of 4,277 passed. The 2 failures are in `chat-channels` and are in the list of 7 below. | The peak is the largest single process, from `/usr/bin/time -l`. Peaks for the server typecheck were between 3.42 GiB and 3.56 GiB across 8 runs. Peaks for the test projects were 3.25 GiB and 3.37 GiB across 2 runs. I did not measure a cgroup. **Runtime check of the fixes.** Types do not prove that a test still works. So I ran all 309 changed server test files, 11,929 tests, before the rebase onto paperclipai/paperclip#18737: | Result | Tests | Detail | | --- | --- | --- | | Passed | 11,913 | Includes the 2 tests and the 54 `dot-runner` tests that passed in a later run, as described below. | | Failed | 7 | The same 7 tests also fail on the commit before the fixes. | | Skipped | 9 | These tests have their own skip conditions: `sentry` 5, and 1 each in `bundled-plugin-registration`, `customer-success-cloud`, `onepassword-connection`, and `workspace-runtime-https-live-exercise`. | The 7 tests that fail with and without this change, on my machine: - 3 tests in `chat-channels` and `email-channels` stop with `ENOENT` on a `SKILL.md` file. The path is in the parent checkout, above the git worktree that I used, so I expect that the worktree is the cause. I did not prove this. - `workspace-instance-cleanup` compares `/private/var/...` with `/var/...`. This is a macOS symlink difference. - 2 tests in `git-reproducible-managed-checkouts` get `unknown_base_git_files` and expect `unpushed_managed_checkout`. I did not find the cause. - 1 more test in `chat-channels`, `materializes direct external-chat finals once`, gets `undefined`. I did not find the cause. CI is the check for these 7 tests. `ci / verify` passed on `eab4dcc1f4`. Failures during this work: - My fix to `agent-runtime-connector-access` changed an invalid permission key in the setup, but not in the assertion. The runtime check found this. I corrected the assertion, and the file passes. - `heartbeat-plugin-environment` failed one wait in the full run. It passed three times alone, so I think that load caused the failure. It can fail again under load. - `dot-runner` skipped all 54 tests in the full run because `cargo` was not on my `PATH`. It then passed 54 of 54 with the toolchain on the `PATH`. - My first `lib` value removed the default DOM types, and the error count went up by 375. I added the DOM libraries. - My Daytona stub with bare `any` left 48 errors, because generic helpers then inferred `unknown`. I added shapes that return promises. - One fix to `agent-secrets-routes` used `class_3_static_lease`, and the test failed. That class allows only specific paths. I reverted to `unclassified`. - `ci / General tests (server (8/12))` failed on `259737b700` because of the flaky port in `idle-completed-history`. paperclipai/paperclip#18853 fixed this test on master, so this pull request keeps that fix. **Audit of the diff.** I counted these patterns in added and removed lines, to find fixes that only hide an error: | Pattern | Added | Removed | | --- | --- | --- | | `@ts-ignore` | 0 | 0 | | `@ts-expect-error` | 0 | 1 | | `.skip` | 0 | 0 | | `as any` | 9 | 9 | | `: any` | 11 | 2 | | `as never` | 8 | 2 | | `as unknown as` | 35 | 4 | | `Promise<unknown>` | 16 | 5 | | `expect(` | 128 | 113 | No hunk removes more `expect(` calls than it adds. Three removals of test content were intentional, and I checked each one: - `muse-control` removed `signalToken`. Line 1012 already asserts that it is absent. - `merged-pr-confirmation-sweep` removed legacy aliases. Other tests cover them. - `heartbeat-workspace-branch-containment` removed a `finalize` branch that no test row can reach. Some partial fixtures stay unchecked through `as unknown as` and `Promise<unknown>`, mostly in `issue-agent-mutation-ownership-routes` and `issue-comment-reopen-routes`. **Accuracy of the check.** One program over the tree and the three shards reported the same errors at the same positions. Memory, as the peak of the largest process on one thread, was 3.93 GiB for one program and 2.43 GiB for the largest shard. I measured these two values before the fixes. Not run or not proven: - I did not run the full `pnpm test:run`, or a root `pnpm build`. I expect CI to run the full suites. - I did not run the 4 GiB container check, or the full `scripts/typecheck-low-memory.mjs`. The peaks above come from macOS, so they are not a Linux cgroup measurement. - I did not run the UI typecheck after the merges. - `workspace-runtime-https-live-exercise` is skipped, so its `cleanup` fix did not run. ## Risks - **A large diff in test files.** 328 files change. A pull request that edits the same tests can get merge conflicts. New tests on master can also add type errors before this merges. Any such error fails this check and needs a fix here. - **A higher cost for changes to source types.** A change to a type can add errors in test files that the author did not touch. The author must then fix them. This is the purpose of the check, but it is new work. - **Changed test behavior.** Some tests now run paths that they did not run before, such as the 9 chat wizard fixtures and the 3 span tests. They pass here, but they can be more sensitive to timing or to the environment. - **The Daytona stub can go stale.** The stub is written by hand. If the plugin uses a new SDK member, the typecheck reports an error until someone updates the stub. - **The release path.** `release-verify.yml` and `scripts/release.sh` run `pnpm -r typecheck`, so they now run this check. - **The tsconfig change.** The test transform reads `server/src/__tests__/tsconfig.json`. The options that I changed are for type checks, and the runs above passed. - **A gap that remains.** Tsconfigs in `packages/paperclip-runner` and in most sandbox provider plugins exclude `*.test.ts`. This check does not read most of those test files, so they can still drift. That work is separate from this pull request. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with reasoning, tool use, code execution, and subagents. The runtime does not show the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (7 tests fail locally, with and without this change. See Verification.) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
dcc0878b32 |
perf(agents): save healthy account settings without a runtime probe (#18690)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The agent runtime page saves account, model, and environment settings. > - The save handler starts a computer and runs a live account test before it writes the edit. > - A slow computer or provider can make a healthy account save take tens of seconds. > - Saved account health, access, installation, and compatibility can be checked without a runtime test. > - This pull request uses those checks for ordinary configuration saves. > - Legacy adoption, task adoption, and Copilot model checks keep live verification. > - The benefit is a fast save while Test and execution retain runtime checks. ## Linked Issues or Issue Description **What happened?** Saving a new AI account selection on an existing agent can take 20–30 seconds. Model edits can take the same path. The configuration PATCH acquires a computer, prepares account credentials, runs a provider/runtime probe, and releases the computer before it saves. **Expected behavior** A healthy installed account can be saved after current access and compatibility checks. A save confirms persistence. Test verifies the runtime. An unverified legacy account must still pass live adoption checks. **Steps to reproduce** Run the PostgreSQL regression with a 20-second controlled delay at computer allocation. On the original route, a successful save takes 20,191 ms and calls allocation and the runtime probe once. With the first patch, the same fixture takes 51 ms; with the environment-guard correction, it takes 37 ms and calls neither operation. Each timing has one sample. This is a local fault experiment with real HTTP and PostgreSQL. It is not a hosted latency measurement. Related: #17760 addresses binding comparison and prospective probe environment resolution. This change deliberately removes the live probe from ordinary healthy-account saves. It keeps live checks for legacy and task adoption. #17965 fixes the result of an explicit managed Runner test. Neither change removes this save-time wait. #18687 optimizes a different Apps configuration read. ## What Changed - Replace the internal probe boolean with explicit none, legacy, and always modes. - Use legacy mode for ordinary configuration saves. Keep current account health, access, installation, compatibility, environment, audit, and revision checks. - Keep the environment company check before the live-probe branch. Keep the live task-adoption transaction and Copilot model checks. - Add twelve real PostgreSQL save-route regressions, including company-bound environment checks. Keep the existing inherited-sandbox adoption fixture for legacy and task adoption. - Cover healthy and legacy settings changes separately. Keep provider rejection and reconnect tests on the legacy save path. - Update the specification and feature/access maps to distinguish saving from runtime verification. The configuration route owns this policy. The account service still owns account access and health. The existing environment test service still owns live probes. No new endpoint, migration, dependency, background job, or provider protocol is added. ## Verification Base: `3f8ebfff8aa9febd0733c6f65b07485dcadfd0e2`. - Original route with initial regressions: three failures and two passes. Healthy saves reached computer allocation. The controlled 20-second baseline saved successfully but failed the no-allocation assertion. - Initial save/adoption/Copilot selection: 21 passed. The name filter excluded 141 unrelated tests. The seven new cases cover persistence, revision and activity records, model edits, roster edits and stale retries, missing installation, revoked grants, legacy verification, and controlled allocation latency. - Explicit Test-route suite: 55 passed. The first attempt reached the 30-second import hook limit. The same suite passed with a 90-second hook limit. - Saved harness preflight suite: six passed. Account revocation, compatibility, probe failure and target release checks remain. - `node scripts/check-maps.mjs` and `git diff --check`: passed. - Review revision: both full save/account suites passed, 96 tests total. The controlled save took 37.08 ms with zero computer allocations and zero provider probes. The five new environment checks passed after the three foreign-environment cases failed against the first patch. - Local full `pnpm -r typecheck`: unavailable because cargo is missing. - Local full `pnpm build`: unavailable at the verified Copilot runtime prerequisite. Server typecheck preparation also reaches missing cargo. - Local full `pnpm test:run`: stopped after concurrent broad checks exceeded the container memory limit. No full-suite pass is claimed. Direct server TypeScript checking and its bounded retry were also killed by the container memory limit. The first remote Typecheck + Release Registry check passed; the next head still requires fresh CI. Reproduce the controlled timing: ```sh cd server PAPERCLIP_CONFIG_SAVE_PROFILE_DELAY_MS=20000 pnpm exec vitest run src/__tests__/agent-configuration-save.test.ts --hookTimeout 90000 ``` The fixture keeps real Express, authorization, account selection, PostgreSQL persistence, revisions and audit writes. Only remote allocation and the provider probe are controlled. The default tests do not contact a provider. Successful configuration saves no longer depend on either controlled operation. A later explicit Test or task must still verify the actual runtime. Review follow-up: the first Greptile review found that the new healthy-account path skipped the environment company guard. The corrected route checks the effective environment before it decides whether to probe. Five real PostgreSQL cases cover foreign, same-company and global environments, inherited defaults, and an existing account binding. Against the first patch, the three foreign-environment cases returned 200 and failed their expected 403 assertions; the two allowed cases passed. Initial fixture attempts hit unique environment constraints; the fixture now uses distinct sandbox rows and names. The first CI server shard found 11 tests that required a live probe for every save. Those expectations are changed to the documented healthy/legacy split, with additional healthy-selection cases. The build and Runner jobs were cancelled after this test failure. Their logs show no separate compile or test verdict. No cancelled check is counted as a pass. Latest commit `6257e9e67f289e1c650a8b4c2b818fcca37e2800`: [CI run 41542980091](https://github.com/paperclipai/paperclip/actions/runs/41542980091) passed typecheck, build, all test shards, Runner checks, and E2E. All 55 check runs are complete: 52 passed and 3 were skipped by their workflow conditions. The commit status check also passed. [Greptile review](https://github.com/paperclipai/paperclip/pull/18690#issuecomment-6590555746) is 5/5 for this commit with no new findings. The earlier environment ownership thread is resolved. These remote results complete the checks that the local tool and memory limits prevented. The earlier local failures remain recorded above. ## Risks A healthy saved account can become invalid at the provider before the next runtime use. Ordinary saves no longer detect that through a live test. They still reject unhealthy, revoked, incompatible, or uninstalled accounts in current storage. Explicit Test and execution retain provider authentication and project-authentication checks. Legacy-account adoption, task connection adoption, and Copilot account-specific model checks keep their existing live validation. There is no migration. Reverting the route change restores the earlier synchronous probe behavior. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. This runtime does not expose the exact model ID, context-window size, or reasoning setting. Those values are not inferred. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites; full-suite limits are above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
049c1eacf5 |
Preserve historical native logs during ephemeral enrollment (#18864)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral web servers need retained run history in shared storage.
> - Some old native runs have surviving local logs but no completion
digest or verifiable native source coverage.
> - Enrollment must preserve those bytes without calling them a complete
native transcript.
> - This pull request adds a read-only archive path to offline
enrollment.
> - Users can still read old output after the original server filesystem
is removed.
## Linked Issues or Issue Description
**What happened?**
Offline enrollment blocks on interrupted runs and unfinalized historical
native logs. The current importer requires canonical native-event
coverage even when the original local file survives.
**Expected behavior**
Preserve the exact surviving file in shared storage. Keep the original
run status, completion metadata, and source events. Do not allow writers
or native repair to modify archived history.
**Steps to reproduce**
Use an S3-configured instance with a terminal native run whose local log
lacks a final size or digest. Stop all app deployments and run ephemeral
maintenance. Before this change, interrupted status or unproven native
coverage prevents enrollment.
Related preservation work: #18826, #18835, #18841. This is a bounded
migration fix, not a new live logging protocol.
## What Changed
- The run-evidence migration owner can archive unfinalized native local
files during offline enrollment.
- Require the offline migration environment, zero active or queued runs
across the instance, and expired controller/finalizer leases. Check
global quiescence at each page boundary. Recheck each run's ownership
and metadata before publication.
- Verify stable local file identity and metadata, any recorded
size/digest, uploaded prefixes, and remote SHA-256.
- Store raw history in a separate `<run>.archive` journal with source
cursor zero and backend `s3_archive`. Leave completion fields and native
events unchanged.
- The existing log store reads archives but rejects begin, append,
enqueue, flush, and finalize. Native source repair does not run for
archives.
- Keep the default background importer's coverage checks. Continue to
reject incomplete mirrors and corrupt files.
- Add real PostgreSQL/S3 and packaged maintenance coverage. Correct a
pre-existing large-chunk fixture: the current verifier can prove exact
compactor output, so the negative case now uses a different canonical
source.
## Verification
- Passed on final head
|
||
|
DottaandPaperclip
|
683fa4725e |
fix(evals): refresh native runtime and pin reviewed catalog repairs (#18780)
## Thinking Path > - Paperclip lets people inspect and manage their agents’ work. > - The timeline must report canonical ownership history, and OpenCode tool failures need bounded diagnostics. > - The measured candidate also had independently reproduced question-navigation and oracle defects. > - This change promotes the reviewed production fixes and pins one immutable, qualified source composition. > - Original queries, model, counts, clocks, privacy controls and qualification holds remain intact. ## Linked Issues or Issue Description Refs #18775, #18777, #18779, #18711, #18713, #18718, #18699, #18798, #18799. **What happened?** The measured candidate predates merged native-runtime fixes and contains fixture defects. The timeline omitted recorded reassignment/removal history. Finish-input errors lacked a closed correlation and shape witness. **Expected behavior:** project recorded ownership transitions without inventing facts, distinguish successful native execution from failed verification steps, and retain content-free finish diagnostics. **Steps to reproduce:** repeat an unchanged assignee while editing another field; then reassign or unassign it. Only recorded transitions should appear. Invalid OpenCode finish input should produce a correlated rejection witness without values or schema text. Related #18699 adds gated company OTLP diagnostics. This change uses the existing local callback; it adds no exporter or capture policy. ## What Changed - Promote canonical timeline creation, reassignment and removal projection from #18777. Initial child ownership requires a recorded fresh-creation receipt; delegation requires a readable recorded parent. Unknown legacy facts fail closed. User actors/owners require active company membership; agent actors/targets must exist in that company. Spans, events and edges share that admission filter. Remove the introduced all-history absence query and use guarded indexed UUID lookups. - Add the shared/OpenAPI `unassigned` event kind, preserving newer main contracts and issue-route/service code through reviewed semantic joins. Existing UI consumers remain generic; no new visual marker is claimed. - Add bounded OpenCode finish diagnostics: schema digest, hashed call identity, closed shape enums and validation category. A throwing sink does not change admission or cancellation. - Pin immutable candidate `fd5e3b1e5a379524c0d1651ed8273e75ffd86541`, composed from combined #18777 source `b734393eff0d7ef85b7ad8abd556d2a66ca3a4e6` plus the reviewed MCP diagnostic delta. It preserves actually merged #18775 `97c07d149b5670088faf15317dcf603c75e2af87` and #18779 `a352c057d475067b4fe67375f06785287dfbe0ce`, and merged runtime fixes #18711, #18713 and #18718. - Qualify narrative handling for adjacent repair/verification sentences, intervening native-failure wording, and negated repair objects. Questions, denials and hypotheticals retain their controls. Preserve Previous question beside the qualified Entity type import; separately restore the already-called `internalResultLink` goal helper binding. That import is a runtime fixture repair, not a type-only change. - Include the narrow older-baseline CI typecheck closure and causal heartbeat test observers in the candidate. All four default typecheck projects execute. Main already includes the protected upstream tooling and newer heartbeat fixture, so neither is overwritten with an older candidate file. - Preserve all 244 identities, original queries, model, native/human counts and clocks, including the explicit prior Studio accounting exception. The full catalog stays on its candidate branch; the 196/244 gate and historical SDK runtime holds remain. ## Verification The primary’s f478 retry repeated the real composer-hidden five-second assertion (1 failure,18 passes,1 skip). Investigation established a setup readiness gap: an outer route response can delay mounting the chat thread beyond the assertion. The successor adds the existing real-thread `aria-busy="false"` readiness barrier before the unchanged composer assertion. The original120-second scenario and digest, unread, source-link, history, and reply checks are unchanged. The full owning browser scenario passed naturally in10.2s and with its first real chat response held6000ms in12.0s. This synchronizes fixture setup; the underlying AWS late-mount latency remains unproved. **Current main gates passed:** exact head `823dab49fcd8470140c178771e5c0c6b78725e97` has [successful normal CI](https://github.com/paperclipai/paperclip/actions/runs/41724620555), including both required aggregates, fresh Greptile 5/5 and zero unresolved review threads. Paperclip Review is neutral with a completed substantive 5/5 and no actionable defect. The current optional Superagent Security Scan is neutral/inconclusive: Security Graph context retrieval returned HTTP503, so its assessment did not start. That result supplies no security PASS credit. The primary source PR retains its separate current-head gate before Root sequences normal protected merges. - Reviewed MCP controls: 130 on main and 129 on the older candidate. Meaningful parent comparisons failed before implementation/correlation correction. Controls cover actual offered schema identity, null/omitted admission, bounded rejection categories and throwing-sink cancellation. - Exact unchanged production postimages retain 63 Postgres/public-route controls, nine plan controls and the focused index-plan proof. Owner OpenAPI passes 32 controls; main’s newer contract passes 33. Earlier setup/sandbox failures remain recorded; unchanged database controls were not replayed. - Narrative owner controls pass 216, retaining all prior 207 controls. The independent peer passes 144 balanced cases; earlier 108 and 39-case matrices remain retained. Actual consumer parent failures reproduced the review findings before repair. - Main’s server NoEmit passed on its protected baseline; only two pin files changed afterward. The candidate’s prior full server and four-project checks passed. All server/package/script/root-lock inputs remain byte-identical; the changed Runner project’s guard passes on the current source. Compatibility driver controls pass 47 and six owning suites pass 82. The one explicitly reviewed Docker-only unit omission concerns an unimported image-build feature; all four intended project guards still execute. - Pin/adapter/contract controls pass 47. All 242 expanded declarations, the two legacy dispatch rows, and all 244 singleton private policies retain their authenticated identities. Descriptor, launcher and private-protocol bytes remain exact; declaration loading invoked zero fixture callbacks. - All 506 SDK input blobs match the meaningfully compiled `6abe5739` source vector. Prior real shared/SDK builds measured declaration/map changes and zero runtime-JavaScript changes; a compiled consumer accepts the new kind and rejects an unknown kind. The reviewed registry remains unchanged. Default source proof, 51 SDK controls, 20 custody controls and 16 UI consumer controls remain qualified; runtime/compiler/parser/Zod/canary pins and held runtime grades are unchanged. - The full Mac heartbeat run on prior observer source retained **58 passes / six failures in 802.78s**. Two nonassignee controls subsequently pass focused checks. Four late timeout causes remain unresolved locally; no flakiness/infrastructure cause or complete local pass is claimed. [Actual AWS CI](https://github.com/paperclipai/paperclip/actions/runs/41712023737/job/126182620050) ran this exact heartbeat file with **64 passes in 107.209s**. Its server/package inputs are unchanged in the current narrative-only successor. The source-identical f478 heartbeat file also passed all64 AWS tests in105.812s; source inputs remain unchanged. Current main ordinary CI passed; the primary source PR must independently close its latest-head ordinary CI gate. - Historical main [run 41713049555](https://github.com/paperclipai/paperclip/actions/runs/41713049555) recorded six synchronized runner-shutdown failures and cancellations; the shutdown cause is unknown. A separate chat passive-control assertion observed one `db.transaction` call where zero was expected. Finite local probes passed 2+2 but did not establish a cause or late stack. No external publication, cleanup cause, source fix or flakiness is inferred. The primary’s separate composer-visibility CI failure also remains an inconclusive historical result; no guessed fix is credited. No old-head retry was used. - Earlier UI/inbox fixture failures and the CLI startup timeout remain historical. UI/inbox repairs are now upstream in #18799/#18798 and main preserves those files verbatim. The CLI timeout cause remains unknown. Broader local repository/Rust qualification is not claimed; fresh hosted CI supplies broader coverage. - All four actual repair merges, an exact merged-tree/diagnostic source join, and fresh controller/source/private-policy/active-run guards precede the ten-case TestOps measurement. No paid or model eval was launched for this PR; the original three passes/seven failures remain unchanged. ## Risks The next ten-case model result is unproved. Earlier MCP timeout and continuation-shape causes remain unknown; diagnostics provide future evidence without relaxing validation or rewriting grades. Prior local/CI failures are retained, including the unresolved chat assertion and local heartbeat timeouts. Initial ownership requires canonical in-window audit facts. Missing legacy facts remain unknown. The additive `unassigned` contract is synchronized across shared/OpenAPI/server consumers, with no new UI marker. SDK runtime holds and the broader 196/244 qualification gate remain. Normal protected merges require fresh checks and resolved feedback; no CODEOWNER or branch-protection bypass is authorized here. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: repository analysis, code changes, source validation, and local test execution. The context-window size is not exposed by this harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — current focused/type/pin controls pass; the prior full Mac heartbeat failures remain disclosed and receive no local PASS credit - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — current main normal CI and required aggregates passed; optional neutral reviews and separate primary gate are disclosed above - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — exact current main head; full thread pagination verified - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
451d06e2fc |
fix: resume interrupted Runner preparation and confirm failure report delivery (#18853)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns run recovery and task continuation after a restart. > - A shutdown can interrupt Runner preparation before any provider work starts. > - Workspace recovery can save the files but leave that task behind a legacy execution hold. > - Retry then accepts a request without starting a successor. Startup also leaves the task blocked. > - This pull request verifies the interrupted preparation, preserves its files, and admits one bounded continuation. > - It also keeps a dropped Sentry capture eligible for a later report. ## Linked Issues or Issue Description **What happened?** A hosted task remained blocked after a graceful server shutdown interrupted Runner preparation. Its remote workspace was recovered. Retry returned an accepted request, but no successor started. Startup did not resume it. The task also had no matching Sentry event. The recovery cause is verified. The exact cause of the missing historical Sentry event is unknown. A separate regression with the real Sentry SDK proves that processing backpressure can drop a capture while the task cache marks it as reported forever. **Expected behavior** Retry must continue eligible interrupted work from its saved workspace. Startup must schedule one bounded retry when durable evidence proves that no provider started. A later task read must be able to retry an unconfirmed Sentry report. **Steps to reproduce** 1. Interrupt Runner preparation during a server shutdown before native runtime selection. 2. Retain and recover its remote workspace under the legacy recovery hold. 3. Press Retry, or restart after file recovery has completed. 4. Observe that the task remains blocked without a successor. For the reporting case, hold the first Sentry event in its asynchronous enrichment step. Fill the SDK processing buffer. Capture a second task failure, release the first event, and read the second failed task again later. **Paperclip version or commit** Reproduced against source commit `4d31588dea`. The hosted incident served `6ef37398c8d799c5142a3a6590ecc70ad352b663`. **Deployment mode** Hosted Linux server with remote workspaces and external PostgreSQL. Local regressions use the source checkout on macOS arm64, Node 26.4.0, PostgreSQL 17.10, and the audited optional Sentry SDK 10.71.0. Related public changes: #18541 added task-state failure reports. #18678 retained remote exports before Retry. #18803 fenced remote workspace writers before continuation. The small backend-port fixture correction follows #18739. It is included here because the same existing setup defect blocked CI. ## What Changed - Verify pre-native shutdown interruption against the immutable adapter claim, controller ownership, run events, native records, stopped leases, and exact workspace recovery receipts. - Retire only the server's unchanged bootstrap failure block. Admit explicit Retry through the existing continuation path. - Discover completed workspace recovery at startup and schedule one retry under the existing failure budget. Let saved input own its continuation. - Apply the existing safe replacement dispatch checks to preparation retries. A later human block, competing execution, dependency, pause, approval, or budget gate must still prevent dispatch. - Keep execution evidence available through the bounded run projection. - Confirm Sentry task-report delivery from a successful SDK HTTP response. Keep unconfirmed captures pending for 60 seconds before a later task read can retry them. Both caches remain bounded and start no timer. - Document recovery and report delivery semantics. - Align existing handoff/budget assertions with the guarded preparation retry reason. Keep the owned-runtime fixture outside reserved proxy ports, with bounded allocation attempts. ## Verification - Red before the recovery fix: all three new Retry/startup cases failed because no successor was created. Green after the fix: all three passed with real saved workspace archives and PostgreSQL. - The pre-merge revision passes all 384 tests in the six affected recovery, run-control, run-state, retry, and dispatch suites. - The late-block regression failed before its repair. Its four focused recovery/dispatch cases then passed, including final dispatch cancellation. - Review found a second stop check that still rejected verified local and lease-free preparation. Both new cases failed before that repair. All 17 interrupted-preparation regressions passed on the pre-merge revision. - The Sentry regression failed with the real SDK before the fix. All 91 affected reporting tests then passed. The transport stays in memory. No test sends events to Sentry. - Full repository typecheck passed with `node scripts/typecheck-low-memory.mjs`. The full `pnpm build` and module boundary check passed. - CI found untyped delivery-hook parameters when the optional Sentry SDK was absent. Explicit callback types fix that build. The SDK-absent server build and all 91 SDK-present reporting tests pass. - Server typecheck passes with the SDK installed and the real Sentry SDK contract CI job passes. Greptile reviewed the pre-merge commit `103e175c44` at 5/5 with no open findings. - The local full-suite attempt initially overlapped the rebase and stopped at test-line selection. Its unchanged-head rerun passed the first 309-test chat shard. I stopped the old-head run before applying review fixes. Current-head CI owns the remaining full-suite verification. - CI exposed two old retry-reason expectations. The handoff and retry-budget checks pass with the new guarded reason. A separate existing process-adoption fixture chose port 42953 in the reserved 42000–42999 proxy range. The fixture now rejects reserved ports, following #18739. Its real process/adoption/termination test passes locally. - The final Runner package job [initially failed](https://github.com/paperclipai/paperclip/actions/runs/41718507919/job/126205912033) one unchanged Codex credential case after it could not acquire its port quorum, then timed out. The complete unchanged 42-case credential suite passes locally. One unchanged CI retry passed; the original competing port owner was not identified. No Runner implementation or credential test was changed. - Required CI passes on `103e175c4490e1bdf804c64a8635b0e4568545ae`, including `ci / verify` and `ci / e2e`. That revision has Greptile 5/5 with no open findings. The optional Paperclip Review check on that earlier head remained queued, waiting for its assigned agent. - Merged current `master` without replacing published commits. The retry gate retains both shutdown interruption checks and the new workspace preparation timeout policy. All 311 tests in the five combined recovery, retry scheduling, completion, and native startup suites passed after the conflict resolution. Server typecheck passed. Final head `005d380b00d45a2393c4a2a190eab27611168d3c` includes the next conflict-free master update. All 17 interrupted-preparation regressions passed again on that final head. Greptile reviewed that exact head at 5/5 with no new findings. Required CI passes on that exact head, including `ci / verify`, `ci / e2e`, typecheck, build, the full test shards, and the real Sentry SDK isolation check. The optional Paperclip Review remains queued, waiting for its assigned agent; it is not a required branch check. - On the final head, six test jobs were cancelled after their spot runners received shutdown signals at the same time. Another server shard passed 2,419 tests, but its unchanged 103-record cleanup backlog case exceeded the 15-second limit. The unchanged full cleanup suite then passed all 29 tests locally. One unchanged retry of only the seven unsuccessful jobs passed. The [original cancellation and timeout logs](https://github.com/paperclipai/paperclip/actions/runs/41724898038/attempts/1) remain in the workflow history. No cleanup implementation or test was changed. - The signed-in hosted UI reproduced the failed Retry before the fix. The fixed path is covered through the real HTTP Retry route and assembled heartbeat/recovery services. This PR has not been deployed to the hosted incident. A live repaired Retry remains unverified until rollout. ## Risks - Automatic continuation must not repeat provider work. It requires positive preparation evidence and rejects retained provider, process-launch, coordinator, result, live-controller, unsafe lease, or mismatched workspace evidence. - Repeated recovery calls must create one successor. Tests cover concurrent requests, repeated startup visits, pending Retry, retry exhaustion, and changed task ownership. - Sentry delivery state stays local to the process. A restart or cache eviction can report a failure again. An unconfirmed event needs a later task read to retry. - No schema or API contract changes are required. ## Model Used OpenAI GPT-6 through Codex. This session does not expose a more specific deployment ID or context window size. Reasoning, code editing, test execution, and browser diagnostics assisted the change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2920d88034 |
fix: preserve cancelled runs through native finalization (#18857)
## Thinking Path > - Paperclip helps people manage AI agents that do work. > - Native runs can accept a result before workspace file finalization ends. > - A user can Stop the run while file finalization is still pending. > - The direct finalizer replaced that cancelled run with the earlier provider status. > - The restart projection already preserves a cancelled run. > - This pull request preserves cancellation in the direct projection too. > - Saved results and normal task decisions remain available. ## Linked Issues or Issue Description **What happened?** An accepted successful result can wait for workspace finalization. If run-only Stop completes first, the run and its source wake become cancelled. Late finalization then changed the run to succeeded, cleared the cancellation error, and replaced its finish time. The wake remained cancelled. **Expected behavior** Preserve the winning run cancellation, finish time, errors and audited acknowledgement. Keep the accepted provider result and complete file finalization. The server task arbiter still decides task status from the accepted result. A valid result can finish the task even though the execution was stopped. **Steps to reproduce** Accept a native result through the control-plane port. Hold an actual workspace-finalization operation. Stop the run through the heartbeat service with a board request identity. Release the operation and call the direct finalizer. The new database regression exercises this sequence. Related work: #18804 settles exact claimed native wakes. This change preserves a wake that Stop already cancelled. #15212 and #12301 address the opposite race in legacy cancellation. They do not change this native finalizer projection. ## What Changed - Evaluate run status and errors against the current locked database row. Preserve a cancelled run when a late accepted result finishes finalization or when workspace/status-decision retries exhaust. Merge finalization failure metadata into the current row so concurrent receipts survive. - Keep finalization receipts and accepted results. Preserve existing server task-arbiter behavior and accounting fields. - Add real database tests for successful and failed accepted results, delayed workspace work, and Stop after the task decision but before run projection. - Check cancelled wakes, audited acknowledgement, no successor, restart projection and idle recovery. Update execution documentation. ## Verification - Before the production change, the actual producer sequence failed: the cancelled run became succeeded, its error code became null, and its finish time advanced. This failure remains saved privately. - The focused projection, cancellation-request and Stop-accounting suites pass all 67 tests. These include the same delayed-finalization sequence, both task status orderings, and complete and late accounting receipts without another provider turn. - Independent review found a second terminal writer in retry exhaustion. On the unchanged previous head, two real database cases changed a cancelled run to failed or succeeded after workspace or status-decision retries. A third case let actual board Stop win after the permanent workspace failure admission snapshot and still changed cancelled to failed. All three now pass. Existing uncancelled exhaustion controls still pass. No mock supplied the canonical cancellation state. - Initial review found that routine regression notes do not belong in the feature maps. Both map additions were removed. Production and test code are unchanged by that correction. - Runner TypeScript build and all server typecheck layers pass. Map, module-boundary and diff checks pass. - An additional direct typecheck of the normally excluded projection fixture reports two existing notification mocks whose transaction parameter is typed as `object`. The same check with the previous test file reports the same two errors. The new regression cases add no type errors. The standard server typecheck remains green. - The previous head passed required CI after one targeted retry of jobs that received the same runner shutdown signal. Those original failures remain recorded. The exhaustion correction passes fresh required CI on exact head `84404d34b9d982d8bca24b63ae2046051f5169b8` (workflow 41726120483, attempt 1). Both required aggregate gates pass. Greptile reviewed this head at 5/5 with no new comments; all review threads are resolved. An independent source review found no blocking issue in the correction. Runner TypeScript and all server typecheck layers pass locally; the exact production server and Runner build passes with the server stamp bound to `84404d34b9d982d8bca24b63ae2046051f5169b8`. The full required CI workflow passes. - This is local fixture evidence. It does not establish hosted recovery, a remote provider stop, or a deployed version. ## Risks - Accepted result finalization can still finish the task after run-only Stop. This is existing server policy; cancelled run API write authority stays revoked. - The fix protects cancelled execution history. It does not invent provider termination, saved-file or accounting proof. - No schema, endpoint, retry budget, provider call or new background worker changes. ## Model Used OpenAI `gpt-6.1-sol`, Extra High reasoning. The model used code execution, repository tools and test output. The host does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f187de99aa |
Preserve domain deadlines across owned idle sleep (#18656)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
0a1e4c3d0f |
fix(workspaces): retry pristine remote setup timeouts (#18852)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task needs a workspace before its model can start. > - Fresh workspace preparation can reach its transport deadline. > - Failure handling records that timeout as a generic setup failure and stops the task. > - An unpublished workspace has no saved task work to repeat. > - This change routes that narrow failure through the existing bounded retry scheduler. > - Saved work, retained executions, and normal admission gates keep their existing protection. ## Linked Issues or Issue Description Refs #18827, which preserves remote timeout diagnostics. This change builds on that merged repair. Related #16727 concerns controller-managed clones and optional partial clones; this change keeps remote clone contents and deadlines unchanged. **What happened?** A fresh remote workspace reached its 300-second command limit before model execution. The task then stopped with `setup_failed` and no scheduled continuation. A separate full clone completed in 137 seconds on a disposable Daytona sandbox. The original delay remains unexplained. This change repairs the missing recovery path; it does not claim to remove the underlying delay. **Expected behavior** A fresh, unpublished workspace preparation timeout can use the existing two automatic retries. Retry scheduling remains durable and company scoped. Stop, pause, dependencies, budgets, task ownership, saved work, and retained native execution must still prevent unsafe work. **Steps to reproduce** 1. Start preparation for a fresh remote workspace with no saved version or published allocation. 2. Make its initial directory/clone command return a transport timeout. 3. Observe that the base code records a generic setup failure without bounded recovery. 4. With this change, the owner validates current workspace authority and preserves a specific bootstrap failure for the normal retry scheduler. **Paperclip version or commit** Base: `e0276e0166`, including #18827. **Deployment mode** Native Paperclip Runner with sandbox workspace storage. Verification uses real PostgreSQL, filesystem commands, and a separate live Daytona transport probe. ## What Changed - Recognize only the initial preparation timeout while the same company, writer, and epoch still own a pristine workspace. - Keep restore, additional repository, provisioning, save, changed ownership, dirty workspace, and retained native execution failures outside this recovery policy. - Persist the typed failure before using the existing retry scheduler, admission gates, and attempt budget. - After successful terminal lease cleanup, reclaim an unpublished bootstrap-only reservation even when a provider returns no stop receipt. Preserve receipt requirements for saved/published work, unknown execution, and native identities; do not synthesize termination evidence. - Prevent generic recovery from granting another budget after exhaustion. Keep a specific recovery notice. - Add regressions for persistence, concurrent scheduling, restart, exhaustion, Stop, pause, dependencies, fresh allocation, lost replies, and saved output after cold restoration. - Document the recovery contract. ## Verification - Red: the real-PostgreSQL workspace regression first failed because the timeout had no recovery classification. The completion regression then failed because it became `setup_failed`. A retained-native negative failed before the admission guard. The review follow-up reproduced a failed successor reservation after production lease release returned void. - Green: all 127 transport/archive/real-PostgreSQL workspace cases pass after the review repair. All 12 receipt/void handoff cases pass against the final query. The 141 complete completion/retry/recovery cases and the fresh-allocation selection case pass. Safety cases cover concurrent scheduling across restart, the two-attempt cap, Stop/pause/dependency gates, native identities, saved/published files, active leases, failed cleanup, and cold-restored output. - `node scripts/typecheck-low-memory.mjs` and `pnpm build` pass at `ba96cff771707415e4a5e893c8b3a97499ead25b`. `git diff --check` and the module-boundary check pass. The full local test wrapper completed all 1,835 chat cases across six partitions, then was interrupted during its serial general-server phase after the complete current-head CI suite passed. Its interrupted run is not a full local-suite pass; its log is retained. The owned process tree and database children exited. - The [complete GitHub Actions run](https://github.com/paperclipai/paperclip/actions/runs/41718108375) is successful at this exact head, including all server/workspace/chat partitions, all nine serialized groups, all eight browser shards, build, typecheck, native Runner, package, Rust, and release-policy checks. The first Rust worker shut down during cache restore before tests; one retry passed without source changes. Expected Storybook skips remain skips. - Earlier failures remain accounted for: the first typecheck found two incomplete process-result fixtures, now corrected. An intermediate completion run exposed a test double that returned no promise and a teardown timeout; the complete 141-case suites subsequently passed. The first-head CI had four worker shutdowns and one worker communication loss. A three-second monitor settlement wait failed in CI and locally, then all 21 cases passed unchanged at base and head. An earlier CI retry found a real-process history adoption assertion failure; that exact case passed locally at base and head. Both server groups pass in final-head CI. These timing causes remain unknown. [Earlier CI evidence](https://github.com/paperclipai/paperclip/actions/runs/41713712522). - Live supporting evidence: unchanged production preparation code completed a full repository clone in 137 seconds using Daytona SDK 0.203.0, Node 24.22.2, Git 2.39.5, and Linux x86_64, with 1 CPU, 1 GiB memory, and 1 GiB disk. The approved diagnostic account could not access the original fleet snapshot, so the probe used public `node:24-bookworm`. Both probe allocations were deleted and absence verified. Full task acceptance at the fixed revision remains unverified: the original fleet snapshot and an explicitly configured disposable Codex subscription fixture were unavailable. No CLI login or hosted subscription was copied. - Greptile is 5/5 at the final head with no actionable findings. The Paperclip review finding is repaired, replied to, and resolved. The separate **Paperclip Review** check remains queued. Its task failed workspace export after publishing the first assessment; the staging service was then observed showing a workspace recovery page after an update failure. Subsequent browser rechecks timed out, including a fresh tab, so the normal recovery control could not be used. Reconcile that existing export after service access returns, then let the saved current-head review request run. That external review is the remaining check; this PR is not yet fully green. - The original hosted task was last verified blocked. This PR has not been deployed and the task has not been retried by this work. Reviewers can run `pnpm test:run`, `node scripts/typecheck-low-memory.mjs`, and `pnpm build`. The remote-workspace integration suite additionally needs an owned disposable PostgreSQL database named `remote_workspace_*test` through `PAPERCLIP_REMOTE_WORKSPACE_TEST_DATABASE_URL`; otherwise that suite is skipped. ## Risks Each retry keeps the existing preparation deadline, so persistent slow clones can consume the bounded budget before intervention. A retry receives fresh allocation admission; it does not replay the command against the old allocation. The workspace owner checks persisted authority before classifying the failure, and normal admission still rechecks that authority. Historical generic setup failures do not gain automatic replay. No database migration, provider-pack update, model fallback, credential change, or deployment is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser tools. The host does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
909d86b59a |
fix(native): retry runs that fail before provider work (#18854)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs start the provider through Runner. `session.open` starts the provider session before any prompt. > - When `session.open` fails, the server records the generic `native_session_interrupted`. Recovery evidence then reports `ambiguous_state`, so the first failure blocks the task with no automatic retry. > - The failure is often temporary. On one instance an agent's provider account hit its usage limit. For about 14 hours every later run failed at `session.open`, and 47 tasks stayed blocked. Outside limit windows, the same error also occurs once and a later retry succeeds. > - The server already has a bounded same-run bootstrap retry for failures that are proven to come before provider work. It did not recognize this failure, and it did not consider a known provider limit. > - This pull request classifies a definite `session.open` failure as a pre-session failure. Bootstrap retries back off and wait out a known provider limit. The task is blocked only after the shared budget is used, with a clear reason. > - The benefit is that a temporary startup failure or a provider limit no longer strands tasks on the first failure, and every existing replay guard stays in force. ## Linked Issues or Issue Description Refs #18847. That pull request keeps the provider cause on `session.open` rejections and names typed categories (for example `classification=provider_rate_limit`). This change does not depend on it. With #18847, a typed limit at `session.open` uses the same cooldown path, because the server reads the cause with `providerFailureFromError`. Refs #18622, which added the `ProviderFailure` cause. Refs #13993, an open change about legacy adapter startup backoff; it does not touch the native recovery path. **What happened?** A native run (`paperclip_runner`) failed at `session.open` before any prompt: `ACPX sidecar command session.open was rejected (retryable=false, classification=session_ensure_failed)`. The server recorded `native_session_interrupted`. Recovery evidence had no checkpoint and no provider event, but the cause was not in the pre-session list, so the mode was `ambiguous_state`. The coordinator went to `terminal_failure`, a board-owned `active_run_watchdog` action was created, and the task was set to `blocked` on the first failure. During a provider usage limit, every later run of the agent failed the same way, so each task was blocked. **Expected behavior** A run that provably failed before provider work retries automatically, with backoff and a bounded budget. If the agent has a known provider limit, the retry waits for the cooldown. The task is blocked only when the budget is used, and the recovery action explains why. A run where provider work may have started is never replayed. **Steps to reproduce** 1. Make Runner reject `session.open` for a native run (for example a provider at its usage limit). 2. Observe the coordinator: `terminal_failure`, `recoveryMode: "ambiguous_state"`, `providerSessionEstablished: false`. The task is `blocked`. 3. The new `native-startup-retry.integration.test.ts` reproduces this with a real database and the executor. **Paperclip version or commit** `master` at `5877d751c0e`. **Deployment mode** Native Runner with an ACPX Claude provider. The fix is in the server and is not specific to one deployment mode. ## What Changed - `nativeSessionFailureSourceCode` returns a new cause, `provider_session_open_rejected`, when Runner reports that `session.open` failed or was rejected. Those outcomes are definite, and the command runs before any prompt. An `indeterminate` outcome stays `native_session_interrupted`. Earlier, more specific causes (for example a missing runtime package) keep their classification. - `nativeProviderRecoveryEvidence` treats the new cause as pre-session. The durable evidence still decides. A bootstrap retry happens only with no provider session, no provider event and no checkpoint. Otherwise the mode stays `ambiguous_state` and the task is blocked as before. - New module `native-runtime/native-startup-retry.ts` plans each bootstrap retry. It is called only for `bootstrap_retry` mode. - Backoff: the existing `infrastructureRetryDelayMs` (about 30 s, then about 60 s, plus or minus 20 %). The jitter derives from the run and attempt, and it is persisted with `nextAttemptAt`. - Limit cooldown: evidence is a `rate_limit` or `quota_exhausted` provider failure on the current failure (the #18847 shape), or on an agent run that started in the last 24 hours. A `native_provider_usage_limit` or `provider_quota` error code also counts. The cooldown is the provider's `retryAt` (or `retryNotBefore`), or one hour after the failure when no reset time is known. A later successful run of the agent clears older evidence. The cooldown is capped at 24 hours. The retry runs after the cooldown plus the backoff delay. - A failed evidence read is logged and falls back to plain backoff, so it cannot stop the failure from settling. - The budget does not change: three provider attempts, shared with checkpoint resumes and replacements. When it is used, the coordinator records `native_session_retry_exhausted`. The task is blocked, and the board-owned action says: "Provider startup failed on 3 attempts before any provider work began. Automatic retries are stopped." When a limit is known, the message includes the expected reset time. - While a retry waits, the coordinator detail, the recovery action evidence and the scheduled wake record `providerCooldownAt`. The next action names the reset time, and the task projection shows "Retry scheduled". - `PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS` (one hour) moves to `system-recovery-state.ts`. The legacy provider-quota recovery default now refers to it. The value does not change. - A chat retry of an exhausted `provider_session_open_rejected` run is accepted like the other pre-provider causes. The existing proofs still apply. - `doc/execution-semantics.md` describes the backoff, the cooldown and the exhausted state. ## Verification - Red: before the fix, the first new integration case failed. The coordinator was `terminal_failure` with `recoveryMode: "ambiguous_state"`, `originalFailureCode: "native_session_interrupted"` and `providerSessionEstablished: false`, and the task was `blocked`. Four more cases failed the same way. The two never-retry cases passed before and after the fix. - Green: `npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts` in `server` (embedded PostgreSQL, the real executor, and the real resume claim). 8 cases pass: - a rejected `session.open` is retried with backoff (attempt 1 in 24 to 36 s, attempt 2 in 48 to 72 s), and the task stays `in_progress` with an agent-owned action and a `resume_native_run` wake; - after three attempts the task is blocked, the action is board-owned with the clear reason, and the resume claim refuses another attempt; - a recent limit failure holds the retry until one hour after that failure, and the claim refuses it before the cooldown; - a provider reset time from an earlier run, or from the startup failure itself, sets the cooldown; - a distant reset time holds the retry for at most one day; - a later success, another agent's limit, and an expired limit do not delay the retry; - a durable provider event, or an indeterminate `session.open` outcome, still blocks the task with `ambiguous_state` and no retry. - `native-session-executor.test.ts`: all 1,087 tests pass, including a new classification case for `failed`, `rejected`, `indeterminate` and a different command. - `chat-channels.integration.test.ts -t "requires terminal-safe native ownership for an exact chat retry"`: 23 pass, including the new `session_open` case. That case fails without the list change. - Related suites pass: native session resumption, native provider failure, capacity and overload integration tests, provider failure classification, native status arbiter corpus, native restart recovery. - `npx tsc --noEmit -p server` passes. `git diff --check` passes. - [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/41713988964) is green on head `32561aa335d`. This includes `ci / verify`, `ci / e2e`, typecheck, build, all server, chat and workspace groups, all serialized server suites, and all Runner checks, which run the native-runtime server tests. In the first attempt, eleven jobs stopped because their runners received a shutdown signal; no test reported a failure. A rerun of those jobs passed. - A local run of the whole `server/src/services/native-runtime` directory was stopped before completion because it runs serially for a long time. CI ran those files. - Runtime dependency: the new work is one indexed query (`company_id`, `agent_id`, `started_at`) with `LIMIT 20`. It runs only when a bootstrap retry is planned or exhausted, not on the normal path. No new timer or polling: the wait uses the existing persisted `nextAttemptAt` deadline and the existing resume dispatch. - Not verified: a live provider account at its usage limit, and a live remote sandbox that waits through a full cooldown. The full repository suite was not run locally; CI ran it. ## Risks - A task that waits for a cooldown stays `in_progress` with its execution lock, for up to one day. This is the existing same-run retry state. Stop, reassignment, pause and an explicit Retry keep their current behavior through the existing claim checks. - During the wait the failed run keeps its environment lease in the same way as any same-run retry. A remote lease with an unexported prepared workspace is retained by the orphaned-lease sweep until the retry runs. Other active leases of the failed run are released after the usual five minutes, and the retry acquires a new one. - A `rate_limit` failure without a reset time (for example a short HTTP 429) delays only startup-failure retries of that agent, by up to one hour. Normal runs are not delayed, and a later success clears the evidence. - A startup failure that needs user action (for example a sign-in failure) now uses two automatic retries, about 90 seconds in total, before the task is blocked. Before, it was blocked at once with an ambiguous reason. - The agent's other queued runs are not held during a cooldown. Each one fails at `session.open` without provider work and then waits for the same cooldown. An agent-wide admission hold has no clean extension point in the current queue, so it is not part of this change. - Re-arming consumed scheduled monitors is out of scope. This change makes it matter less: a run that consumed a monitor wake now retries itself on the same run. A monitor that a run consumed before its retry budget was used stays consumed. - No schema or API change. The new error code appears on runs and in run error codes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with extended reasoning and tool use (shell, file editing, test execution). The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
4bdcd1cffc |
fix(recovery): release holds for unstarted dispatch cancellations (#18838)
Release false recovery holds for legacy dispatch cancellations whose retained evidence proves execution never started. Keep audit history, current admission gates, review-wait Retry, and retained workspace obligations. Share the execution-absence proof across cancellation policies and make the dispatcher error-code policy exhaustive. Add PostgreSQL coverage and stabilize the browser history fixture by waiting for pause cleanup before seeding it. Fixes #15483 Refs #18085 Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com> Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2a16dcf629 |
fix(workspaces): preserve shared directory trees during enrollment (#18835)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A volume-free web server needs existing task files in durable object
storage.
> - Historical workspace records can share a physical folder or contain
other task folders.
> - A task-level migration hold cannot fence every alias, and capturing
a parent recursively duplicates nested task data.
> - Offline enrollment imports children first and verifies their durable
coverage before omitting them from a parent archive.
> - Snapshot retention preserves those child versions for as long as a
covering parent depends on them.
> - Existing files, task bindings, and archived states remain intact
through the storage migration.
## Linked Issues or Issue Description
Builds on merged #18826.
**What happened?**
The per-workspace importer could capture a shared project folder while
another workspace record still owned the same path. It also recursively
included runtime-created nested task workspaces in each parent import.
Packaged enrollment rejected existing workspace data instead of invoking
its migration owner.
**Expected behavior**
Require offline maintenance for overlapping paths. Import nested tasks
before their parents, verify every excluded directory against its saved
snapshot, preserve dependencies through retention, and resume
interrupted imports without replacing existing remote authority.
**Steps to reproduce**
Create a project workspace and another task workspace below its
`.paperclip/worktrees/` directory. Try importing either while ordinary
writers are running, then run offline enrollment. Interrupt a parent
import after upload and resume it.
## What Changed
- Add physical-overlap admission checks and an offline entry-point
guard, including global running and queued executions.
- Import pending records in leaf-first pages through the existing
migration owner.
- Verify nested snapshots by downloading, restoring, and comparing the
original source before and after parent upload; check transitive
dependencies.
- Publish parent dependency receipts under child snapshot row locks.
Retention pins those children and revisits them when the parent is
deleted.
- Index imported-child dependency lookups with a partial GIN index; a
PostgreSQL query-plan regression covers 2,500 unrelated snapshots.
- Propagate cancellation through capture and object storage operations
and preserve existing retry journals.
- Wire workspace migration into packaged maintenance and document its
preservation contract.
## Verification
- The original overlap and duplicate-parent regressions failed before
implementation.
- 73 migration and retention tests pass against real PostgreSQL and
files, including interrupted uploads, resumed batches, equal-path
aliases, changed or missing child data, nested descendants, and active
or queued executions.
- Six S3-compatible storage integration tests pass, including migration
through the maintenance owner and restoration of both parent and child
after deleting the original filesystem.
- The standalone process case was initially skipped because the worktree
had no bundle. After rebuilding the server and runtime bundle, that test
passed against PostgreSQL and S3: authenticated ready/export receipts,
preserved decision secret, blocked application routes, and clean SIGTERM
shutdown.
- Server typecheck, server build, runtime bundle build, and
module-boundary checks passed. Full repository CI passed on
|
||
|
DottaandPaperclip
|
5877d751c0 |
feat(server): bind idle credential eligibility to incoming wake proof (#18639)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
0b87308d5d |
fix(native): defer warm continuation during predecessor finalization (#18811)
## Thinking Path > - Paperclip manages agent work and preserves its saved results. > - Native warm sessions can continue on a retained remote Runner. > - A completed predecessor can still have a live finalization lease. > - The continuation must wait for that owner before it can start work. > - Two admission checks treated that temporary lease as ambiguous state. > - This pull request uses the existing deadline-based wait for both checks. > - The same business run can continue once, with its saved context and gates. ## Linked Issues or Issue Description **What happened?** A compatible remote warm continuation can fail before provider work starts. Its predecessor has completed, but a finalizer still holds a live lease. The warm-session check reports `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. The earlier retained-sandbox check has the same problem on the command transport. **Expected behavior** Keep the unstarted continuation on its existing run. Wait until the recorded owner deadline. Recheck ordinary ownership, Stop, credentials and checkpoint evidence before execution. **Steps to reproduce** 1. Complete a remote warm turn and retain its controller records and provider session. 2. Give that completed predecessor a live finalization lease. 3. Admit a compatible follow-up with full local controller projections. 4. Before this change, admission fails permanently. It must instead persist a bounded pre-provider wait. **Paperclip version or commit** The PostgreSQL regression failed before production edits on `adb49a4434`. The current head is `5762dfbe4e`, based on master `e491b23eb5`. **Deployment mode** Native remote sandbox execution. No database migration or provider artifact change is required. Related: #18718 refreshes stale warm-controller projections. #18343 adds controller replacement recovery. #18554 restores suspended remote sessions. This change handles a live predecessor finalizer after those source checks. The shared historical error does not identify its failed predicate. This regression proves the finalizer race; it does not prove that every historical instance of that error had this cause. ## What Changed - Defer a compatible warm continuation while its completed predecessor has a live finalization lease. - Use the same wait in the preceding retained-sandbox read check. - Preserve the current run, provider attempt count and checkpoint. Reuse the existing resumption scheduler. - Add PostgreSQL cases for both finalization phases, lease expiry, company scope, incompatible configuration, unfinished commands, Stop and duplicate resumption claims. - Add an opt-in three-turn Linux Runner case. Update the child-controller fixture to own and register a local API listener. - Document the waiting behavior and acceptance case. - Repair two readiness barriers in the refresh-worker integration fixture after a required CI shard failed twice: wait for paginated startup and for active worker promises to settle before checking idle SQL. ## Verification - **Native regression RED:** before production edits on `adb49a4434`, the PostgreSQL boundary case received the permanent identity error instead of `NativeControllerRecoveryWaitingError`. - **Regression GREEN on the current head:** all **71** focused recovery, settlement, heartbeat safety and refresh-worker cases pass on `5762dfbe4e` with Node 24.21.0 and PostgreSQL 18. They cover incompatible identities, company/configuration mismatches, expired or missing deadlines, unfinished commands, Stop, retained files and duplicate resumption claims. - **Real process acceptance on the current head:** the three-turn `finalizer-wait` case passes in 58.01 seconds. PostgreSQL, TLS, a rebuilt Linux Runner and the deterministic Codex app-server fixture confirm zero attempts and no provider turn during the wait, one claim resuming the same run, retained history, a healthy third warm turn and final suspension. The sibling `warm-suspended` case from master also passes, in a fresh container, in 46.02 seconds. Both runs use `5762dfbe4e`; the artifacts were rebuilt from that checkout with the repository's pinned Rust 1.97.1 toolchain. - Acceptance used Node 24.21.0 and glibc 2.41. Earlier probes exposed the missing fixture API listener and the preceding retained-sandbox guard; both are corrected. A stale disposable listener and an incompatible libc image failed setup before the successful fresh-container run. These setup failures are not the original regression's RED. - Repository typecheck/build passed locally before the rebase. Server typecheck passes after the rebase. **Full CI passes on the current head:** 48 successful jobs and one expected scope skip, including typecheck, build, Runner checks, all serialized-server shards, general tests, E2E and the final aggregate. Module-boundary and whitespace checks also pass. - Broad local inventories are incomplete: six chat shards passed (1,834 tests), but untouched server/UI/adapter cases hit timeouts. The exact shared-base expiration case passes alone; a full-file rerun passed all 89 assertions but timed out in teardown. A Node 24 rerun timed out in another lifecycle case and teardown. A later PostgreSQL recheck could not allocate shared memory; this task's redundant broad controllers were stopped and their logs retained. - CI attempt 1 failed a Cursor Cloud cancellation deadline, an E2E cleanup cancellation with a PostgreSQL lock timeout, and the refresh-worker startup settling limit. One failed-only retry passed Runner and E2E; the same refresh startup case failed again. Two aggregate jobs had previously stalled in the hosted-runner queue and blocked the retry; that obsolete attempt was cancelled. No repeated blind rerun is being used. - **Readiness repair RED/GREEN:** real PostgreSQL 18 behind a loopback proxy with 20 ms delay per direction reproduces the exact startup error before the test change; the same probe passes afterward. An initial controlled probe hit the dedicated-database naming guard before tests ran and is excluded from RED evidence. The full-file check then exposed a separate false-idle barrier: a worker could be running while its scheduler had no timer. The fixture now also checks the existing active-work tracker. All **24 cases pass** with both repairs. The at-most-six scoped SQL reads and zero recurring idle-SQL assertions are unchanged; startup alone has an explicit bounded condition wait. No production behavior or timeout changes. - Greptile and Paperclip Review are both **5/5 on the current head**, with no open findings. Both reviews used source inspection; the runtime evidence above was obtained separately. The previous review workspace was recovered on the first production attempt after repairing its partial-clone export. Its preserved result and real follow-up both finalized successfully, saved snapshots and released their writers and finalization leases. Three current-head CI shards received hosted-runner shutdown signals and were cancelled. All three passed their one justified retry. The queued aggregate had to be cancelled to schedule that retry; the final aggregate also passed. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41706617909). - This acceptance does not qualify a live Codex model or Daytona API run of the native repair. No deployment or recovery of the original task is claimed. ## Risks - A wait must not grant a second executor. The current attempt stays at zero until the existing scheduler claims it. Expired or missing deadlines retain their existing hold. - Saved state must survive the wait. Admission still requires the same company, task, session, configuration and physical sandbox evidence. - Current Stop, budget, credential and permission gates remain with ordinary resumption. No polling loop is added. ## Model Used OpenAI Codex, GPT-6 family. The runtime does not expose the exact serving model ID or context-window size. Capabilities: reasoning, repository inspection, code editing, shell tools and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1f55073f96 |
fix(cursor-cloud): support API-only controllers on managed stacks (#18310)
## Thinking Path > - Paperclip helps people manage AI agents at work. > - Cursor Cloud hosts execution behind its API. > - Product setup still expected a Paperclip Computer. > - Managed-only tenants therefore rejected valid setup. > - This admits the validated API-owned controller. > - Authority, Stop proof, budgets and cleanup remain enforced. ## Linked Issues or Issue Description Refs #18068 and #18069; both dependencies have merged. **What happened?** Managed-only setup still requested a Computer. Live tests exposed missing Cursor Stop proof and duplicate replay. **Expected behavior** Use Cursor’s API controller. Verify provider/controller termination and replay each log identity once. **Steps to reproduce** Enable native Runner/managed-only. Configure Cursor model/repository/key without a Computer, then Test, finish setup, run and Stop. ## What Changed - Derive API-only placement from validated native Cursor configuration; reject placement/command/workspace/home overrides and keep Kubernetes-only execution closed. - Skip Computers/checkout/workspace realization. Keep scoped tools, completion, private staging, budgets, leases and recovery. Persist placement; reject contradictions. - Align Cloud cards, Test, Save and Finish setup; hold unready drafts. Legacy keeps its Computer, credential and policy checks. - Verify Stop from exact API ownership, audited intent, provider-terminal proof, no Computer/process ownership and controller retirement. Recheck immediate/deferred retirement; wake existing recovery on durable evidence. - Clear warnings when the actual stopped target later has authoritative confirmation. Keep missing/other-run negatives and the existing predicate. - Publish safe status refresh after full-proof ACK. Retain bounded native/physical cancellation history; add no poller or weaker proof path. ## Verification Core merged as `3910352c923a`; Product merged as `27d35314be8e`. Public branch `5f86eb66b081` has tree `67344151d303`. Its exact-head Greptile summary is 5/5; the fresh audit found zero unresolved threads and complete pagination. The controlled workspace retry and required summary passed. All 59 current-head contexts are terminal and clean. Prior green-36 evidence keeps its identity. [Current CI](https://github.com/paperclipai/paperclip/actions/runs/41680511731) first failed workspaces-a 1/2: the CLI company-import/export beforeAll 30-second runtime-info/child-PID/HTTP readiness wait expired. All 431 UI files/4,714 tests and 285 other CLI tests passed (one skip). The cause remains UNKNOWN. Child output records startup recovery complete, but buffered CI prefixes do not establish its relation to the waiter deadline; runtime-file/PID/last-health state was not captured. Both unchanged local probes were RED. The second observed dependency preflight still pending at the deadline; observer overhead was not quantified. Neither proves the CI cause. After all functional jobs finished, only the queued metadata summary was cancelled; the other 48 records stayed unchanged. Attempt 1 retains 46 successes, the original failure, one skip and the cancelled summary. One same-head debug retry was admitted at 18:42 UTC on November 3. Its workspace job passed at 19:23 UTC on the same checkout: UI 431 files/4,714 tests and CLI 33 files/286 tests, including company import/export. Test caching was disabled. The new `ci / verify` summary passed; workflow attempt 2 completed SUCCESS. Required `ci / e2e` also passed. The final check at 19:53 UTC on November 3 found all 59 current-head contexts terminal and clean: zero pending and zero failing. The 47 inherited completed/skipped rows are not new executions. No code, assertion or timeout changed; success does not explain the original failure or erase the local REDs. CI checkout is `1ea9d8608ced`/tree `78cb1ec13d59` (parents `2e37c73c05fe`, 5f86); both workflows use 2e37. Its 20 extra paths add low-memory build/Rust task-memory. Scripts cohere; later matrix changes only labels, with no newer debug-cache steps. Static audit found no blocker. Supported API arguments omit the memory flag: no default monitor/instruction changes. Flagged external launchers can change Cursor instructions/digests and remain unqualified. The merge added 41 master-identical paths, retained the green-36 inbox hunk and found UI fixtures matching upstream. Startup/backup, recovery projection, sleep diagnostics and binary backup change production. Static audit found eight Stop/ACK owner blobs and 37 sleep-work SQL templates unchanged, with sleep fences/single-flight accounting intact. Exact precommit tree: 416/416 across eight server files, binary backup 1/1, server types PASS; repeated scopes are not summed. The deferred-retirement P1 reproduced three real-DB/owned-process failures; missing proof stayed unverified. Recovery passed 47 process cases/33 proof negatives. Warning repair binds actual targets to same-run proof. Validation passed 729 scoped UI cases, a separate 359-case thread run, 111 backend cases (84 CAS, 6 assembled, 21 activity/privacy), types/tokens. Proof validators/predicate are unchanged; no new poller or increased timeout. The d8d3 attempt stays FAILED: 45 passes, two workspaces-a failures, derived verify failure and one skip. Both shards shared three failures: two successful mocks omitted the account guard’s required userId. The two-line fixture repair changed no guard, assertion or timeout. Three failures reproduced; 54 access/auth tests then passed, including account-change/denial negatives, plus UI types/tokens. The 69285 attempt stays FAILED: 49 jobs, 46 passed, serialized shard 9 failed, its verify aggregate failed and one selection job skipped. The shard passed 575/576 cases. Its aborted-snapshot case expected nine computes but saw eight: a final HTTP 200 coalesced while seven ordinary reads had not proved the aborted compute settled. Earlier CI and the unmodified local baseline passed with a miss; no local RED is claimed. The repair awaits the snapshot, allows exactly one extra GET if coalesced, then requires a miss and retains nine computes/all HTTP429 assertions. No production code, hook, loop, sleep or timeout changed. Six inbox and seven cache/admission cases passed locally, plus server types; 30 unrelated cases were filtered. Green 36 passed 576/576 without taking the extra-GET branch; local runs also missed that schedule. Current CI event merge 1ea9 now passed 576/576 and naturally exercised the former failure: coalesced (count 2, 1ms), then exactly one fresh miss (count 1, 16ms), with strict nine-compute/HTTP429 assertions intact. No instrumentation or retry was added. Earlier compatibility passed scoped remote/inbox, routine, API-placement and UI tests, plus types/migrations/snapshots/maps/tokens. The 89-path adoption restored fleet routing without workflow overrides or added skips. Each result retains its source identity. Live qualification remains actual `9af1a5de42a2`: unchanged Artifact grader PASSED 17/17 and active-Stop grader PASSED 21/21. All three provider test resources were deleted canonically with GET404; post-delete download returned the exact 27 bytes. Histories stayed stable (Artifact 121 and Stop 67 canonical identities), with 21 company runs, zero active work and no accounting/cancellation/reservation debt or integrity findings. Saved settings reopened; no-prompt Test/Linux admission passed with `claude-sonnet-4-5`. No new Save is claimed. The first Stop sampling case stays FAILED on a 10.373-second gap. Unchanged retry kept all 24 samples: 35.076 seconds quiet, maximum gap 4.584 seconds, plus a separate CANCELLED/artifact-absent read after 180 seconds. Initial grading rejected locator spelling before functional checks; metadata changed only after same-file/inode/byte proof, with raw evidence/oracles unchanged. Original Question/Plan/reconnect and historical deleted-artifact results keep their identities. Git trees verified 11 history-only conflicts and six earlier mainline paths. Hosted 9af to the public BRANCH differs in 222 paths; 13,500 other mode/blob leaves match. Runner runtime implementations and six observer/driver pins match 9af on that branch only; inherited `drivers/acpx/runtime-host.test.ts` differs. No all-Runner/src, CI/latest-master runtime or image equality is claimed. Later static audits through `8ab89f41e499` found no concrete API-only blocker. That base adds completed-history sleep proofs and an MCP cleanup migration; protected Stop/ACK owners are unchanged. These later changes are outside this CI and have no live qualification here. Host duplicate-card and later recovery/UI repairs retain separate proof, not deployed 9af identity. Direct-key coverage requires absent aiConnection; API-only placement skips inherited remote-workspace disk-full paths, and config-read gates match their parent. Other retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; title/startup/custody/Stop/deletion unknown causes and bounded recoveries; superseded warning/50 duplicate identities; pre-repair ACK/Core CI failures. Fixture RED/GREEN passed 7/7 browser and 121/121 shutdown tests with a failing broken-unsubscribe control; stable WSP wrapper passed 128 after physical teardown failure. No full local-suite pass is inferred. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and behavior checked live; unpriced provider dollar cost stays UNKNOWN. The placement exception applies only to validated native Cursor API execution. Other local work remains denied under managed-only policy. Missing/stale Stop or cleanup proof stays pending; damaged archives remain failed. Explicit Legacy keeps its existing policy owners. ## Model Used OpenAI Codex/GPT-6 family: tools, execution, browser checks, delegated review. API ID/context window unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; retained failures are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
4d31588dea |
test(server): load model route setup before test deadlines (#18828)
## Thinking Path > - Paperclip uses native agents and local provider adapters to do work. > - The server model route must keep provider catalogs separate. > - Its tests use the real HTTP route with controlled service and adapter fixtures. > - Each test reset the module graph and imported the route again inside short test deadlines. > - Slow imports timed out. Unfinished work could then change mock call counts in a later case. > - This pull request loads the stable route graph during collection and keeps each case's fixture resets. ## Linked Issues or Issue Description Refs #18816 (affected required CI run). **What happened?** The pending Kimi model-route test exceeded its 15-second deadline in required Linux CI. The unchanged test file also failed locally: the first setup hook exceeded 30 seconds, later tests exceeded 15 seconds, and the last OpenCode case saw two calls instead of one after unfinished work reached the shared mock. **Expected behavior** Route import and transformation must finish before the case deadline starts. Each case must still test the real HTTP route with its own mock state, environment and profile values, and adapter registry cleanup. **Steps to reproduce** Run `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/adapter-model-refresh-routes.test.ts` with a cold module graph. On a loaded worker, repeated module resets and imports can exceed the hook and test deadlines. ## What Changed - Register stable service mocks once and load the real route and middleware during test collection. - Remove per-case module invalidation. Keep all five original HTTP assertions, fixture resets, profile and environment restoration, and adapter unregister operations. - Keep the existing hook and test deadlines. Product source code does not change. No user-facing documentation changes are needed. ## Verification - The unchanged file reproduced the hook timeout, test timeouts, and leaked call-count failure locally. - All five original cases pass with the fix. The cold graph takes about 47 seconds during collection; the cases complete within the unchanged deadlines. - All five cases also pass at the same head in a shuffled order with seed `20261103`. - Module-boundary and diff checks pass. - Full repository typecheck, tests, and build run in required CI. No hosted product behavior is claimed from this test-only change. - Current-head CI attempt 2 failed in the typecheck harness because the trusted master workflow called `scripts/typecheck-test-projects.mjs`, which the older source head did not contain. The log is retained. A normal merge of master supplies the helper from #18738. All five original HTTP cases pass again on that merged baseline. Fresh CI and exact-head reviews passed at `86f0bf27267fe3f44ab813fb0a57ebbb94bae7fe`. - The refreshed run passed all server checks but one browser shard failed: `task-reload.spec.ts:63` waited five seconds for a task heading after the intentional bundle failure. The trace showed the saved task breadcrumb and properties, with the conversation still loading. The exact failure log and trace were retained. One targeted retry of that shard and its aggregate passed without source changes. This establishes recovery of the check; the original browser loading delay remains unexplained. - Final exact-head checks: 52 success, two skipped, and two neutral. Both Greptile and Paperclip reviews are 5/5. No review threads remain. ## Risks - Stable imports share module state across cases. The test retains explicit mock resets, adapter unregister, and profile/environment restoration. - The affected model route reads profile qualification and environment fixtures when it handles the request. These cases do not require a new module instance for each environment change. - No runtime or schema changes. No test deadline increase. ## Model Used OpenAI `gpt-6.1-sol`, Extra High reasoning. The model used repository tools, code execution, and test output. The host does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f836fec59e |
perf(budgets): avoid company lock for routine eligibility checks (#18830)
## Thinking Path > - Paperclip helps people manage AI agents at work. > - The budget service checks whether an agent can start work. > - Preliminary checks acquired the company accounting lock even with no budget policy. > - Independent checks then waited behind accounting writes for the same company. > - This pull request reads one consistent snapshot before it decides whether reconciliation is needed. > - Healthy checks avoid the writer lock. State changes and final paid admission retain their existing authority. > - The benefit is less company-wide contention during routine eligibility checks. ## Linked Issues or Issue Description Refs #17697 (the preliminary-check portion). Refs #17817 (related serialization in agent configuration). Related approach: #17966. That PR adds a precheck for policies that cannot block. This PR also avoids the lock for healthy active hard-stop policies and manual pause reads. It uses a repeatable-read snapshot and leaves public transactional admission locked. It does not add the inactive-policy spend filtering proposed in #17966. **Current behavior** Every public `getInvocationBlock` call takes the company writer lock. Healthy checks can wait behind an unrelated receipt or settlement operation. **Proposed behavior** Read current scope state, policies, exposure and relevant incidents in a read-only snapshot. Return directly when no transition is needed. Otherwise discard the snapshot and reconcile current state under the company lock. **Reason and benefit** Independent preliminary checks can finish while another transaction owns the company row. Final reservations and paid-work admission still serialize against current accounting state. ## What Changed - Separate preliminary eligibility inspection from mutating reconciliation in the budget service. - Share scope loading, pause decisions and soft-incident resolution conditions between the two paths. - Keep no-policy checks free of exposure scans and incident reads. - Add inspection and reconciliation spans to the existing operator-gated Observability path. - Add PostgreSQL contention tests and regressions for pauses, holds, incidents, recovering native accounting and company boundaries. - Use the shared fixture startup timeout. Per-test deadlines and the 500 ms lock timeout stay intact. - Update the implementation contract and tracing documentation. Add dated investigation and future-design notes. The future accounting architecture remains a proposal. ## Verification - 73 targeted tests passed across eight files on implementation commit `244db565`. No tests were skipped. The files cover budget service, enforcement scoping and delivery, live receipt attempts, budget history, run settlement, heartbeat accounting and budget tracing. - Real PostgreSQL tests hold the company accounting lock while routine checks finish. The same reader's authoritative admission instead fails with PostgreSQL lock timeout `55P03`. - Full workspace typecheck passed before the rebase: `node scripts/typecheck-low-memory.mjs`. - Full workspace build passed before the rebase: `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm -r --workspace-concurrency=1 build`. The Runner release binary was rebuilt from source. - `node scripts/check-module-boundaries.mjs` and `git diff --check` passed. - Initial local setup failed before assertions: cold fixture setup exceeded 20 seconds, then a missing SDK build was exposed. The fixture now uses the shared 90-second startup allowance. The SDK was built before the successful test run. An incomplete local Copilot artifact also blocked the first build; the pinned installer repaired it before the successful canonical retry. - The branch rebased cleanly onto master `83b5f7d486`. All 73 targeted tests then passed again on clean head `244db565ae206502b100c3cdf5cd86a0ae91b372` (eight files, no skips). The full local Vitest suite was not run; remote CI must verify the wider suite. - Initial remote CI failed three monitor-scheduler cases at the suite's three-second heartbeat-idle wait. The unchanged suite then passed all 21 tests locally (`pnpm exec vitest run src/__tests__/issue-monitor-scheduler.test.ts`, 70.13 seconds). The one failed-job rerun was cancelled after the review follow-up was pushed; it produced no pass/fail result. No scheduler assertions or timeouts were changed. On updated head `ba2ab2a0`, the unchanged monitor suite passed all 21 tests in 38.381 seconds in [server shard 10/12](https://github.com/paperclipai/paperclip/actions/runs/41702687492/job/126151059714). That shard passed 115 files and 2,677 tests, with no failures (one file and 20 tests skipped by existing suite configuration). - The independent review identified missing recovering-run coverage. The added contention case requires the locked fallback while another accounting writer holds the company row, then verifies the recovery block without pausing or cancelling recovery. The regression is in follow-up commit `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`. All 19 budget-service tests passed locally (no skips, 48.89 seconds), and `pnpm --filter @paperclipai/server typecheck` passed after the follow-up. - The updated CI run then hit a separate plugin-refresh fixture timeout at its final re-enable settle. The unchanged `external-object-refresh-work.test.ts` suite passed all 24 tests locally (52.02 seconds). Inspection identified a possible fake-clock race: its quiet helper can observe no scheduled deadline while reconstruction is still active, then encounter the existing 30-second reconstruction cooldown. The one same-head failed-job retry [passed](https://github.com/paperclipai/paperclip/actions/runs/41702687492/job/126158227876): refresh suite 24/24, shard 115 files and 4,125 tests passed (one file and four tests skipped by existing configuration). No assertions, timeouts, or product code were changed. - [Full CI](https://github.com/paperclipai/paperclip/actions/runs/41702687492) passed on exact head `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`: build, typecheck, Runner checks, all server/workspace suites and E2E. Both required aggregates are successful. All 59 checks are terminal: 54 successful, two advisory neutral and three expected skips. The PR is mergeable with a clean merge state. - Deployment latency and CPU effects have not been measured. - Greptile reviewed updated head `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`: 5/5, no actionable findings or unresolved Greptile threads. - Independent Paperclip review on the updated head: advisory 4/5, no blocker, recovery gap confirmed resolved. One canonical maintainability advisory remains visible and is documented below. ## Risks - Preliminary inspection can precede a concurrent policy or pause change. It grants no paid dispatch authority. The final locked admission and reservation checks remain required. - Healthy configured policies add an incident lookup. An unheld blocking scope can compute exposure during both inspection and locked revalidation. Already-held scopes skip the first exposure computation. - Holds and incidents must retain their existing transition effects. The fallback reads them again under the company lock, rather than mutating from the earlier snapshot. - Eligibility inspection and reconciliation still maintain separate transition lists. Future reconciliation changes must update the inspection predicate and its contention coverage together; consolidating the effects model remains a nonblocking review follow-up. - No schema or API response change is required. This change does not relax the current budget enforcement contract. ## Model Used OpenAI Codex with model `gpt-6.1-sol`. Used reasoning, repository inspection, code execution and delegated review. The session did not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e491b23eb5 |
fix(runner): retire sandbox suspension receipts after admission (#18829)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Native Runner sessions preserve a conversation across task turns. > - Remote suspension records a local receipt for the stopped Runner identity. > - The built-in sandbox path now writes the same receipt as plugin tasks. > - Admission cleared that receipt only for plugin tasks. > - A later warm turn could reject a clean journal because the old receipt named an earlier run. > - This pull request clears the receipt after successful admission for both sandbox paths. ## Linked Issues or Issue Description Refs #18343 and #18554 for the shared suspension receipt producer. Refs #18718 for the related controller projection refresh. **What happened?** After a suspended built-in sandbox session resumed, its next warm turn could fail with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. Its journal named the completed resumed run. Its local suspension receipt still named the older run. **Expected behavior** Successful admission must retire the prior suspension receipt. The next compatible warm turn must reuse the admitted session and provider history once. A failed admission must retain recovery evidence. **Steps to reproduce** 1. Complete and suspend remote turn A. 2. Resume turn B through the built-in sandbox transport. 3. Keep the controller warm, with full Runner state owned by the sandbox. 4. Admit turn C. The old local receipt conflicts with B's journal identity. **Paperclip version or commit** Reproduced in the executor regression on base `367f29478d`. **Deployment mode** Native Runner with the built-in remote sandbox transport and a warm lifecycle. ## What Changed - Clear the local suspension receipt after a remote sandbox session passes provider recovery and durable admission. Apply this to built-in runners and plugin tasks. - Extend the existing three-turn warm admission regression to cover a suspended receipt and remote-only Runner state. - Keep an active-owner control that rejects admission and retains its receipt. Keep the existing foreign-owner, retirement, configuration, plugin, and unchanged-session controls. - Add a real Linux Runner continuity case that suspends A, resumes B, and keeps B warm for C. Register the test fixture's owned local API listener. - Document receipt ownership and the new continuity case. ## Verification - The new behavioral regression failed before the production change. Turn C raised the exact identity mismatch above. The active-owner control passed and retained its receipt. - All 1,087 native executor tests pass after the fix. - Both opt-in Linux process cases pass at the final commit: ordinary warm reuse and suspension followed by warm reuse. They use PostgreSQL, TLS, current-source Rust binaries, and a deterministic Codex fixture. Each preserves one conversation, starts exactly three turns, and reuses the warm Runner and provider on turn C. - Full repository typecheck passes with `node scripts/typecheck-low-memory.mjs`. Full production build passes with `pnpm build`. - All current-head CI checks pass at `a95a6fdfe8`, including the full test shards, browser shards, Rust checks, build, and typecheck. The policy job needed one retry after repository checkout timed out before checks ran. - Greptile and Ponytail each give 5/5 at the final head. There are no open review threads or findings. - The local full `pnpm test:run` attempt passed 351 cases in its first chat shard. The session interruption stopped it during the second shard, so it has no complete local verdict. The complete CI test run passes. - Module boundary checks and `git diff --check` pass. ## Risks - A receipt removed before admission could discard recovery evidence. Removal stays in the successful `onSession` callback. The negative admission control checks retention. - A stale owner must not gain authority. Journal, run, lease, and scope checks remain in place before the callback. - Historical contradictory or quarantined state still requires the existing verified recovery path. This change does not delete those records. - No migration or provider configuration change is required. ## Model Used - OpenAI Codex, based on GPT-6. The runtime does not expose a more specific model ID or context limit. Used reasoning, repository tools, code execution, and tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d8a9b3eb0c |
fix(ui): show task identifiers in activity toasts (#18802)
Project the company-scoped identifier for authorized task events and use it in toast titles, fallback bodies, and links. Retain authorized delivery when the optional identifier lookup fails. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
fb6554ac62 |
fix(native): settle source wake with terminal projection (#18804)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Each agent wake request can bind to one heartbeat run. > - Native recovery can finish and project a run after its live executor has stopped. > - The legacy completion callback settles the wake, but native terminal projection did not. > - A succeeded run could therefore leave its exact wake claimed with no finish time. > - This pull request settles that claim in each native terminal projection transaction and reuses durable projection recovery for old committed rows. > - The benefit is consistent run and wake receipts without a provider replay or a new sweeper. ## Linked Issues or Issue Description Refs #13856 and #10109. These open PRs describe orphan claimed wakes and cleanup after terminal persistence. This change fixes the native projection owner and uses its existing durable worker. It does not modify those contributor branches. **What happened?** A native run can be accepted and committed, then reach a terminal state through recovery. Its source wake stays claimed because the live completion callback does not return to settle it. **Expected behavior** Terminal projection must settle only the same company's exact source wake bound to that agent and run. Completed, failed and cancelled outcomes must retain their meaning. Other live, queued, cancelled, unbound or newly rebound wake records must remain unchanged. **Steps to reproduce** Commit an accepted native result in the real database fixture. Leave its source wake claimed. Call native finalization with run-status projection enabled, without the live completion callback. Before this change, the run becomes succeeded and finished while the exact wake remains claimed. ## What Changed - Settle the exact source claim inside native finalization, committed recovery, and exhausted-finalization projection transactions. Fence it by wake ID, company, agent, run and current claimed state. - Lock the matching claim and register settlement notifications before the write. A failed transaction rolls back both run projection and wake settlement. - Let the existing durable projection worker discover a terminal run with an unfinished exact claim. A repaired committed row leaves the pending set. No new timer, sweep or table is added. - Add real database regressions for native recovery, all three terminal outcomes, old committed receipts, idempotence, scoped denial controls, notification ordering and rollback. Cover provider/workspace finalization exhaustion and denial when the caller does not own terminal projection. - Preserve the existing provider, accounting, workspace, permissions and result contracts. No API, schema, user command, access-map entry or documentation contract changes are needed. ## Verification - The new native coordinator/finalizer regression failed before the production change: the run was succeeded and committed, but the bound wake was still claimed. - The initial focused database run did not reach the assertion because the SDK was unbuilt. A second attempt hit sandbox loopback restrictions. A third exposed skipped package library hydration. These setup failures were corrected before the meaningful red was captured. - Real PostgreSQL projection and response suites passed 89 tests before the independent review correction. The reviewer requested notification intent registration before mutation. The correction adds a notification-order and rollback case. All 90 tests now pass across both suites. - Server typecheck passed after building the isolated checkout's Runner and SDK prerequisites. Fresh corrected-source server typecheck and production server build also pass. Full repository checks run in current-head CI. The earlier serialized-9 CI failure was the independently repaired inbox snapshot fixture in #18798 (expected nine reader calls, observed eight). A normal merge of the containing master supplies that fix; the native suites plus the repaired inbox suite now pass 96 tests together at merge head `456c92ef8d60c3fcbeaf66ff8010a7ebd3d1109d`. Server typecheck passes. That head was superseded by the reviewed retry-exhaustion correction below. The subsequent review found a third terminal write in retry exhaustion. Real database provider and workspace exhaustion regressions failed with claimed/null-finished wakes before the fix. The same fenced helper now runs inside that transaction. At final source head `9718d4c2ec6066accf549dc6fea795891f6b90ec`, all 100 focused tests, server typecheck and production server build pass, including projection-disabled denial and rollback of run/coordinator/wake on notification failure. - Final current-head CI passed: 54 success, 2 skipped, and the neutral Paperclip Review at 5/5. Greptile also reports 5/5. Required ci / verify and ci / e2e passed. All review threads are resolved. - No live task write, provider work, accounting repair, deployed replay or manual rollout was performed. ## Risks - A stale wake binding must never be settled. The update requires the exact wake ID, company, agent, run and claimed state. Tests cover unbound and newer-run bindings, foreign company and agent, queued and cancelled records, and retained live ownership. - Old claimed debt now participates in the existing projection predicate. It uses the source wake primary key and stops matching after settlement. Existing quiet-worker tests remain in the focused suite. - Wake settlement must share the projection transaction. Notification failure and outer rollback tests preserve both original rows. - Interruption, timeout, retained historical terminal-failure coordinators, and delayed response presentation keep their current owners. This change does not infer or replay provider work. ## Model Used OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source inspection, code execution and regression testing. Context-window capacity is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked existing public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation where needed; existing contracts are restored - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e0c6992426 |
fix(heartbeat): prevent saved input starvation during queue reloads (#18819)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Accepted messages and task Retry requests use durable wake receipts. > - The recovery worker loads receipts in pages of 100 and checks batches of 25. > - Company activity can reload those pages before the pending batch drains. > - Reloads reset ready receipts' dispatch times and can repeatedly favor receipts already checked. > - This pull request preserves the priority of unchecked ready receipts so saved input can reach normal admission. ## Linked Issues or Issue Description **What happened?** Saved input can remain deferred after the preceding run finishes. Repeated company eligibility changes reset the dispatch times of pending receipts during paginated reconstruction. A PostgreSQL regression checks only 75 of 178 receipts on the original code despite twelve eligibility changes. **Expected behavior** Each ready receipt must reach its admission check. Existing wait deadlines, retry backoffs, authorization, and execution gates must still apply. **Steps to reproduce** 1. Seed more than one page of deferred receipts. 2. Keep those receipts pending after their checks. 3. Send company eligibility notifications between scheduler passes. 4. Check that every receipt is reconsidered, including receipts later in the backlog. **Paperclip version or commit** Reproduced on `367f29478d`. **Deployment mode** Built from source with PostgreSQL. The defect occurs before provider execution. Related: #10195 reports a similar starvation symptom in a different promotion loop. #18621 added saved-input Retry after a successful run. This change fixes fairness in the deferred-receipt worker. ## What Changed - Add an optional entry merge callback to scoped deadline reconstruction. - Use it in the deferred-receipt worker to retain unread ready receipts' priority while refreshing their state. - Place receipts whose wait has expired into the ready queue during reconstruction. - Keep real wait, claim, and dispatch deadlines authoritative. - Add a paginated starvation regression, real process dispatch coverage, and GitHub lock-collision recovery coverage. - Document the recovery fairness rule. ## Verification - The PostgreSQL starvation regression checks only 75 of 178 receipts before the fix. It checks all 178 afterward, both with ordinary receipts and with expired execution waits. - Seven focused scheduler cases pass, including moved interrupt grace, moved native error deadline, fresh claim/wait deadlines, and unchanged-gate grace. - GitHub endpoint lock-collision recovery, retained board authority, HTTP Retry, and five current admission-gate cases pass. - Real process acceptance passes on the final code: task Retry crosses the repeatedly invalidated backlog, reconstructs its persisted queued run after a worker restart, saves `receipt-proof` in run output, and has one successor for the Retry receipt. The test holds executor capacity while advancing queue time, then starts fresh workers at PostgreSQL time before releasing a real process lease. The defect is in admission scheduling before provider execution; this acceptance uses PostgreSQL and a local process adapter. - The full workspace build and typecheck (`node scripts/typecheck-low-memory.mjs`) pass locally. The affected worker suites pass 52 of 53 cases together; the existing native multi-company batch case times out in that combined run and passes in isolation without code changes. All new regression and acceptance cases pass. - Local `pnpm test:run` stops in its first chat shard on three 15-second timeouts (Teams long native HTTP picture delivery, direct final-response materialization, and contended inbound notices). All three pass in isolation without code changes. Later full-suite groups did not run locally after that shard failure. - [The full GitHub Actions matrix passes](https://github.com/paperclipai/paperclip/actions/runs/41697489374) on `ac91df2cc6db6c77529d923177b375b8c92efd45`, including all general-test, serialized-server, browser, build, and typecheck jobs. Server shard 8 initially failed an upstream process-adoption test and passed its one rerun without code changes. - [Greptile is 5/5](https://github.com/paperclipai/paperclip/pull/18819#issuecomment-6612091535) on the same current head, with no actionable findings or unresolved review threads. The separate Paperclip Review check is neutral; its previous request was bound to an older commit. Focused command: ```sh pnpm exec vitest run server/src/__tests__/deferred-execution-receipt-work.test.ts server/src/__tests__/scoped-deadline-work.test.ts ``` ## Risks - Ready receipt order changes during repeated reconstruction. Admission still reads current durable state. - A ready receipt must not inherit an obsolete priority after it acquires a future deadline. The merge applies only when both pending and loaded entries are immediate. Deadline safety tests cover that boundary. - The merge callback is optional. Other scoped workers retain their current reconstruction behavior. ## Model Used OpenAI `gpt-6.1-sol` in Codex, max reasoning effort, with tool use and local code execution. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused regressions and acceptance; full-run timing limitations documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
8ab89f41e4 |
fix(build): typecheck test projects in 4 GiB (#18738)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors and agents typecheck this repository, often on machines with limited memory > - paperclipai/paperclip#18737 made `pnpm -r typecheck` fit in 4 GiB because the server now checks in layers > - The test projects outside `pnpm -r` still import server source through relative paths, so each one checks most of the server as one program > - These projects peaked between 6.9 GiB and 14.9 GiB, so they fail on a 4 GiB machine > - This pull request makes these projects read the server layers as declarations, and adds a CI step that keeps them that way > - The benefit is that every TypeScript project in the repository can now be checked in 4 GiB, and a later change cannot silently undo this ## Linked Issues or Issue Description Refs paperclipai/paperclip#18737, which is merged. This pull request builds on the server layers that it added. No issue exists for this gap. **What happened?** `tsc -p tests/runner-e2e/tsconfig.json` peaked at 14.9 GiB at default settings. `tests/speko-native` peaked at 12.2 GiB, `server/src/__tests__/helpers/maintenance` at 9.6 GiB, and `tests/lifecycle-baseline` at 6.9 GiB. On one thread, `tests/runner-e2e` still peaked at 7.7 GiB. **Expected behavior** Each project passes with 4 GiB of memory. **Steps to reproduce** Run `tsc -p tests/runner-e2e/tsconfig.json` in a Linux container with `--memory=4g --memory-swap=4g`. The kernel stops the process. ## What Changed - Added `tsconfig.layers.json` to the four projects. Each file extends the project's `tsconfig.json` and references the generated server declarations. - Added `scripts/typecheck-test-projects.mjs`. It builds the server declarations and then checks the named projects, or all four. It takes `--low-memory`. - Changed `test:e2e:runner:typecheck` and `test:lifecycle-baseline:typecheck` to call that script. `docker-runner-check.yml` calls the first script by name, so that workflow needs no edit. - Added `--declarations-only` to `server/scripts/typecheck-layers.mjs`. The script now also generates `tsconfig.declarations.json`, so hand-written configs do not name individual layers. - Exported `runPnpm` from `scripts/typecheck-low-memory.mjs`, so the new script can use it. - Added a step to the `Typecheck + Release Registry` job. It runs straight after the server typecheck and reuses its declarations through `--reuse-declarations`. - The script now fails if a project loads server source, even when the types pass. Test support files, declaration files and JSON files do not count. This check does not enforce a memory limit. - Added a unit test that fails if a new project under `tests/` is neither in `PROJECTS` nor named as a single program. `tests/runner-acceptance` is the one named project. - Added `server/src/types/express.d.ts` to `tests/lifecycle-baseline/tsconfig.json`. That plain config failed alone with 11 TS2339 errors, and it now passes. - Exported `isTestFile` from `server/scripts/typecheck-layers.mjs`, so the source check uses the same rule as the layers. - Turned off `declaration` in the layered configs. These projects emit nothing, and declaration checks gave a false TS2883 error when `tests/runner-e2e` read the server as declarations. - Added tests, one of which runs the real compiler. Updated `doc/DEVELOPING.md` and `tests/lifecycle-baseline/README.md`. The references are not in each project's `tsconfig.json` because Vitest reads that file. A test showed that Vitest fails when a referenced config does not exist yet. The `tsconfig.json` files do not change, so editors and Vitest see the same configs as before. No new `package.json` scripts were added, as `.greptile/rules.md` requires. ## Verification I tested commit `9763880fab`, which is based on `1c9ca5829f`. I copied the commit into a named Docker volume and ran a fresh install. Each row ran in its own Linux arm64 container with `--memory=4g --memory-swap=4g` and 8 CPUs. The first row started with no build output. All rows used `--low-memory`. CI does not set a memory limit. I used the limit only for this measurement. | Project | Errors | Peak (cgroup) | Max process RSS | OOM kills | Time | | --- | --- | --- | --- | --- | --- | | `tests/runner-e2e` | 0 | 3.34 GiB | 3.06 GiB | 0 | 57 s | | `tests/speko-native` | 0 | 3.60 GiB | 3.27 GiB | 0 | 51 s | | `tests/lifecycle-baseline` | 0 | 3.43 GiB | 3.15 GiB | 0 | 50 s | | `server/src/__tests__/helpers/maintenance` | 0 | 3.39 GiB | 3.13 GiB | 0 | 52 s | `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm test:e2e:runner:typecheck` and `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm test:lifecycle-baseline:typecheck` also passed in the same limit, with peaks of 3.55 GiB and 3.52 GiB. I also ran the CI sequence in one container with the same limit: `pnpm --filter @paperclipai/server typecheck`, then `node scripts/typecheck-test-projects.mjs --low-memory --reuse-declarations`. Both passed, with a peak of 3.46 GiB. The new step took 16 s. The server layer build sets the peak in each row. Against the declarations, the projects peaked between 0.9 GiB and 2.2 GiB at default settings on macOS. Server source files that each project loads, without test support, declaration or JSON files: | Project | Plain `tsconfig.json` | `tsconfig.layers.json` | | --- | --- | --- | | `tests/runner-e2e` | 1,345 | 0 | | `tests/speko-native` | 1,647 | 0 | | `tests/lifecycle-baseline` | 1,343 | 0 | | `server/src/__tests__/helpers/maintenance` | 1,716 | 0 | I counted these on `3f0f31b2f9`. The source check in the script confirmed 0 for all four projects on this commit. **Limits of this proof:** - Without `--low-memory`, the 4 GiB run does not pass. The kernel stopped the layer build for `tests/runner-e2e` on `3f0f31b2f9`, and the script exited with status 1. I did not repeat this run on the final commit. The script does not turn on low-memory mode by itself. - The Linux runs used arm64, not x64. - The new CI step has not run in CI. `pr-trusted.yml` runs from `@master`, so the step first runs after this pull request merges. Check its first run. **Other checks on this commit:** - `server/scripts/typecheck-layers.test.mjs`: 38 tests pass. One test uses the real compiler to show that the source check tells a layered config from a config with no references. - With the references removed from `tests/speko-native/tsconfig.layers.json`, the script exits with status 1 and reports 1,644 server source files. - With the declarations removed, `--reuse-declarations` exits with status 1 and reports TS6053. - A type error in a test project makes the script exit with status 1. - `tests/lifecycle-baseline/tsconfig.json` passes with 0 errors. - `tests/lifecycle-baseline` Vitest suite: 533 tests pass. - Workflow tests that read `pr-trusted.yml` and CI routing: 114 tests pass. **Earlier checks:** - `server/src/__tests__/server-package-build-script.test.ts` passed 10 tests on `3f0f31b2f9`. This test failed in `ci / General tests (server (9/12))` on an older head. It tested a build script split that the final #18737 removed. - A changed signature in `server/src/services/run-liveness.ts` failed `tests/lifecycle-baseline/authority.test.ts` through the rebuilt declarations, before the rebases. When I restored the signature, the check passed. **Not run locally:** - `pnpm test:run` and `pnpm build`. CI covers these. **Failures that this pull request does not cause:** `pnpm test:e2e:runner:unit` has 13 failures on this commit, in `cursor-cloud-replay-fault`, `first-task`, `operating-receipts`, `pi-controls-evidence` and `workflow-security`. The same 13 tests fail on `master` at `6ef37398c8`. A comparison of the test names shows no differences. PR CI does not run this suite. ## Risks - Low risk to runtime behavior. No shipped code changes. - **Slower check:** `pnpm test:e2e:runner:typecheck` now builds the server layers on each run. Before the rebase, it took 30 s on macOS at default settings, about the same as the old single program for that project. On this commit it took 57 s in low-memory mode in the 4 GiB container. The new CI step reuses the declarations, so it took 16 s. - **Narrower check:** Server source is now checked only by the server's own layers, not again under each project's compiler options. The projects also skip declaration-portability checks. These projects emit nothing, so those checks had no output to protect. - **Unchanged path:** `tests/runner-e2e/native-completion-checks.mjs` still runs `tsc -p tests/runner-e2e/tsconfig.json` as one program. It needs about 15 GiB at default settings. I did not change it because that file and its inputs are part of a fingerprinted source contract, and no workflow calls it. - **Layer growth:** The new checks do not measure memory. If the layers or the import cycle grow, the checks still pass and the peak rises. This is intentional. Crossing 4 GiB does not fail a build. - **Project list:** The CI step checks only the projects in `PROJECTS`. The unit test covers new projects under `tests/`, but not a new project in another directory. - **Reused declarations:** `--reuse-declarations` trusts the declarations on disk. CI uses it straight after the server typecheck on the same commit. If that typecheck fails, the job stops first. Stale declarations can pass locally, so `doc/DEVELOPING.md` warns against this. - **Job dependency:** A type error in a test project now fails `Typecheck + Release Registry`, and so `ci / verify`. - **Not covered:** The new step runs no tests. `node scripts/typecheck-low-memory.mjs` and the Docker image build also have no new guard. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
4a38ff9c92 |
fix(ci): make Cloud readiness reruns and fixtures reliable (#18823)
## Thinking Path > - Paperclip manages agent work and preserves task state. > - Cloud readiness verifies an exact source revision before release. > - Its workspace tests must wait for execution and database cleanup to finish. > - Two attempts hit the same ten-second polling limit and read a running row. > - The second attempt also lost AWS runners and exceeded a fixture cleanup limit. > - This pull request waits for owned execution, bounds client shutdown, and gives verification reruns an independent runner pool. > - Assertions, source identity and required verification gates stay in place. ## Linked Issues or Issue Description **What happened?** [Cloud readiness run 41684102136](https://github.com/paperclipai/paperclip/actions/runs/41684102136) failed twice on master `367f29478d14003cc5c0f161b7ce8f3164993634`. Attempt 1 failed `heartbeat-workspace-busy.test.ts:473`: expected `succeeded`, received `running`. Attempt 2 failed the same helper in the inherited-workspace resume case at line 744. The helper polls for ten seconds, then returns the active row as if the wait finished. Attempt 2 also exceeded the 30-second `workspace-preparation.test.ts` teardown limit. Its client close had no deadline, and database plus directory cleanup shared the default hook budget despite the fixture's existing 90-second startup budget. Other attempt-2 failures show runner shutdown or lost communication. Several shutdown signals arrived at 18:05:17 UTC. Read-only AWS inspection confirms that all eight infrastructure-failed instances carry `runs-on-workflow-job-interrupted=true`. Five have explicit `Server.SpotInstanceTermination` reasons, including the server-1 runner that lost communication. Three were terminated through the fleet's shutdown path. CloudTrail records the Rust runner's interruption tag at 18:03:37 UTC and its termination by the RunsOn fleet role at 18:05:31 UTC, despite EC2 showing a user-initiated termination. Repeating the entire workflow reused the same Spot fleet. **Expected behavior** Wait for owned heartbeat execution before reading final state or scheduled retries. Bound database-client shutdown and allow the fixture's cleanup cost class to finish. Rerun verification on GitHub-hosted runners after an interrupted fleet attempt. Preserve all checks and report prior failures. **Steps to reproduce** 1. Delay the test adapter beyond the helper's ten-second polling window. 2. Invoke the ordinary local shared-workspace run or resume the deferred sandbox run. 3. Observe the old helper return `running` before the execution completes. 4. Evaluate the old runner expressions with `github.run_attempt = 2`; they still choose the interrupted AWS pool. **Paperclip version or commit** Base: `367f29478d14003cc5c0f161b7ce8f3164993634`. **Deployment mode** GitHub Actions Cloud readiness, with disposable PostgreSQL fixtures. Related: #11293 fixed the earlier gap between cancellation and retry-row insertion. This change waits for the complete execution instead. Duplicate searches covered the workspace test, retention fixture and runner shutdown reports. ## What Changed - Replace both workspace-test polling helpers with the existing heartbeat completion barrier. Keep final status, retry and coordination assertions. - Bound workspace-preparation client shutdown to one second. Apply the existing embedded-PostgreSQL fixture budget to its teardown. - Use AWS only for the first exact-source attempt. Route later verification, chaos and cloud-migrator planning/build attempts to GitHub-hosted runners. The publisher keeps its existing routing. The final branch also incorporates upstream #18821's CODEOWNER routing fix. - Add runner-expression coverage for numeric, string, missing and later attempt values. Document retry routing. ## Verification - The original workflow's failed assertions and interrupted jobs remain linked above. A local two-suite run also passed all 157 assertions and failed only the original 30-second teardown hook. - All 341 workflow contracts pass. After the master synchronization, all 347 routing and CI-planner controls pass. Actionlint passes for the three changed workflows. - A matched delayed-adapter comparison uses disposable PostgreSQL 17 databases, the same 11-second adapter delay, and the same 90-second experimental outer budget. The old helper fails with `running`; the completion barrier passes. Production test deadlines stay unchanged. - Earlier experiments hit a 30-second experimental outer limit and the macOS host's shared-memory limit during native database startup. The completed comparison uses task-owned databases on an existing test server. Those earlier attempts are not counted as passes. - All 31 workspace-busy cases pass with normal test deadlines. The 126-case workspace fixture also passes with its 90-second teardown budget. - Final-head cleanup measurements: main client close 7 ms, owned database cleanup 20,938 ms, and workspace-home removal 826 ms. A measured fixture held 124,494 PostgreSQL files (208 MiB) after repeated table resets. The measured database cleanup alone consumes most of the old 30-second hook budget. The separate baseline reproduced a timeout at that hook; the fixture now uses its established 90-second cost class. - Repository typecheck and build pass on synchronized head `2aceb3abd144a569efddabed3b7e0550ea23ceb2`. - All 157 focused cases pass on the synchronized branch, including complete PostgreSQL 18 fixture shutdown. [Full hosted CI](https://github.com/paperclipai/paperclip/actions/runs/41697804471) passes on the final head, including the required verification and E2E aggregates, all server/chat/workspace groups, Runner lanes and all eight browser shards. The separate GitHub quality-review workflow also passes. All 60 reported checks are terminal: 54 pass, three are informational neutral results, and three are expected skips. There are no failures or pending checks. The complete local `pnpm test:run` command was not run; targeted fixtures already encountered this host's shared-memory limit. Hosted CI supplies the full Vitest and browser matrix. - Greptile and Paperclip Review both give synchronized head `2aceb3abd144a569efddabed3b7e0550ea23ceb2` 5/5. There are no open review threads. - Module-boundary and whitespace checks pass. ## Risks - Hosted reruns can have different cache hit rates and queue times. They execute the same source and verification commands. - The first attempt retains the existing AWS Spot fleet policy. AWS evidence establishes several interruption-triggered failures; later verification attempts use an independent pool. This does not change the fleet's external capacity policy. - The completion barrier waits for owned background execution. A hung execution still fails the test's outer deadline. - The larger teardown budget remains finite. PostgreSQL's bounded stop protocol and cleanup errors still fail the hook. - No production execution, database schema or provider contract changes. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository inspection, code edits and test execution. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ccc63ea458 |
fix(server): prove completed history before idle sleep (#18646)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
1c9ca5829f |
perf(ci): overlap Runner workloads and cache debug workspace builds (#18794)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Required CI checks protect its server and Runner contracts. > - Recent full runs waited on general server files and the final Runner lane. > - That Runner lane ran package tests before native server tests. > - PR jobs also rebuilt unchanged debug workspace binaries. > - This pull request overlaps those workloads and reuses exact trusted build inputs. > - The benefit is a shorter critical path with the same test coverage and AWS authorization. ## Linked Issues or Issue Description Refs #18544, #18773, #18774, #18798, #18799. **What existing behavior does this improve?** The wall time of required PR tests and the precision of affected-test selection. **Current behavior** General server and native files use two workers. The second Runner Vitest lane runs its package shard and then the native server group. Each Runner job compiles debug workspace units again. A Vite raw import also retains its consumer for unrelated source edits. **Proposed behavior** Use three isolated workers for general server files and four for native files. Reuse the two Runner slots for a complete package suite and a complete native server suite. Restore all debug binaries only on an exact input key. Record raw imports as explicit file inputs. ## What Changed - Separate the Runner package and native server workloads without adding CI jobs. Preserve the old command arguments for master-pinned workflows, local commands, and other shard counts. - Increase file workers while retaining fork isolation, serial cases, and the shared-database guard. - Add an exact debug workspace cache. Derive binary targets from Cargo. Include source, fixtures, generated inputs, manifests, compiler identity, flags, Cargo configuration, and checkout path in its key. Discard incomplete restores. Only verified master pushes can save it. - Keep older PR checkouts working when they lack the new cache helper. - Track Vite `?raw` imports as repository file inputs. Keep unknown loaders, untracked files, and opaque effects conservative. - Add cache and coverage guards and document the behavior. AWS runner authorization is unchanged. - Keep paired test benchmarking available when install hooks correctly disable materialized dependency reuse. The old benchmark asserted a cache hit in that safe fallback. - Wait for active dispatch work to finish before asserting idle reads. Analyze the two 16,400-row database fixtures before their queries so they have current planner statistics. Suppress only PostgreSQL notices inside the cleanup transaction to avoid logging every cascaded table; warnings/errors remain visible. Row counts, real queries, assertions, database reset, and timeouts remain unchanged. ## Verification - 196 affected CI tests passed, including routing, coverage, selection failure fallbacks, shared-database serialization, and existing/new cache guards. Final worker/cache adjustments also passed their focused checks. - `pnpm build` and `pnpm -r typecheck` passed locally. Full test coverage is being verified by the Linux CI profile; `pnpm test:run` was not repeated as a full serial local run. The paired CI benchmark passed. Both paired benchmarks passed, and full CI passed again after integrating newer master commits. The external Paperclip Review app is the only pending status. - Actual debug artifacts remained fresh after simulating checkout timestamps: Cargo completed in 0.82 seconds; the content key was unchanged. This is local reuse evidence, not a measured GitHub cache transfer. - Same real UI inventory for a retention-test-only change: 861 files; retained selection fell from 413 to 401. All 12 omitted files are raw-input consumers; edits to their inputs remain covered by regression tests. - The first local worker comparison is invalid. Development dependency links disappeared during its second run. The two-worker baseline also had two macOS path assertions in company-skills. Neither result is counted as a successful speedup. Dependencies were restored. A local repeat was interrupted before producing results. The completed Linux comparison below supplies the valid paired evidence. - [Paired Linux benchmark](https://github.com/paperclipai/paperclip/actions/runs/41671244016): baseline and candidate ran the same 113 general-server files and 1,777 cases on one host. Wall time fell from 381.29s to 346.89s (9.0%). Both exited successfully and file/case inventories matched exactly. The fixture command took 31.77s versus 28.65s. This is one sequential pair, excluding setup, queueing and the full CI critical path. - [Repeat paired Linux benchmark](https://github.com/paperclipai/paperclip/actions/runs/41675496862): the same 113 files and 1,974 cases took 485.06s in the baseline versus 368.54s in the candidate (24.0%). File and case inventories again matched exactly. The baseline includes newer master changes than the first pair, so these are two independent comparisons, not repeated timings of an identical tree. - Initial full AWS profile: package suite passed 4,891 tests (289 files, one skipped); native suite passed 172 tests (9 files, 12 pre-existing platform/feature skips). Native execution took 239.06 seconds; the older sample ran 165 native tests in 391.29 seconds. The package lane now takes 11m01 including setup and debug compilation. This is an indicative cross-run comparison, not a paired full-CI speedup claim. - The initial run failed two UI fixtures, one serialized inbox settlement assertion, and one dispatch idle assertion. The two UI failures reproduced locally. The fixes preserve authorization, admission, and zero-idle-read assertions. All four affected files passed locally: 48/48 tests. Token gates passed. All four also passed in the next AWS profile. That profile passed every other workload but timed out the 16,400-row keyed attention fixture at its existing 180s limit; this is accounted for by the explicit fixture-statistics change. Local focused comparisons pass both large cases before and after; their roughly 33s test execution is unchanged locally, so this is not claimed as an additional speed gain. A full local file run passed both large cases (7.57s / 18.05s) but hit the unchanged 30s teardown deadline on its first test while emitting thousands of cascade notices. Cleanup now suppresses those notices within its transaction; the complete file then passed 34/34 in 107.72s. The latest-head AWS run passed the previously failing shard: the keyed bulk case took 40.25s (previously exceeded 180s), the keyless case took 8.02s, and the complete attention file took 89.61s. The 16,400-row fixtures and original timeouts remain intact. - [Full CI profile before master integration](https://github.com/paperclipai/paperclip/actions/runs/41675497699) passed on commit `5858956d4ac2c99c911ad2447b63e1f8fc84e31e`, including all tests, build, typecheck and final verification. Greptile reviewed that exact commit at 5/5; the only actionable review thread is resolved. The separate Paperclip Review app is still queued. - [Final merged-head CI](https://github.com/paperclipai/paperclip/actions/runs/41678956312) passed on `499aa206a6e03bf8f3249692ea731367a000591d`: 48 successful jobs and one intentionally skipped file-selection job. Backup checkpoint qualification and legacy security status passed. Greptile reviewed this exact commit at 5/5. The PR is conflict-free; only the separate Paperclip Review app remains queued. - Integrated current master after #18798 and #18799 merged the inbox and UI fixture fixes separately. Preserved master's inbox settlement fix and verified its entire file (6/6 tests). Those three files are no longer part of this PR's diff. Removed the temporary `profile-ci` label after both successful benchmark pairs; the normal planner still schedules every job because the CI configuration changes are global inputs. - Second AWS profile: native tests took 229.97s with the same 172 passes / 12 skips; package tests took 427.44s with 4,891 passes / 10 skips. The complete package job took 10m44, including setup and 79s debug compilation. - The PR uses the trusted workflow from master. Candidate worker/layout code runs now; workflow-level debug restores begin after merge and the first successful master cache writer. ## Risks - Extra workers can expose load-sensitive tests. The final full CI profile passed, but one run is not a long-term flake-rate measurement. - The cache is an optimization only. Misses rebuild, and test commands still execute. A new Cargo workspace member requires a cache-contract review. - Compiler alternatives remain experiments. Required TypeScript checks are unchanged. Typechecker experiment: three sequential checks per tool on identical built projects, with TypeScript 7.0.2, Bun 1.4.4, and tsrs 0.20.1-ts7.1.0-beta. Median seconds (TypeScript / Bun / tsrs): shared 0.429 / 0.083 / 0.157; UI 2.554 / 0.390 / 0.740; server 9.949 / 1.194 / 1.782. TypeScript and Bun passed all three projects. All tools caught the same four deliberate assignment, property, nullability, and generic-constraint errors. tsrs reported one additional server TS2345 diagnostic in the run-dispatch adapter, so it did not pass the project comparison. These local compiler times exclude generation and build steps; they are not full CI savings. ## Model Used OpenAI GPT-6 Codex. Tool use and code execution assisted this change. The exact deployment identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8c2502503d |
fix(github): preserve repository access during PR webhooks (#18805)
## Thinking Path > - Paperclip manages AI agents and their work. > - Agents need current repository access before they can start work. > - Managed GitHub access checks run while webhook delivery can update the same grant. > - Pull request events update webhook health but do not change repository permissions. > - The refresh guard treated those health updates as access changes and could exhaust its retry. > - This change keeps repository preparation available during normal pull request activity. ## Linked Issues or Issue Description **What happened?** A pull request webhook can make a managed GitHub repository refresh fail with `github_access_changed`. Repository discovery then omits the failed connection. Workspace admission can report that repository access is no longer available, even after the checkout succeeds. **Expected behavior** Pull request activity must preserve valid repository access. Installation changes and newer access snapshots must still stop a stale refresh. **Steps to reproduce** 1. Start a managed GitHub access refresh. 2. Process a pull request edit while the provider request is in flight. 3. Repeat on the bounded retry. The old guard rejects both checks because `lastWebhookAt` changes. ## What Changed - Use `accessRevision` and `lastAccessRefreshAt` as the existing access fences. Stop using webhook delivery time as an access fence. - Allow the existing commit merge to retain the latest webhook timestamp and health after a pull request event. - Exercise real webhook processing and PostgreSQL during refresh. Cover a pull request event, an access conflict followed by that event, and installation suspension. - Document the distinction between access changes and delivery health. ## Verification - Red: both pull request race cases failed on the old code with `github_access_changed`. - Green: all seven focused refresh cases pass. These include the existing concurrent access-change checks and the new suspension case. - Command: `pnpm exec vitest run server/src/__tests__/tool-access-service.test.ts -t 'binds a managed GitHub identity'`. - Full affected group: 526 tests pass across the tool-access, repository, and webhook suites. - `pnpm -r typecheck` and `pnpm build`: pass. - Current-head CI: 54 checks pass. The duplicate Paperclip review is neutral. Two Storybook checks are skipped. Greptile and the completed Paperclip review both rate this commit 5/5 with no findings. There are no unresolved review threads. - The duplicate local `pnpm test:run` did not complete. Its first four chat shards passed. It was stopped after CI completed on this commit; the full CI test matrix passed. The complete affected local group passed separately. - Retained earlier failure: the first affected-group attempt passed 525 tests and had one socket hang-up in the profile attention feed test. That test passed alone without changes. The unchanged base tool-access suite also passed all 498 tests, and the complete fixed affected group then passed. The cause of the socket closure remains unproven. - Environment: source checkout on macOS arm64, Node 26.4.0, pnpm 9.15.4, disposable PostgreSQL. Provider responses and Cloud transport use controlled fixtures. The webhook handler, refresh logic, transactions, and saved grant reads are real. - Live acceptance remains unverified. The available CLI token cannot list GitHub App installations. The affected staging deployment still serves the old code. This PR does not deploy or recover the original task. ## Risks - Removing an access fence could admit stale permission data. The access revision and refresh timestamp checks remain. The real suspension webhook regression confirms that stale provider data cannot restore removed access. - This change adds no retry, timer, schema change, or credential forwarding. ## Model Used OpenAI Codex, based on GPT-6. The exact runtime model ID and context window are not exposed in this session. Used reasoning, code editing, and terminal tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5f1b59ba2c |
fix(db): preserve binary data in checkpoint backups (#18650)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
e401ed006b |
test(inbox): wait for snapshot completion before fresh reads (#18798)
## Thinking Path > - Paperclip helps people manage AI agents and inspect their work. > - Mine combines task and run data behind the issue-read admission limit. > - Its cancellation test aborts a request while the server still owns its snapshot computation. > - The test waited for seven separate list requests and assumed the snapshot had also finished. > - CI can return those list requests first, so the next snapshot read correctly shares the existing computation. > - This pull request completes a snapshot read before it checks that a fresh read starts new work. > - The regression keeps its admission-limit checks without depending on database read order. ## Linked Issues or Issue Description **What happened?** Full [CI run 41671193176](https://github.com/paperclipai/paperclip/actions/runs/41671193176/job/126041533414) failed `retains admission until aborted snapshot work settles (coalesced reader: false)`. It expected nine computation starts but saw eight. The aborted snapshot was still running after the seven list requests returned. A follow-up snapshot request could share that work. **Expected behavior** The test must retain the occupied slot while the aborted work runs. It must wait for snapshot completion before it expects a new computation. **Steps to reproduce** Run the full serialized CI shard containing `server/src/__tests__/inbox-mine-routes.test.ts`. The assertion depends on read order and can pass in isolation. The unchanged focused cancellation cases passed locally after the CI failure. The test does not await the aborted server computation, which explains the race. **Paperclip version or commit** CI failure: `1859d17b3ada7865c0ad4c5835eda23c32e33a48`, based on `f7ad1dc7cacedf7153a45e854ddfdd544aa87abd`. This branch now includes master at `adb49a4434`. **Deployment mode** Source checkout; real HTTP requests and disposable PostgreSQL. Refs #18790, which introduced the snapshot test. The user-reported [run 41665098924](https://github.com/paperclipai/paperclip/actions/runs/41665098924) also had three UI fixture failures and several runner shutdowns. All three UI failures reproduced locally. This PR first repaired them; concurrent #18799 merged the same repair while CI ran. The branch preserves that upstream repair, so the remaining diff is this inbox test. No duplicate inbox completion repair was found. ## What Changed - Complete a snapshot request as a barrier for any remaining aborted snapshot work. - Then assert that a fresh snapshot request is a cache miss and starts exactly one new computation. - Retain the eight-slot limit, rejected ninth request, and coalesced-reader assertions. ## Verification - The first full CI matrix exposed the eight-versus-nine assertion failure linked above. Both UI shards passed after the fixture correction. - All six real-database inbox route cases pass with the completion barrier. Server typecheck also passes after that repair. - Before the master merge, both affected UI suites and the account-scoped access reader passed 30 tests. The reader still rejects another account's response. Token gates pass. - On final merged head `f93d1e309a1c76bb6ded57f76efe08b876afa164`, the combined 36 focused tests, `pnpm -r typecheck`, and `pnpm build` all pass. - The local `pnpm test:run` completed four chat shards with 1,279 passing tests. It was stopped during shard five when master advanced. It is not reported as a full-suite pass. The final full CI matrix verifies the current head. - Full [CI run 41674890115](https://github.com/paperclipai/paperclip/actions/runs/41674890115) passed on final head `f93d1e309a1c76bb6ded57f76efe08b876afa164`: 48 jobs passed and one optional selection job was skipped. All PR checks are green with none pending. - Greptile and Paperclip Review are both 5/5 on that same head. Greptile added zero comments, and no review threads are open. ## Risks Low risk. The diff changes test synchronization only. It uses actual request completion and adds no timing delay. Production behavior, permissions, schemas, and API contracts do not change. No operator documentation change is needed. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1423d745e8 |
feat(server): expose owned sleep blocker diagnostics (#18632)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
f79264d253 |
perf(recovery): project current wait facts during action reconciliation (#18793)
## Thinking Path > - Paperclip manages work and execution for AI agents. > - Recovery actions keep unresolved work visible and preserve its owner. > - Action reconciliation checks whether a durable wait or another live path has appeared. > - Its current-wait check collected a complete retry fingerprint through seven queries. > - This caller used only the durable-wait flag and reason. > - This pull request reads those facts in one fresh statement and shares the existing reason policy. > - The local service uses less CPU. Fresh mutation guards and full retry fingerprints stay in place. ## Linked Issues or Issue Description **What existing behavior does this improve?** Active recovery action reconciliation. **Subsystem affected** server/ — orchestration and recovery. **Current behavior** Each admitted source action collects blockers, children, interactions, approvals, work products, runs and wakes. It sorts and hashes these records. This caller uses only the durable-wait flag and reason. **Proposed behavior** Read three current existence facts in one company-bound statement. Use the same reason policy as the complete fingerprint collector. Keep child checks, outside-action checks and mutation services. **Reason and benefit** The actual local PostgreSQL service uses less CPU across 8, 32 and 100-action synthetic cohorts. This removes unused work on mostly empty results. The live workload mix and fleet benefit remain unknown. **Breaking changes** No API or schema changes. Query facts now share one statement snapshot. Scalar issue fields still come from the source cohort read. **Additional context** No duplicate projection change was found in the public PR search. The roadmap has no overlapping core feature. This is a narrow performance change. ## What Changed - Add a current durable-wait reader and share reason selection with the full fingerprint collector. - Use the reader only in active action reconciliation. Keep the full collector for retry and child policy. - Add PostgreSQL regressions for reason priority, scope, board waits, native holds, healthy children and concurrent decisions. - Add a manual reproducer and local paired results. Each invocation uses a unique evidence directory and exclusive output files. ## Verification - Focused PostgreSQL regressions: 11 passed. - Separate actual-service proof: 7 passed. Four CPU tests were excluded. Results, records, semantic relation edges and notifications match the pinned baseline. Late query facts and a concurrent action decision are covered. - Retained primary local paired CPU medians: 8 actions 7.091 → 2.886 ms; 32 actions 21.922 → 8.374 ms; 100 actions 68.384 → 32.718 ms. All nine CPU and wall pairs improve at these sizes. The data file retains all 72 primary samples and ranges. - Warm application counts: 81 → 33 queries at 8 actions; 321 → 129 at 32; 1001 → 401 at 100. Rows rise from N to 2N because each EXISTS statement returns one row. - See `doc/benchmarks/2026-11-03-active-recovery-projection.md` and its JSON data for commands, provenance and limits. - `pnpm -r typecheck` and `pnpm build` passed on head `f2f8c4a755ac67679d78c6005f420a21a101c1ad`. - Shipped manual helper smoke with `--proof-only`: 7 passed, 4 timing tests skipped. All count-proof sample arrays are empty. Unique output names and cleanup of the three temporary source files were verified. - Exact-head CI completed with 52 successful checks, three intentional skips and a successful Snyk status. Greptile completed successfully at 5/5 with no inline findings. The `review` workflow passed commitperclip dependency and PR quality gates. No separate Paperclip Review check or verdict was published. The local `pnpm test:run` attempt passed all six chat shards (1,834 tests), then reported 62 Muse failures and one gateway failure in the general-server batch. The initial cases each reached the unchanged 15-second timeout. The Muse case was followed by fast failures. After the full batch breadth and complete CI coverage were established, the owned known-failed run was interrupted once with SIGINT. Remaining general-server, serialized and workspace scopes were not completed by that local invocation. Its interrupted tool exit code is not a pass. Each initial timeout case then passed in one fresh isolated control. A single fresh full Muse file passed all 256 tests. Source review found no changed projection path in either initial case. The original timeout causes remain unresolved; later passes and green CI do not erase them. The first reserved CPU run lost its raw files after a harness path error caused an unintended repeat to overwrite them. Its summary and log remain historical evidence. The separately retained repeat is the primary inspectable evidence above. It had no renewed quiet reservation. The runs are not pooled. The corrected untimed proof and public helper prevent output replacement. No third CPU run was made. ## Risks - One statement observes its query facts together. This does not reproduce every race history of seven concurrent statements. - Existing action update predicates, issue mutation locks, board ownership, native reconciliation holds and company guards remain in place. - The evidence is a local source service benchmark on macOS and real PostgreSQL. It is not deployed CPU or request latency acceptance. The census was startup-adjacent. Conditional scale suggests only a modest fleet return. - The original local batch timeout causes remain unexplained. No hosted, Sentry or OTLP improvement is claimed. No merge or deployment is requested. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, tool use and code execution. The host did not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused and isolated controls; the broad local failure/interruption is described above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f7ad1dc7ca |
fix(workspaces): make attached project repositories writable (#18770)
## Thinking Path > - Paperclip lets people assign work to AI agents and inspect the result. > - A task can use repositories from more than one project. > - Remote workspace preparation treated every additional project as a read-only source. > - This rejected valid Git repositories before the provider started, even when no source was read-only. > - The task page then hid the preparation reason behind a generic failed-attempt notice. > - This pull request prepares authorized repositories as writable task copies and shows redacted workspace validation errors. > - People can work across both projects and see the reason if folder preparation fails. ## Linked Issues or Issue Description **What happened?** A remote task with a selected Git-backed project and another mentioned Git-backed project failed before provider execution. It reported `remote_workspace_readonly_sources_unsupported`. Neither project had a read-only setting. The task notice showed only that the attempt failed. **Expected behavior** Each attached repository should be writable inside the task workspace. This should preserve source access checks and saved work. A workspace validation failure should explain its cause on the task page. **Steps to reproduce** 1. Configure remote workspace storage and a native sandbox environment. 2. Select a Git-backed project for a task. 3. Mention another authorized Git-backed project in the task. 4. Start the task. Preparation rejects the additional source before it starts the provider. **Paperclip version or commit** Reproduced against `dc9f387edc`, before this change. **Deployment mode** Managed remote workspace storage. Local regression checks use disposable PostgreSQL and independent filesystem processes. Related merged work: Refs #18403 and #18504. Those changes established remote storage and metadata-only references. They still reject configured cross-project repositories. No duplicate writable-repository repair was found. Refs #18774 for the test-only retention-fixture repair reused to unblock CI. ## What Changed - Add every configured Git repository of an authorized referenced project to `.paperclip-repositories/` as an editable task copy. - Retain saved paths, refs, pins and files. Recheck retained project access and source trust. Keep explicit read-only, local-source import and remote feature gates. - Keep committed snapshot repository coverage separate from later requests. Honor configured default refs, resolve remote branch names, and publish secondary checkouts only after checkout succeeds. Share configured-ref resolution between root and attached-repository scripts. - Include redacted workspace validation messages in the task interruption projection and existing failure notice. Preserve Retry permissions and render messages as plain text. Explain retained-folder access and trust denials while preserving the dedicated low-trust project-access path. - Add source, authorization, redaction, UI and save/restore regressions. Update the implementation contract and workspace maps. - Reuse the reviewed test-only repair from #18774. Batch the shared-workspace history fixture, verify its complete privacy-source map, and drain owned test bodies before cleanup. Production retention behavior is unchanged. ## Verification - Red before repair: four source-resolution cases failed with the reported read-only error. The task-error projection test also failed because it omitted the cause. - Green: 49 source and transport tests, 16 task projection and redaction tests, and 10 UI tests. The assembled preparation check passes. All 48 remote workspace lifecycle tests pass. The full preparation suite passed all 126 assertions but its cleanup hook exceeded 30 seconds. The unchanged base reproduces the same cleanup timeout with all 126 assertions passing. The focused preparation check finishes cleanly. - The real PostgreSQL/Git lifecycle checks cover both first-turn and later project attachments. They verify the configured release branch, saved edits, replacement allocations, unchanged source folders, committed archive contents, and released writers. - Review regressions were red before repair: the default ref became null, later attachment failed during restore, non-default remote branch checkout failed, and invalid secondary refs left partial checkouts. They now pass. - The local browser shows the workspace validation explanation and Retry action. The fixture uses a saved failed run; this proves the error display, not a successful live sandbox run. - `pnpm -r typecheck` and `pnpm build` pass. Map, module-boundary and token checks pass. - A local two-turn acceptance probe repeated at `17ed67eaa7` used real Codex CLI 0.165.0, `gpt-6-astra`, and `xhigh` effort. Codex wrote files in both task repositories. The services saved them, removed the first allocation and its Git sources, restored another allocation, and Codex read the saved files and wrote a follow-up in both folders. This uses the real source resolver and persistence services with a local command-runner stand-in for sandbox transport. It does not qualify Daytona, managed subscription forwarding, or the original task. The first probe waited for stdin; closing the test harness pipe fixed that setup error before provider work. - `pnpm test:run` passed the first chat shard, then failed one chat response concurrency assertion in the second shard. That case passes in isolation on this head. The same isolated chat case passes on the unchanged base. Both corresponding unchanged-base chat shards also pass (351 and 324 tests). The cause of the earlier wider-suite chat failure is not established. The preparation cleanup timeout and chat failure remain part of this record. - CI at `17ed67eaa7`: all other verification jobs passed. Browser shard 6 initially reported a request callback from the preceding test after teardown; its single unchanged-code retry passed. Server shard 10 timed out the 2,100-row retention confirmation fixture at 15 seconds on both attempts. That exact case passes locally, but the repeated CI failure required a repair. Its setup repeatedly rebuilt a shared-workspace privacy map. The logs and failures remain part of this record. - Reused the test-only repair from #18774 without changing its source file. At `ab774e9418`, all 54 retention tests and all 25 ordinary production privacy tests pass locally (79 total). Full CI [41650626903](https://github.com/paperclipai/paperclip/actions/runs/41650626903) passed on that commit, and its exact-head Greptile review was 5/5. The fixture preserves its 2,100 rows, query-bound assertions, privacy-source map, and completed-history test deadlines. - The additional review identified retained-source access/trust denials that still bypassed the workspace explanation. Real-resolver regressions reproduced both plain-403 failures before the correction. Both now yield a safe workspace validation explanation, while the dedicated low-trust denial and saved workspace remain unchanged. The optional duplicate ref-resolution block is also consolidated. - Final review corrections: 65 source, transport and error-projection tests passed, followed by all 48 lifecycle cases against an owned disposable PostgreSQL database (113 total). The initial combined command skipped the explicitly gated lifecycle suite; the subsequent dedicated invocation enabled and passed all 48. Full workspace typecheck and build pass. The first standalone lifecycle launcher lacked the repository TypeScript loader; rerunning with that loader resolved its import setup error. At previous head `7be02ea09c4ac76df560e8b517050cdf398f33f0`, full CI [41654422329, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41654422329/attempts/2) passed, Greptile is 5/5 on that exact head, and all review threads are resolved. The first attempt timed out the unchanged voice-session report/history case at 15 seconds; 3,027 cases in its shard passed. That exact case passed locally, then its single unchanged-code shard retry passed. The wider-suite timeout cause remains unproven. The completed Paperclip advisory check is neutral; its earlier findings are corrected. No staging deployment or original-task recovery is claimed. - Master advanced while checks ran, producing conflicts in two adjacent test additions. Merged `27d35314be` into this branch as `d5c8d552939c86056abf8fe10833881b2ade41a4`, preserving both workspace-error tests and the audited Stop/receipt tests. Kept workspace maps to minimal corrections under the updated map rule. All 82 source, transport and interruption cases, all 48 real-database save/restore lifecycle cases, full typecheck, build, and map checks pass locally on the combined code. Current-head [CI 41660719214](https://github.com/paperclipai/paperclip/actions/runs/41660719214) and Greptile are pending; the earlier verified head has not been merged. ## Risks - Additional remote project repositories are now editable task copies. Shared source folders and upstream push permissions do not change. - Snapshot coverage uses existing workspace metadata. Older workspaces capture their previous inventory at reservation; successful save publication advances coverage and snapshot identity in one transaction. No SQL migration is needed. - A retained copy can contain files from a project that is no longer mentioned. Preparation rechecks that project's current access and trust policy before reuse. - Explicit read-only descriptors still fail. Source-less projects remain metadata-only. Local folders still require import. No migration or provider-pack update is needed. - Error display reads only workspace validation errors. The existing run-secret registry and username censor run before display truncation. A redaction failure omits the explanation. - The retained fixture repair pauses its row trigger only within the isolated seed transaction, rebuilds the exact source map, and restores the trigger before commit. A rollback regression and 25 ordinary privacy cases pass. Its paged-case test budget honors the existing bounded inner waits; production deadlines are unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution. The exact serving model ID and context-window size are not exposed by this session. The test-only fixture changes are reused from #18774, whose model record identifies OpenAI GPT-6.1 Sol. The optional live acceptance probe uses `gpt-6-astra` with `xhigh` effort through Codex CLI 0.165.0. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b8f8b904fa |
feat(routines): preserve automation ancestry through webhooks (#18772)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Routines let external services start agent tasks through a normal webhook. > - A repair routine must recognize failures from work that it created. > - The receiver added in #18714 put Sentry payload parsing and repair policy in core routine intake. > - This pull request replaces that receiver with verified creation provenance on normal routine calls. > - Sentry can filter the resulting tags before it sends a normal routine webhook. > - The benefit is reusable provenance and loop prevention without a provider-specific route. ## Linked Issues or Issue Description **Problem or motivation** A Sentry repair alert can start another repair when its own work fails. The dedicated receiver in #18714 solved this through a Sentry-specific payload contract. Ordinary routine webhooks also need a trusted way to retain the automation that created their tasks. **Proposed solution** Use the existing public `/fire` route and its current authentication. An optional source run credential proves a live Paperclip caller. Store the caller separately from webhook JSON. Follow recorded task creation links and export bounded ancestry tags on run-failure events. Let Sentry alert filters exclude repair-origin work. **Alternatives considered** Keep the dedicated Sentry receiver, accept sender-supplied origin IDs, or filter only the immediate routine. The first couples core intake to one provider. The second accepts unverified claims. The third misses child tasks, standalone follow-ups, and calls through another routine. **Roadmap alignment** This improves the shipped Scheduled Routines and Activity log capabilities. It does not add a new scheduler or incident system. Related work: #18714 is the implementation replaced here; #9727 adds source-task navigation and UI attribution; #10206 proposes exception-led routine dispatch. This PR uses existing task creation records and does not implement either of those broader proposals. ## What Changed - Remove the Sentry-only route, parser, authentication exception, and payload-specific tests and documentation from #18714. - Add optional `X-Paperclip-Source-Run-Token` support to normal routine webhooks. Validate the standard run JWT, company, instance, agent, live run, and cancellation state. Authenticated routine API calls also retain their validated caller. - Add nullable `routine_runs.created_by_run_id`. Keep the reference after source-run deletion so missing history becomes unknown. Separate callers during coalescing, including legacy open tasks. - Follow company-scoped routine receipts, creator runs, task parents, and legacy creation activity. Mark missing, foreign, cyclic, private, or excessive ancestry for safe alert filtering. - Add event-local `automation_origin_status`, `automation_routine_ids`, and `task_shareable` tags through the existing Sentry SDK hook. Cap ancestry at 32 reads and share a 750 ms deadline across metadata and ancestry lookup. Keep the SDK one-second wait and eight-lookup concurrency bound. Redact the source credential from HTTP logs and Sentry request headers. - Preserve unexpected database failures as server errors without rejection bookkeeping in an aborted transaction. Document 409, 415, and 422 responses for the normal webhook. - Keep `routineScopedRunId` as generic opt-in `X-Paperclip-Idempotency-Scope: routine` support. Default trigger receipts and timestamped HMAC replay protection retain their current scope. - Add [the Sentry repair tutorial](https://github.com/paperclipai/paperclip/blob/codex/routine-creation-provenance/docs/guides/board-operator/sentry-run-failure-repairs.md), API documentation, OpenAPI headers, and capability-map updates. ## Verification At commit `0f4a0051528edd1367a921a76a8050c3fd087872`: - `pnpm -r typecheck` and `pnpm build` passed for the full workspace. Migration numbering and safety checks passed. - The focused service, route, Sentry, redaction, and OpenAPI suites passed 228 tests in five files. The database Sentry resolver and OpenAPI invocation passed 33 tests in two files. These are separate invocations with some overlap. - The full routine HTTP suite passed all 15 tests. The authenticated API case failed before the caller-wiring fix and passed after it. The fixture loads the real route-owned services. No test deadline was raised. - The real `@sentry/node@10.71.0` suite passed all 12 tests. It uses an in-memory transport and sends no events to Sentry. It covers terminal failures, saved-message interruptions, event isolation, and source-token redaction. - Regression tests also cover forged, expired, foreign, and cancelled caller credentials; payload spoofing; inherited routine ancestry; legacy coalescing; deleted history; private work; cycles; query and time limits; database error causes; and routine-wide receipts. The new error and budget cases failed before their fixes. - `node scripts/check-maps.mjs` and `git diff --check` passed. The map check covers 35 areas and 192 capabilities. - The full local Vitest command was stopped for a required master merge after four chat shards passed 1,278 tests. No full local pass is claimed. An earlier combined HTTP run hit an existing malformed-ID timeout whose original cause is not established; the final full HTTP suite passes. An extra local run of the unchanged heartbeat scheduling file passed 12 of 13 tests and failed a readiness-status assertion. Its logs also reported a missing fixture user membership. The same 13-test file now passes in current-head CI. - All GitHub Actions checks pass on this commit. Three jobs initially lost their self-hosted runners. One server shard initially failed while deleting a company in an unchanged heartbeat test because of a `company_skills` foreign key. One retry passed all affected jobs without code changes. That server shard passed all 499 tests, and all eight browser shards passed. - Fresh Greptile review is 5/5 on this commit. All six review threads are resolved. The separate Paperclip Review check remains queued. Its bot reports that work is paused. A maintainer must resume the assigned review work to complete that check. No full green check state is claimed while this external gate is pending. - Live Sentry configuration and deployment are outside this PR's acceptance evidence. The tutorial describes setup, a controlled failure, and a repair-origin exclusion check for operators to run after deployment. ## Risks - Normal app webhooks retain the sender's full JSON in company-readable routine history and the execution task. Alert filters and task visibility are not content scrubbers. Operators must scrub sensitive event data in Sentry before forwarding it and keep connection tests in setup mode. - Existing users of `/sentry-run-failures` must move to `/fire` and add the documented Sentry filters before enabling repair intake. The old route is removed. Stored tasks and receipts remain. - The additive migration uses `ADD COLUMN IF NOT EXISTS`. It does not rewrite historical tasks or infer missing provenance. - Unknown or unshareable ancestry is excluded by the tutorial's alert rule. Missing history, an enrichment timeout, or more than five routine IDs can therefore suppress a repair webhook while retaining the Sentry error. - An external webhook without a verified source credential starts a new receiving-routine origin. Its JSON cannot establish upstream Paperclip ancestry. Sentry must not store a Paperclip run credential. - Routine-wide receipt reuse needs a stable sender key. A static Sentry custom header is not a per-event key. The tutorial keeps delivery deduplication separate from loop prevention. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-6 model family, with reasoning, repository access, code editing, and test execution. The session does not expose the exact model variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b40e6edaea |
perf(inbox): combine startup reads and reduce refetch pressure (#18790)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Mine shows the work and failures that need the viewing user's attention. > - A reload waits for account access, task rows, parent captions and failure state. > - Those reads form several browser request stages before useful rows appear. > - Live invalidations can also restart the same task-list request more than once. > - This change supplies startup data together and removes redundant refresh work. > - It preserves current authority, failure presentation, archive state and row stability. ## Linked Issues or Issue Description Refs #18645. Related: #18642. This change implements the read-receipt behavior independently on current master. It does not use that PR's branch or patch. **What happened?** Mine starts its primary request after the session, company and board-access reads. It then requests parent captions and rich failure context. Four observed loads spent another 0.45–0.70 seconds after the primary rows arrived. One slower load waited 1.71 seconds for parent context. The same run observed two HTTP 429 responses. These timings identify request waits; they do not attribute the delay to SQL or Node CPU. Read receipts also invalidate general task collections and dashboards. Overlapping parent and child invalidations can start two requests and abort the first. **Expected behavior** The initial page should use one account-safe startup projection and one complete Mine response. A read receipt should refresh the affected viewer's inbox state. Each invalidation batch should refresh a covered query once. Required captions and current failure state must be ready before first reveal. Existing rows should remain usable during a transient refresh error. **Steps to reproduce** 1. Open a populated Mine inbox with parent tasks outside the visible index and a current failed run. 2. Reload with browser HTTP cache retained and application query state empty. 3. Inspect the session, company, board access, task-index and context request sequence. 4. Open or mark a task read and inspect the unrelated collection refetches. **Paperclip version or commit** Baseline: `44a9a043b80c18314d8dd89556f444dae696d3d6`. **Deployment mode** Authenticated board UI. Local proof uses the real APIs and database with a synthetic session resolver. No hosted candidate deployment is claimed. ## What Changed - Add optional, same-user `boardBootstrap` data to the existing session response. Seed account-keyed company and board-access reads from it. Keep ordinary endpoint reads for fallback, retry and live reconciliation. A returning account cannot reuse revoked cached access. Cache collection and later real refetches remain effective. - Add the board-only Mine snapshot endpoint. Reuse the canonical issue selector, private-task filtering, run-read decisions, secret redaction and interruption/recovery presentation. Assemble the fixed 100-task page, parent captions and bounded 200-run failure window on the server. Start independent reads together. Optional source errors keep successful task rows usable. - Keep array-shaped source caches for archive and undo. Key Mine source, context and reveal state by account. Validate the response's user stamp before committing it. Remove array-only cross-tab Mine broadcasts, which cannot carry its paired failure context. The server retains authenticated in-flight coalescing. - Scope read-receipt invalidation to the affected viewer's inbox, badges and chat unread state. Keep company and task audit views fresh. Preserve genuine task mutations, run updates and subscription-gap reconciliation. - Remove covered scalar-prefix invalidations within a batch. Preserve predicates, exact matches and different query/refetch constraints. A later event can still refresh again. - Retire Mine in-flight reuse after local company activity or run lifecycle changes. A post-archive read starts fresh. Old work keeps its admission slot until it settles, even if every HTTP client aborts. The eight-compute limit stays in place. - Add API, component and browser regressions. Extend the production timing harness with matching API/UI artifact receipts, retained-cache reloads, request status counts, layout checks and optional controlled network latency. Update API and verification documentation. ## Verification - Full `pnpm -r typecheck` passes. - Full `pnpm build` passes. - Token gates, map checks and diff whitespace checks pass. - 72 Inbox component tests pass. Startup-specific checks pass, including account switching, revoked cached access, malformed/absent bootstrap data and cache collection. The canonical authenticated bootstrap route also passes against real PostgreSQL. - Six real PostgreSQL Mine route cases pass, including full equivalence with legacy authorized projections, privacy, fixed bounds, membership revocation, coalescing, archive freshness and aborted-client admission. Four focused legacy coordinator controls pass. Six assembly tests pass. - Refetch and invalidation regressions pass. Meaningful pre-fix tests observed two request starts and one abort for overlapping invalidations; the fix produces one start and no abort. The archive race test observed the post-archive GET joining the old body before the fix. - All 11 existing startup browser checks pass, including Tasks/Mine code loading, desktop/mobile bounds, non-streamlined Coordinator, task navigation, failed preloading and Back position. The nine focused startup, fallback, partial-source error, rate-limit recovery, archive/undo and geometry cases also pass. The 429 cases use the inherited retry window. Archive and keyboard undo are corroborated through the real API and reload. Authentication and failure fault injection in these cases are explicitly simulated. - Matched production before/after timings are complete. Full aggregate Vitest coverage is left to CI. At 15:15 UTC, the final-head runner gate and `review` workflow pass. The 45 build/test/policy jobs are queued; the Build job has no `ubuntu-latest` runner assigned and has executed no steps, so remote aggregate results remain unverified. Paperclip Review and Superagent Security Scan are in progress. Superseded runs for this PR were cancelled. No current-head check has reported a failure; the CI checklist remains open. The remaining step is runner allocation and completion of those external checks; persistent queueing needs the repository CI owner's attention. - Greptile completed on final head `ecde295565309275c4f2fae903caf6892f1eaafd` at 5/5. The map-scope finding is fixed and resolved; there are no open review threads. ### Matched browser timings Five alternating baseline/candidate pairs per network scenario, measured serially in headful Chrome 156.0.8078.80 on Apple silicon with Node 24.22.2 and a 1440×900 viewport. Both production UIs use their matching real API revision against one fixed disposable PostgreSQL fixture: 34 tasks, 31 visible task rows, one paused agent and one terminal failure. No active agent/provider work. HTTP cache is retained; each initial sample reloads the document and clears application query state. Warm navigation and Back retain query state. CPU and throughput are native. The 100ms scenario uses CDP minimum request-to-response-header latency without request interception. It is a controlled request-dependency experiment, not a staging reproduction. Values are median milliseconds with min–max in parentheses. Paired saving is baseline minus candidate for each alternating pair; its median differs from the difference of arm medians. Positive saving favors the candidate. | Network / journey | Baseline | Candidate | Median paired saving | | --- | ---: | ---: | ---: | | Native / initial rows | 187.6 (128.9–208.5) | 191.7 (130.3–210.9) | −23.3 | | Native / Recent → Mine | 100.17 (99.90–101.87) | 100.07 (99.88–101.37) | −0.16 | | Native / task → Back | 70.45 (56.63–72.02) | 66.64 (56.52–67.71) | +4.94 | | 100ms / initial rows | 930.3 (914.1–1015.7) | 799.0 (697.1–826.0) | +211.8 | | 100ms / Recent → Mine | 100.09 (97.73–100.16) | 100.18 (96.65–100.98) | −0.02 | | 100ms / task → Back | 74.97 (62.77–75.82) | 75.47 (61.51–76.19) | +0.35 | The controlled-latency initial pairs all improve, saving 108.3–217.0ms. Native initial pairs have mixed signs (−67.5 to +78.1ms), so they establish no reliable native page-speed improvement. Warm and Back timings are effectively unchanged. Median primary-request start is 110.4→135.2ms natively and 497.8→480.9ms with 100ms latency; these runs do not establish a large startup-only timing gain. API requests through the two-second post-reveal window fall from median 32 (32–37) to 27 (25–28) natively, and 37 (33–38) to 30 (25–32) with 100ms latency. All 20 samples preserve the same ordered 31 task links, first-row DOM identity and geometry. All recorded API responses are HTTP 200; zero unfinished requests, 429s or JavaScript errors. Lower 429 incidence on staging remains unproven. Measured source: baseline `44a9a043b80c18314d8dd89556f444dae696d3d6`, candidate `4c12b2ef0ecf0754676e99bd6b9399e84408646c`. Final head `ecde295565309275c4f2fae903caf6892f1eaafd` changes only two feature-map files and existing browser test expectations since that measured candidate; production executable inputs are identical to the measured candidate. Asset hashes, API health revision receipts and every raw sample were retained. These results do not claim hosted-page improvement or deployment of this PR. ## Risks - Startup enrichment adds work to the session read. Failure of this optional projection falls back to the ordinary company and access endpoints. It does not invalidate a valid session or replace per-request server authorization. - Account changes, stale cache entries and same-company navigation can expose old presentation data. Account keys, principal stamps, reveal scopes and canonical bootstrap replacement have dedicated negative cases. Permission failures hide cached task rows; transient network, 429 and 5xx failures preserve them. - Current failure state can differ from the historical failed run. The snapshot uses the existing canonical rich projection and verified missing-ID coverage. Partial context failures retain an explicit retry path. - Mutation retirement follows events delivered to the current server process. This adds no cross-replica invalidation protocol or distributed read-after-write guarantee. - New UI assets require the matching API. Older UI/session consumers retain their existing response and endpoint behavior. There is no database migration, provider call or deployment change. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, code execution, browser automation and delegated source review. The exact runtime model ID and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
5dee4993f1 |
fix(server): distinguish aborted HTTP responses in access logs (#18781)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators use HTTP logs to investigate task updates and database contention. > - Node starts each response with status 200 before it sends any headers. > - The custom HTTP formatter printed that status when the client disconnected before a response. > - This made an aborted request look like a successful task update. > - This pull request labels incomplete responses as aborted and preserves error severity. > - Operators can distinguish HTTP delivery from database commit evidence. ## Linked Issues or Issue Description - Refs: #18753. - Related: #14369 removes writes from legacy comment attribution reads. This change addresses a separate logging defect. The missing-write incident in #18753 remains unconfirmed. We reproduced a misleading `PATCH /issues/:id 200` log when the client received no HTTP response. We did not reproduce an HTTP 200 delivered before the mutation committed. ## What Changed - Label premature connection closure as `request aborted before response` or `request aborted during response` in both success and error log paths. - Log aborts as warnings, including on routes whose successful polling logs are normally silent. Keep explicit errors and 5xx responses at error severity. - Keep private webhook paths and query values out of aborted-request logs. - Add real HTTP disconnect coverage and real PostgreSQL task-update tests with synthetic data. - Document that HTTP completion logs do not establish database commit or rollback. ## Verification - Baseline: the new HTTP disconnect test failed because the log said `PATCH /api/issues/synthetic-task 200` at info level, while the client received no response. - `node node_modules/vitest/vitest.mjs run server/src/__tests__/http-log-aborted-response.test.ts server/src/__tests__/http-log-redaction.test.ts server/src/__tests__/issue-update-lock-contention-postgres.test.ts` passed: 81 tests in 3 files. - `pnpm -r typecheck` and `pnpm build` passed. The build used a second clean clone at the exact PR commit. - GitHub CI completed at `e3e96d836119df5fa1c7f22615484265de5aa700`: 52 checks passed and two Storybook checks were skipped. Greptile returned 5/5 with no findings. - Local `pnpm test:run` passed all six chat shards (1,834 tests). Its general server stage then reported four failures. We stopped the run after preserving these results and comparing the failing cases. The later workspace and serialized stages were not completed locally. The full local suite is not green. - The four observed failures were the two OAuth concurrency cases in `tool-access-service.test.ts`, the bounded workspace conflict case in `workspace-runtime.test.ts`, and the `enableNativeRunner`-disabled stop-control case in `muse-control.test.ts`. - Comparisons used the same sanitized environment on unchanged master `2b14ec03a48f17606db53fd7fc0edc53bc930409` and this PR. Both OAuth cases pass in isolation on both revisions. The full tool-access file passes all 498 tests on the PR; master has a different 15-second timeout, with 497 tests passing. The workspace case passes on the PR but fails port-reservation setup on master. The Muse case passes on both. These results do not explain or clear the original broad-suite OAuth and Muse failures. - Tested on macOS arm64, Node.js 26.4.0, pinned pnpm 9.15.4, and disposable embedded PostgreSQL. The API fixture uses a synthetic process-adapter agent. No model provider runs and no production database or real credentials participate. - The PostgreSQL fixture verifies an actual lock waiter with `pg_blocking_pids`. A blocked issue update does not return 200 before release. After release, status, monitor policy, inline comment, and audit receipt are persisted before the completed response. - Holding the company row with `FOR NO KEY UPDATE` does not block these fixture writes. They return 200 with persisted data while the compatible parent-row lock remains held. - An injected database trigger failure returns 500 and leaves the issue, comment, and mutation audit unchanged. Two influence-attempt receipts remain under the existing documented cap policy. - The combined lock/disconnect/failure test verifies no delivered response, an aborted log, and rollback after an injected database failure. The injected failure is not evidence of the reporter's production root cause. - Source inspection at release `v2026.1023.0` (`1b4c803816e82b0265c8422e1a804327a7ed8c54`) finds the same unconditional success formatter and awaited issue-update transaction. That release was not runtime-tested. ## Risks - Log consumers that parse the literal method/path/status message must handle the aborted-request message. - Client disconnects during streaming or SSE responses also use the aborted warning. Expected disconnects may therefore produce more warnings. - A disconnected request may still commit after the client leaves. This change reports transport completion and does not cancel or retry mutations. - This PR does not close #18753. The original missing-write cause and a received-200-without-commit scenario remain unconfirmed. - Local full-suite failures remain unresolved despite green CI. Keep this PR in draft until that verification gap is accepted or resolved. > This is a targeted bug fix. ROADMAP.md does not list overlapping feature work. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, tool use, code execution, and parallel subagent investigation. - The exact runtime model ID and context window size were not exposed to the agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
27d35314be |
feat: connect experimental Cursor Cloud to native Runner (#18069)
## Thinking Path > - Paperclip helps people manage AI agents at work. > - Native Runner owns execution and task authority. > - Operators need an explicit native Cursor Cloud choice. > - Tools/files need company, task, session and run scope. > - This connects setup and lifecycle to Runner Core. > - Legacy agents retain their selected runtime/history. ## Linked Issues or Issue Description Depends on #18068. Managed-only API placement and API-owned Stop containment are in #18310. Refs #5664, #11803, #17299, and #17471. No duplicate native integration was found. ## What Changed - Add explicit native setup, Test, saved settings and dispatch with no-prompt account/catalog/callback checks. - Scope MCP to company, agent, task, session and run through authenticated HTTPS. Register verified artifacts/work products; preserve unpriced accounting, completion and budgets. - Connect Question/Plan continuation, cold handoff and admitted fresh sessions; recheck waiting admission under the task lock. - Present audited Stop intent while preserving physical containment proof; bound typed Stop metadata. Preserve title edits through preference/navigation readiness. - Select one live/history representation per run to prevent duplicate settled cards. Integrate upstream task/read-state/retry/schema contracts and repair two test fixtures. ## Verification Core #18068 merged normally as `3910352c923a`. Product head `5a60f541c982` has passing repository CI/build/browser checks, including required `ci / verify` and `ci / e2e`, plus a fresh exact-head Greptile 5/5 review with zero unresolved threads. Two non-required external app checks remain pending: Paperclip Review is queued waiting for its assigned agent; Superagent Security Scan is reviewing the diff. These are not claimed completed. The normal merge endpoint enforces the repository rules; no policy or status override is used. The dependent Top PR is addressing its fresh deferred-Stop review finding. Product `5a60f541c982` passed 270 UI and 20 selected Cloud/native/disk-full tests, UI/server typechecks, tokens and maps. Top’s clean normal merge passed tokens/maps/diff checks; its matrix is pending. Earlier receipts retain Core’s 47 Rust, 4 compiled-native retirement and 19 settlement controls, normal Runner build, and runtime 9af build/recursive typecheck. Reproduced loading/shutdown CI fixture failures passed 7/7 browser and 121/121 unit tests, including a broken-unsubscribe negative control. No timeout/oracle was relaxed. The actual-parent restack adopts 75 mainline paths, including schema/permission and generic runtime changes, beyond the prior two fixture repairs and client projection fix. The full difference from hosted `9af1a5de42a2` is therefore larger than four files; backend source also differs. The merged config-read checks match Core391, and the Cursor placement/runtime/relay/selection modules match frozen9af; selected affected regressions pass. Paid proof keeps its actual 9af source/image identity. The separately tested live/history card fix is not claimed deployed on that host. On 9af, original saved settings survived reopening and fresh no-prompt Test authenticated catalog/model/binding; no new Save is claimed. Fresh Linux admission passed. With `claude-sonnet-4-5`, Artifact downloaded exact 27 bytes and matched 121 canonical identities in complete S3-served history. First Stop’s strict quiet FAILED on a 10.373-second gap despite independent cancellation/no late artifact. One targeted unchanged-sampler retry retained all 24 samples: 35.076 seconds quiet / maximum gap 4.584 seconds, 67 ordered canonical identities, and separate CANCELLED/artifact-absent status after the original 180-second deadline. The pre-reload duplicate settled card is disclosed. Exact-ID cleanup for the three new provider agents is awaiting authorization, so full grades and the fresh Artifact’s post-cleanup download remain blocked. A historical deleted-provider artifact was newly downloaded on 9af; original F226 Question/Plan/reconnect and cleanup proof keep their identities. Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; earlier title/startup/custody/Stop/deletion failures with unknown causes/bounded recoveries; `dea023` Stop’s warning/50 duplicate identities; pre-repair ACK and original Core CI failures. WSP’s 251-assertion, 7/8-suite attempt failed physical PostgreSQL teardown; the unchanged wrapper rerun passed 128. Retry alone establishes no cause or full local-suite pass. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and checked through behavior; unpriced provider dollar cost stays UNKNOWN. Callbacks need authenticated reachable HTTPS. Scope, revocation, budgets, completion and cleanup remain authoritative. Operator Stop intent is separate from physical containment. Saved legacy agents retain their selected runtime. Prior shared title-save/deletion failures remain disclosed. ## Model Used OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools, code execution, browser verification, and delegated code review. The serving model’s exact API ID and context window are not exposed. No value is inferred from the model catalog. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; pending gates are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3910352c92 |
feat(runner): add experimental Cursor Cloud provider (#18068)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner gives each provider durable execution, tool, and completion contracts. > - Cursor Cloud currently has a direct adapter outside that native contract. > - Cloud work needs exact ownership across HTTP, SSE, Stop, and recovery. > - This PR adds a qualified Cloud API driver and verified proxy to Runner. > - Product setup and hosted placement remain separate dependent changes. ## Linked Issues or Issue Description Refs #5664, #11803, #17299, and #17471. Product integration is #18069. Managed-only API placement is #18310. The related direct-adapter model/transcript work is not replaced. No duplicate native Cursor Cloud integration was found. ## What Changed - Add native `cursor_cloud` / `cursor_cloud_api` with explicit model/repository validation, bounded HTTP/SSE, durable per-turn checkpoints and a qualified Node proxy launched through the Rust facade. - Commit terminal custody before publication/release; coalesce consumers, preserve ambiguous create/Stop/recovery fences, reject late tools, and distinguish definitely-unsent oversized requests from ambiguous responses. - Support scoped cold attachment, current authority, reconnect and verified artifact staging. Keep unsupported warm/external and Windows paths closed. - Preserve early source cursors and verify immutable legacy prefixes before suffix replay, including supported compaction/masking. Optional proof publication failure falls back to complete cold verification. - Preserve canonical records at the durable native producer with JSON-safe privacy and the existing 8MiB bound; drain admitted successor flushes. Reconcile upstream semantic/pack/schema/lifecycle changes and retain S3 and legacy CLI checks. ## Verification Core head `37c442ec8dd4`: all 66 check contexts are terminal successful/neutral/skipped, including required `ci / verify` and `ci / e2e`; fresh exact-head Greptile is 5/5 with zero unresolved threads. Dependent Product `81e828d20a77` and Top `dd2959689627` include the later client handoff correction and are running their own fresh gates. Core: 47/47 Rust, 4/4 compiled-native retirement, 19/19 settlement and normal Runner build passed. Runtime Top `9af1a5de42a2`: build and recursive typecheck passed. Earlier source-scoped evidence includes 398 focused tests, five relay tests and token/type gates. The 251-assertion, 7/8-suite run failed physical PostgreSQL teardown; the unchanged stable WSP suite passed 128 through the documented wrapper. The downstream fixture corrections passed 7/7 browser and 121/121 startup tests without timeout/oracle relaxation; a broken-unsubscribe control fails. Hosted runtime `9af1a5de42a2` has a passing normal build and recursive typecheck. Latest dependent heads additionally contain a pure client fix for stale live/history overlap, with 48 focused tests, UI typecheck and token gates passing. Its new client is not deployed in the frozen hosted image; server/native/API/provider source is unchanged by these four downstream files. Fresh hosted Linux admission on `9af1a5de42a2` retired its owned runner/proxy/socket with zero prompts. The saved configuration survived reopening, and Test authenticated the bound key/catalog/selected model without a prompt. Artifact with `claude-sonnet-4-5` completed, downloaded the exact 27 bytes, matched 121 canonical identities against complete logs and independently reached FINISHED. The active Stop retry was clicked within its immutable boundary, produced one canonical interrupted turn and acknowledged run-only cancellation, retained a valid 35.076s quiet window (maximum gap 4.584s), and independently remained CANCELLED/artifact-absent beyond the original 180-second command deadline. All provider-observer credential callbacks, DB clients and loaders closed; accounting is settled and explicitly unpriced. The first fresh Stop retains its failed 10.373s sampling gap and every sample. On the upgraded9af server, the historical F226 artifact was reopened and downloaded with its exact original hash after its provider agent had been deleted/GET404 earlier today. Historical F226 full Artifact/Stop cleanup, Question, corrected Plan and reconnect retain their original identities. Deleting the three newly created disposable provider agents and the new-case post-delete grade remain blocked on explicit user approval after automatic review rejected expanding the two original approved IDs. No new-case cleanup or complete fresh matrix pass is claimed; this is separate from the directly observed task/Stop behavior and retained exact cleanup proof. Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; earlier startup-title, title-save, custody/Stop and deletion failures with unknown causes and bounded recoveries. `dea023` Artifact passed 17 checks but Stop failed its false warning and 50 duplicate log identities. Pre-repair ACK failure and original Core CI failure remain recorded; retry alone establishes no cause or full local-suite pass. Provider dollar cost stays UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. The caller’s explicit values are submitted and behavior checked live. Unpriced provider dollar cost stays UNKNOWN. Only qualified per-turn controllers are admitted. Uncertain work and incomplete cleanup stay fenced. Existing damaged/duplicate archives remain failed evidence. Prefix proof adds bounded object-store I/O; the 8MiB producer bound does not guarantee future providers’ output fits. ## Model Used OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools, code execution, browser verification, and delegated code review. The serving model’s exact API ID and context window are not exposed. No value is inferred from the model catalog. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; pending gates are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2b14ec03a4 |
fix(ai-connections): remove account execution locks (#18778)
## Thinking Path > - Paperclip manages AI agents and the accounts that supply their model credentials. > - Several runs can use the same AI connection at the same time. > - Runtime preparation treated unfinished credential cleanup as exclusive account ownership. > - Manual account tests also held a company credential mutex for the entire provider test. > - A failed cleanup could therefore cancel fresh runs before the provider started and keep them in automatic retries. > - This pull request removes that execution gate and those read and probe locks. > - Each invocation keeps its own credential home. Credential updates still use the existing freshness and authorization checks. ## Linked Issues or Issue Description **What happened?** Fresh runs stopped with `ai_connection_busy` and the message “The selected AI account is still completing credential cleanup. Retry shortly.” Pending cleanup from an earlier invocation blocked every new invocation of that account. Concurrent manual account tests also waited on a company mutex. **Expected behavior** AI connections support concurrent runs and account tests. Cleanup for one private runtime must not prevent another private runtime from using the same account. **Steps to reproduce** 1. Prepare a managed OpenAI account and retain an earlier runtime whose cleanup has not finished. 2. Prepare another run with that account. The old implementation rejects it before provider work starts. 3. Alternatively, keep one manual account test open and start a second test with the same account. The old implementation prevents the second provider callback from starting. **Paperclip version or commit** Reproduced with two PostgreSQL regressions on `17e8bd773543d62ab3f444c55255acfd82575efd`. Both fail before the fix and pass after it. **Deployment mode** The reported deployment used authenticated Cloud. Local reproduction uses disposable PostgreSQL and fixture credentials. No live provider credentials are required for these regressions. Related search found #15952, which concerns account removal and provider application archival. It does not remove this runtime cleanup gate. This fix does not introduce the account concurrency cap requested in #14564. ROADMAP.md was checked; this change restores concurrent account use. ## What Changed - Remove the account-wide cleanup assertion and cleanup mutex from run admission. - Read ordinary subscription credentials without acquiring the company file mutex or grant and secret write locks. Only an actual durable replacement recovery enters the existing write transaction. - Remove the mutex held across manual account tests. - Retry retained cleanup without making new runtime preparation wait for it. A home-specific cleanup check considers only that exact private home. - Keep cleanup attempts for one runtime idempotent. Keep physical retirement, credential freshness, reconnect, revocation and company checks. - Update concurrency, refresh and cleanup regressions, and the implementation contract. ## Verification - Before/after: the pending-cleanup and simultaneous-probe regressions both fail on the original code and pass with the fix. - Affected suites: 186 tests passed across `ai-connections.test.ts`, `antigravity-ai-connections.integration.test.ts`, and `retained-google-credential-cleanup.test.ts`. They use real PostgreSQL, encrypted fixture credentials and controlled provider callbacks. - `pnpm -r typecheck` passed. - `node scripts/check-maps.mjs` and `git diff --check` passed. - `pnpm build` passed on the rebased branch. The server typecheck and production server build both passed again on final commit `03bd3973bdf85108bbd82355abdb217f0e1c83bd` after the runtime-home ownership changes. - Caller coverage: all 166 cases pass across native credential handoff, managed Google subscription writeback, model discovery, heartbeat configuration and hired-agent execution. The full hiring file passed all 79 cases after its admission assertion was corrected. Together with the affected suites, 352 targeted cases pass. - The final home-binding check passed all 188 cases across the account, managed Google writeback, model discovery and retained-cleanup suites. - Review follow-up: two real-PostgreSQL cases pass for quota refresh and concurrent Google account tests. The quota case observes a credential row update in progress, verifies that the runtime takes no account file mutex, and confirms that the private home receives the committed replacement token. The Google case completes a second provider callback while the first callback remains open in a separate private home. - Greptile rated the runtime change 4/5 and found only misplaced map patch notes. Those additions were removed. The implementation documentation remains in `doc/connections/AI-CONNECTIONS.md` and the implementation contract. - `pnpm test:run` was attempted. Chat partition 1 passed 351 expanded cases. Partition 2 passed 324 cases and failed one Teams file-receipt API case at its 15-second deadline: `refuses generic resolution with stale version transfer preconditions without changing receipts`. That case exercises publication resolution, not AI account execution. Its isolated rerun passed. The remaining local partitions did not run after that failure. This is not a claim that the full local suite passed. - Wider caller checks initially identified old serialization expectations and two cleanup retry cases. Updated assertions now require concurrent account use. The partial-removal correction lets a retained home finish removal after a durable credential handoff. All affected caller cases pass, including the full hiring file. - Live provider concurrency and deployment recovery have not been qualified. This PR does not claim that the reported Cloud deployment contains the fix. ## Risks - A new invocation can use the currently saved credential while another runtime still owns a refreshed copy. The existing writeback checks decide which credential can replace the saved value. - Actual credential updates and durable replacement recovery retain atomic storage transactions. This change removes execution serialization, not the integrity checks on stored credentials. - Unfinished cleanup retains its private home and its existing idle hold. This change does not certify a previous provider process as stopped. - There is no database migration, credential format change, permission expansion or ambient-account fallback. ## Model Used - OpenAI GPT-6 through Codex desktop, with source editing, shell execution, PostgreSQL fixture tests and read-only browser diagnosis. The session does not expose the exact deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |