mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 19:28:46 +01:00
612d9931a6e789b36ac5ea23524ef8a266fe0d17
3054
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
612d9931a6 |
Support per-turn Runner sessions on ephemeral web servers (#18932)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server controls native Runner sessions in remote sandboxes. > - An ephemeral web server must recover without its previous local filesystem. > - The existing admission rule accepts warm sessions. Managed Claude, Grok, and OpenCode sessions resolve to per-turn execution. > - Per-turn execution also needs durable provider-session bytes after the sandbox is retired. > - This pull request adds encrypted backup publication and removes the warm-only admission rule. > - Both lifecycle modes can use the existing native ownership and checkpoint contracts during a web deployment. ## Linked Issues or Issue Description Refs: #18518, #18529, #18826, #18835, #18841, #18864, #18887. **What existing behavior does this improve?** Native Runner execution on web servers with `PAPERCLIP_EPHEMERAL_SERVER=true`. **Current behavior** The admission rule requires a warm Runner lifecycle. Managed provider credentials can resolve an agent to per-turn execution, so enabling the ephemeral deployment profile blocks those turns. A database controller checkpoint does not preserve provider files after the old web home and per-turn sandbox are removed. **Proposed behavior** Admit native execution in either resolved lifecycle. Keep provider and credential choices unchanged. Publish encrypted provider state to shared storage before sandbox cleanup. Restore it from a fresh web home for the next turn. Use the existing controller lease and generation checks for active takeover. **Reason and benefit** Operators can remove the web server volume without excluding per-turn providers or forcing managed accounts into warm sessions. New native Daytona profiles automatically enter both rolling-deployment qualification lanes. **Breaking changes** The additive migration creates a backup publication ledger. Existing volume-backed local-disk installations retain their local backup path. S3 installations publish encrypted backups and require a stable secrets master key. Existing-volume enrollment must import surviving session data before detaching the volume. ## What Changed - Admit native Runner execution independently of its warm or per-turn lifecycle. Keep host-owned legacy execution excluded from ephemeral deployments. - Store private provider state as encrypted objects. Journal uploads before I/O, read them back, and fence publication against the current native owner. Keep the previous published backup when storage or ownership checks fail. - Restore exact provider bytes into a private cache. Require an exact durable backup before sandbox cleanup. Extend the existing bounded retention pass to retire stored objects without adding idle polling. - Import selected legacy controller and provider state in the existing offline maintenance process. Preserve queued work, task pointers, pending recovery, and original files. Prefer canonical database checkpoints. - Preserve already-failed legacy controllers with missing original artifact proof as encrypted archives. These archives do not become execution authority. Successful retained sessions still require proof. Report archival and executable imports separately. - Verify unmanaged Grok API keys through the existing xAI account validator. The live test exposed a readiness gap that kept remote Grok agents paused after platform checks. Preserve failed credentials and the separate runtime warning. - Ingest Grok whole-prompt usage at the owning ACP boundary. Bind the receipt to the exact outbound request/session; reject replay, conflicts, and incomplete authority. Separate turn cost from cumulative session cost in the sidecar schema and Rust normalization. Preserve managed billing identity and identify direct xAI API credentials. - Preserve published workspace directory names across cold allocations, including destroyed-allocation cleanup. Keep unpublished epoch/lease directories isolated and refuse to overwrite an occupied preparation. This fixes the cold-session identity regression exposed by #18922. - Increase bounded checkpoint-header inspection to 1 MiB. Retain the aggregate inspection limit. - Register all nine native Daytona profiles in per-turn and warm rolling-deployment suites. Delete the entire departing server home, exercise ordinary reads and writes, require active ownership transfer, then check a later turn on another fresh server. Permit only exact notification replay after reconnect. Ownership stays with the existing modules: Runner validates its protocol state; Core validates run scope and controller ownership; storage owns object I/O; the maintenance process owns offline conversion; the E2E wrapper owns process replacement and probes. No provider protocol is added to the shared admission rule. Grok receipt parsing stays in its provider module. The shared sidecar gains an explicit cost basis because its existing price field means session-cumulative spend, while Grok reports prompt totals. Existing consumers retain cumulative behavior when the new field is absent. ## Verification - Focused native admission, archive validation, publication fencing, cleanup, readiness, legacy import, and E2E observer tests pass. Real PostgreSQL cases verify that queued intent and historical recovery rows stay unchanged. - Full typecheck and build pass on the rebased branch. The review follow-up passes server typecheck, 29 focused PostgreSQL/archive/readiness tests, and module-boundary checks. Revision fb8d39a7a1 reached 55 successful CI checks and Greptile 5/5 with no unresolved threads. The Grok follow-up passes Runner/server typechecks, 238 ACP runtime/projection/usage tests, 16 sidecar-schema tests, the Rust cost regression, and Core billing regression. The |
||
|
Devin FoleyandPaperclip
|
1a39dea9aa |
fix(heartbeat): keep a routine claim queued while its slot is held (#18933)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A routine creates one task per tick. Only one open task of a routine can hold an execution lock (`issues_open_routine_execution_uq`). > - Since #18879, a failed native run that waits for a scheduled resume keeps its task's lock. Routine dispatch counts only queued, running or scheduled runs as live, so it can queue a newer task of the same routine meanwhile. > - When that newer run is claimed, stamping its execution lock violates the unique index. The raw database error is not a deferrable claim rejection, so it escapes the queue loop. The queue code notes that this can stop the agent's queue and startup recovery. > - This pull request turns that exact violation into a 409 claim rejection, so the run stays queued until the slot is free. > - The benefit is that a busy routine slot delays one run instead of stopping the queue or startup. ## Linked Issues or Issue Description Refs #18879 (keeps a failed run's lock while its resume is pending), #18916 (defers the same unique-index conflict in restart recovery's lock reclaim) and #18931 (open; keeps one run's restart-recovery failure from stopping startup recovery). This is the same index conflict on a different path: the heartbeat queue claim, not restart recovery. It does not change #18916 or #18931. Its doc paragraph is separate from the one #18931 edits. **What happened?** While auditing the lock paths from #18879 for this index, I found that a queued run of a newer routine task can be claimed while an older task of the same routine still holds the slot through a failed run with a pending resume. The claim's lock stamp then raises `duplicate key value violates unique constraint "issues_open_routine_execution_uq"`. `isDeferrableClaimRejection` accepts only `HttpError` 4xx, so the error escapes `startNextQueuedRunForAgent`. **Expected behavior** The claim is rejected as a conflict and rolls back. The run stays queued and is claimed after the older task releases the slot. Other errors behave as before. **Steps to reproduce** 1. Create two open tasks of the same routine (same origin and fingerprint). 2. Give the older task an execution lock held by a failed run. 3. Queue a run for the newer task and claim it. On `master` the claim throws the unique violation. The new queue test runs these steps against PostgreSQL. **Paperclip version or commit** `master` at `0dedbf23c35`. **Deployment mode** Any server with routines whose tasks run on native runners. ## What Changed - `bindClaimedIssueExecution` in `server/src/services/heartbeat/queue.ts`: if stamping the execution lock violates `issues_open_routine_execution_uq`, it throws `conflict("Another execution for this routine is already in progress")`. The claim transaction rolls back, and the queue loop keeps the run queued as it does for other 4xx claim rejections. Every other error is rethrown unchanged. The issue checkout route already maps this violation to the same 409. - `doc/execution-semantics.md`: one paragraph on the behavior. ## Verification - Failing first: `queue.test.ts` "keeps a claim queued while another task of its routine holds the execution slot". On `master` the claim raised the `issues_open_routine_execution_uq` violation. Now it rejects with 409, the run stays `queued`, neither task's lock changes, and nothing executes. - Failing first, review round 1: `heartbeat-queued-run-claim-isolation.test.ts` "keeps a routine run queued while another task holds the routine slot, then runs it once the slot is free" goes through `resumeQueuedRuns`. On `master` it rejected with the unique violation. Now the routine run stays queued with no error code while another queued run for the same agent executes. After the slot is released, the routine run executes exactly once (executions are counted per run, because the finished task can queue its own follow-up run). - `cd server && npx vitest run src/services/heartbeat/queue.test.ts src/__tests__/heartbeat-queued-run-claim-isolation.test.ts`: 70 passed. - `npx vitest run src/services/heartbeat/recovery.test.ts src/__tests__/routines*.test.ts`: 4 files, 205 passed. - `pnpm --filter @paperclipai/server typecheck`: pass. - Not verified: the queue loop end to end on a real instance. The loop's handling of a 4xx claim rejection is existing behavior. ## Risks - While the older task holds the slot, the newer run stays queued and the loop logs "queued heartbeat run claim was rejected" on each pass. It is claimed once the older task's resume finishes or its lock is released. - Only this one constraint is mapped. Other database errors still escape as before. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
652cd76026 |
fix(ui): expose Stop for failed native retries (#18899)
## Thinking Path > - Paperclip manages agents and their tasks. > - A native run can fail while it still owns a same-run retry. > - The server supports Stop for that pending execution. > - Run details show Cancel only for queued or running records. > - A failed pending retry therefore has no ordinary cleanup control. > - This pull request exposes the existing Stop eligibility and uses the existing cancellation endpoint. ## Linked Issues or Issue Description Refs #14743 and #14733 for correlated native Stop. Those changes cover backend cancellation and protocol qualification. They do not add the failed-run control to run details. Searches of current master and open PRs found no matching UI repair. **What happened?** A native run can show failed while its coordinator still owns a retryable failure with no accepted result. The backend can cancel that retry. Both run-detail implementations hide the cancellation button because the record is terminal. **Expected behavior** The selected run offers Stop when the server confirms that pending native retry. Unknown, legacy, terminal and accepted-result states do not gain this control. Existing queued and running cancellation stays available. **Steps to reproduce** Render either run-detail implementation for a failed native run with an eligible retry. The original UI has no Stop button. The original batched execution projection has no eligibility field. The regression tests exercise both real components and the actual PostgreSQL projection query. **Paperclip version or commit** The missing control and projection field were reproduced on |
||
|
Devin FoleyandPaperclip
|
6c7b136b16 |
fix(native): keep startup recovery running when one run's claim fails (#18931)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - At startup, the server runs native restart recovery before it reports ready. Recovery claims each interrupted or retryable native run in its own transaction. > - On one instance, one run's claim hit the routine execution unique index. The error escaped restart recovery, startup recovery failed closed, health stayed "still starting", and two deploys rolled back. > - #18916 fixed that exact conflict: it defers the run when its claim hits `issues_open_routine_execution_uq`. Any other error in one run's claim still aborts the whole pass, and so does a failed per-run audit write. > - This pull request handles every per-run claim failure the way #18916 handles the routine conflict, and keeps a failed audit write from stopping committed claims. > - The benefit is that one bad row cannot keep the server from becoming ready, and no run is claimed or changed when its claim fails. ## Linked Issues or Issue Description Refs #18916. This change generalizes its catch: #18916 keeps the routine-conflict disposition (`routine_execution_busy`) exactly as it is, and this change adds a separate reason for any other per-run failure. It does not change how the routine conflict is detected or deferred. Refs #18879, whose lock reclaim exposed the startup failure. Refs #18877 (open), which edits other parts of the same files. This change touches only the claim's error path and the audit loop, so a second merge needs at most a small rebase. **What happened?** Two deploys never became healthy. About 25 seconds after the listener bound, the boot log showed `startup native runner restart recovery failed closed` and `startup heartbeat recovery failed`, both from a failed `update "issues" set "execution_run_id" = ...` that violated `issues_open_routine_execution_uq`. #18916 now defers that violation. Any other error in one run's claim (another constraint, a trigger, a timeout) still escapes `claimNativeRestartRecoveries` and fails startup the same way. #18916's own test asserts this: "Other database failures still fail the recovery pass." **Expected behavior** One run's recovery failure is logged and recorded on that run, and the rest of startup recovery continues. The server becomes ready. Only a failure of the candidate scan itself fails the pass. **Steps to reproduce** 1. Create a native run that is due for restart recovery, and make its claim fail (the new test uses a trigger on `native_run_finalizations`). 2. Add a healthy run that is due for restart recovery. 3. Run restart recovery for both. On `master` the batch throws and the healthy run is not claimed. **Paperclip version or commit** `master` at `4a83ba4050f`. **Deployment mode** Any server that runs startup native restart recovery. ## What Changed - `claimNativeRestartRecoveries`: #18916's `.catch` on each candidate's claim transaction no longer rethrows other errors. The whole claim has rolled back, so the run's state is unchanged and nothing is claimed. The run gets `awaiting_evidence` with the same 30-second retry deadline. The reason stays `routine_execution_busy` for the routine slot conflict, and is the new `restart_recovery_failed` for any other error, which is logged at error level. The batch continues. - `recoverNativeRunsAfterRestart`: records a `restart_recovery_failed` deferral as a warning transition on the run ("Native restart recovery failed for this run and will retry"). A failed `getRun` or transition write is logged and no longer stops the dispatch of claims that already committed. - `doc/execution-semantics.md`: replaces "Other database failures still fail the recovery pass" with the per-run rule. ## Verification - Failing first: `native-restart-recovery-isolation.test.ts` (new, PostgreSQL). It seeds the reported shape (two open tasks of one routine: the older with a due scheduled resume and a cleared lock, the newer holding the lock), a run whose claim fails through a trigger, and a healthy run. On `master` the batch threw `injected recovery claim failure`. Now the routine run is deferred with `routine_execution_busy` and the failing run with `restart_recovery_failed`, and both keep their coordinator, attempt and locks. The healthy run is claimed. `recoverNativeRunsAfterRestart` completes and records an info transition for the routine run and a warning transition for the failing run. - Two existing tests expected the pass to reject on one run's failure: #18916's unrelated-constraint test, and failed-workspace retention's rollback test. They now expect the per-run deferral. Their rollback checks (no claim, unchanged coordinator, lease, lock and recovery actions) are unchanged, and the unrelated failure still does not become a routine wait. - `cd server && npx vitest run` over restart recovery (isolation, integration, diskless, preparation), controller recovery, heartbeat recovery, scheduled resume lock, failed-workspace recovery, startup cleanup, native resumption, live recovery work, startup feedback export, recovery deadlines, runtime selection, stop accounting, warm transition restart, workspace reconstruction, remote workspace migration and heartbeat process recovery: all pass except one. `heartbeat-process-recovery` "redacts opaque environment-bound credentials from Sentry diagnostics: result" fails the same way on unmodified `master` on this host (a local environment value is redacted). - Review round 1: two `heartbeat/recovery.test.ts` cases make `getRun` and then `appendRunEvent` fail for a deferred run while a committed claim is in the same batch. Recovery completes and dispatches the claim exactly once; the `appendRunEvent` case also checks the warning transition. Both fail when the catches are removed. In one of four full runs of that file, the unrelated orphan-cleanup case "reconstructs missing native preservation during orphan cleanup without replay: reuse_by_environment" failed; it passed alone and in the other three full runs. - After the rebase onto `4a83ba4050f`: the isolation and failed-workspace recovery tests and `pnpm --filter @paperclipai/server typecheck` pass. - Not verified: a deploy to a real instance. ## Risks - A persistent per-run error now retries every 30 seconds and logs an error each time, instead of stopping startup. The run stays unclaimed, so ownership does not change. - If the database fails for every candidate's claim, startup recovery now completes with every run deferred, instead of failing closed. The candidate scan still fails closed, and the deferred runs retry on their deadlines. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0dedbf23c3 |
fix(chat): retain receiver ownership during runtime construction (#18930)
## Thinking Path > - Paperclip manages agents and their work. > - Chat receivers hold a durable lease before they accept messages. > - Runtime construction can outlast a lease renewal interval. > - Renewal previously required an attached runtime before construction had returned. > - That check could revoke the same receiver that was still starting. > - This change uses the exact pending initialization as proof during construction. > - The existing durable token and endpoint checks still fence every renewal. ## Linked Issues or Issue Description Related: #18899 exposed the existing receiver failure in its chat CI shard. This is a separate lifecycle repair. #18795 provides the relevant production and sleep fixture context. Searches found no overlapping open initialization repair. **What happened?** The receiver acquired its lease and started its renewal timer. The service then awaited credentials, command preparation and runtime construction before attaching the runtime to its context. A renewal during that interval treated the missing attachment as lost ownership. Configuration later failed with a Gateway ownership error, although its original durable token was still current. **Expected behavior** The exact pending initialization can retain its own lease while construction runs. Missing attachment alone must grant no ownership. A changed token, paused or removed endpoint, changed generation or credentials, and shutdown must still fence the receiver. **Steps to reproduce** 1. Use the real configure and PostgreSQL ownership producers with a 60-second lease and 50-millisecond renewal. 2. Hold the actual runtime construction call for 200 milliseconds. 3. Release construction. Untouched code returns the Gateway ownership error. 4. With the same construction delay and a five-second renewal control, configuration succeeds. **Paperclip version or commit** The causal regression fails on unchanged production source at `e7549ab613`. The original CI error was observed in the chat shard of #18899. That CI run did not record the precise setup and renewal order, so this local proof does not establish its historical scheduling. ## What Changed - Store the runtime context identity in the existing pending initialization entry. - Allow pre-attachment renewal only for that exact tracked initialization and current local ownership. - Recheck local ownership inside the existing renewal transaction. Preserve company, endpoint, generation, credential and original-token checks. - Before SDK listener initialization, wait for any older renewal, recheck local ownership, then obtain fresh durable proof. Preserve the existing checks after initialization. - Keep attached runtime identity checks, lease deadlines, renewal intervals, shutdown and listener retirement unchanged. - Add actual configure and database race controls. Document the shared Discord and Photon receiver contract. ## Verification - Untouched production red: the actual construction gate crosses a renewal and returns the same Gateway ownership error. The otherwise identical slower-renewal control passes. No lease, exception or provider outcome is injected in these positive cases. - A subsequent review found a second real race: final runtime removal did not prove that SDK initialization had never started. Four actual initialization-hook controls failed before the pre-initialize fence. The final matrix asserts zero hook calls for lost or changed authority, including a foreign-token control with no periodic renewal due. An attempted cross-company endpoint move is rejected by the existing database foreign key; the valid original initialization still succeeds. - A third actual-producer red holds completion of a real committed renewal, changes the durable token, and proves that joining that old promise could incorrectly accept configuration. The corrected startup check waits for that renewal and then obtains fresh proof; it cannot credit the earlier result alone. - Final relevant full Discord registration group, takeover/callback cases, buffered host-pause recovery, failed construction, credential-fenced pause and shared Photon consumers: 34 passed across two files; 1,840 other cases were deselected. Four executed cases cover Photon transport recovery, failed authentication, duplicate allocation ownership and checkpoint fencing. The tests use the real configure/database producers with fake provider transports and observe SDK initialization; they do not prove physical network behavior. - All server type layers, the module-boundary gate, and production server/vendored Runner build pass at exact final `38cf82ec114bd12c0aa7bbc125ed72b1ae941b4e`. The production build stamp records this exact commit. - Earlier fixture mistakes are preserved separately: a test initially changed every lease key, including the credential lease, and another attempted an invalid cross-company row move. Neither is counted as a production behavior red. The corrected controls target the Gateway key and assert the existing company constraint. - An initial isolated dependency graph resolved an older shared telemetry build. Correct workspace links restored the all-layer typecheck without changing telemetry source. - Full repository checks were not run locally for this narrow slice. Fresh current-head CI and substantive reviews remain required. No hosted task, provider work, fleet action or deployment was performed. ## Risks The ownership helper is shared by Discord and Photon. The selected tests cover both consumers. The pending entry grants only permission to renew the original token; it grants no message, callback or publication authority and adopts no listener. Missing or changed initialization identity fails closed. There is no schema change, historical backfill, new worker or additional idle scan. ## Model Used OpenAI `gpt-6.1-sol` in Codex, with reasoning, repository inspection, source editing, shell execution and test tools. The host did not report the context-window size. A separate coordinator reviews the source before delivery. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
4a83ba4050 |
fix(workspaces): isolate unpublished preparation directories (#18922)
Bind new remote preparation directories to the writer epoch and environment lease. Preserve earlier unpublished files and the recorded path for published warm workspaces. Run the interrupted reused-allocation and saved-file lifecycle regressions with owned Postgres by default, and retain the warm-path recovery checks. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f211d44c00 |
fix(heartbeat): preserve queued coalesced comments at claim (#18913)
## Thinking Path > - Paperclip manages agents and their work. > - The wake queue records task comments before an agent starts. > - A queued run can accept another comment through coalescing. > - Claim rebuilds that run's input from its primary wake. > - Coalescing updated the run and its receipt but left the primary batch unchanged. > - This change saves the accepted batch atomically in the owning writer. > - The benefit is complete input at claim without repeating old work. ## Linked Issues or Issue Description Related: #18887 preserves offline queued turns. #13948 addresses equivalent automatic wake deduplication. This change repairs the accepted comment batch at the existing claim boundary. **What happened?** A queued run started with comment A. Admission accepted comment B, saved `[A, B]` in the run context, and finished B's coalesced receipt. The primary wake still held `[A]`. Actual claim rebuilt input from that wake and dropped B. **Expected behavior** Claim must retain every accepted live comment in order. Stop, changed source authority, deleted comments, and company boundaries must still apply. Already claimed wakes must retain their existing recovery source. **Steps to reproduce** 1. Queue a task run whose primary wake has an authoritative comment batch `[A]`. 2. Admit comment B through `createHeartbeatQueue.enqueueWakeup` and the real Postgres admission writer. 3. Confirm that admission accepts `[A, B]` and records B's coalesced receipt. 4. Claim the original queued run. Before this fix, the claimed input contains only A. **Paperclip version or commit** The untouched producer regression fails on `d7283e875772ca8c684dc86e471e6b48d543b7b1` with both process and native Runner/Codex adapter selection. Provider execution is disabled in the fixture. ## What Changed - Save the accepted comment batch in the unclaimed primary wake, run context, and coalesced receipt in one transaction. - Lock the primary wake before the run. Recheck the expected run status and same-company, same-agent and run bindings; apply the existing actor equality fence only to durable receipts before mutation. - Preserve incoming input as normal deferred work when the locked source changed. Preserve receipt identity and actor fields. - Preserve the original primary actor and trigger comment while updating its accepted batch. Each incoming receipt retains its actual actor. Keep claimed source payloads and legacy envelopes on their existing recovery paths. - Safely defer a same-named different agent without rewriting the original agent's run, wake or receipt. Missing and foreign-company targets remain rejected. Do not rebuild input from historical receipts. - Add actual process/native producer, real database rollback, Stop, deletion, foreign-source, and fallback controls. Update three direct writer fixtures for the required expected-status contract. ## Verification - The untouched process and native producer red on `d7283e875772ca8c684dc86e471e6b48d543b7b1` loses accepted comment B at actual claim. The initial incomplete mock is retained separately and excluded from behavioral proof. - Current process/native matrix: 22 passed, including mixed authors, system primary, same-named different agents, changed status, claimed sources, Stop, deletion and foreign-source boundaries. Queue, Postgres admission and use-case suites passed 234 cases before the source-preserving serializer follow-up; its 22 actual producer cases passed again afterward. - CI on the previous head found a real ordinary mixed-author workspace-copyback regression. The earlier 13-case selection omitted this test; that coverage gap is preserved. The first correction exposed a second real source-provenance/copyback failure. After preserving the primary trigger and actor while updating only its batch, the complete workspace recovery file passed all 188 cases, including ordinary mixed-author input and native copyback. - Real HTTP queued-comment consumption, discard/reorder and Stop boundaries: 101 passed before these review corrections. Existing chat and Slack consumers each passed their selected real writer case. - Previous-head CI also timed out in the CLI child-server startup fixture and a ZIP transfer case. The full untouched transfer file passed 17 cases locally. Startup remains unclassified; its intentionally reserved port is part of the fixture and is not diagnosed as the cause. No CI retry was issued. - An earlier queue run passed its 231 assertions but timed out during teardown; its preserved replay passed all 60 queue cases. This is distinct from the behavioral reds. - Exact `ab59d944895774d89f950248e356e93af00f4832` server type layers and production server/Runner build passed. The unchanged-default serializer file passed all 11 cases. Current-head CI and both substantive reviews remain required; previous grades and checks do not qualify this head. - No hosted task was retried or changed. Local tests do not establish any hosted deferred event's cause, provider execution, native acknowledgement, physical cleanup, or serving revision. ## Risks - Admission now performs bounded row reads and locks for each coalesced wake. It adds no idle scan or remote request. The existing task transaction owns the operation; lock order follows queued claim. - Claimed-wake recovery uses source identity. Its payload remains unchanged, and the existing actual native copyback tests cover that boundary. - A source that changes before the lock can defer the new input. This preserves work instead of consuming it into a stale run. No migration or historical backfill is required. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, shell execution, and test tools. The runtime does not report the context-window size. A separate OpenAI coordinator reviewed the source. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e7549ab613 |
fix(runtime): defer native routine recovery collisions (#18916)
Treat the exact routine execution unique conflict as a pending recovery obligation after the complete losing transaction rolls back. Preserve the current routine owner, provider attempt and checkpoint, and continue the startup batch. Cover real constraint failures, row preservation, one reclaim after release and unrelated database failures. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0884e73b46 |
test(e2e): verify saved assigned skill use (#18783)
## Thinking Path > - Paperclip lets people manage agent work and reusable company skills. > - A saved assigned skill must affect a task's actual execution. > - Existing QA coverage creates and edits skills. > - The daily target needs coverage for ordinary saved-skill use. > - The task receives its receipt data only through one saved skill. > - The admitted immutable version, native read and saved document establish the outcome. > - This PR adds one bounded journey for that gap. > - A shared lease test also needs a stable clock to assert its exact timeout. ## Linked Issues or Issue Description Refs: #18305. Supporting repair #18885 merged as `057898530c702ec14c1e14d163e96dfb045cd8c0`. It closes the recovery-conflict prerequisite. This branch does not duplicate that production patch. **What existing behavior does this improve?** Product E2E coverage for ordinary assigned company skill use on an existing deployment target. **Current behavior** The deployment journey catalog can create and edit a company skill. It has no separate case that requires a native task to load its assigned saved skill and use data available only inside that source. **Proposed behavior** Create one synthetic managed company skill through the public API. Assign its canonical key with the normal Latest selection. Create the task through the production browser composer. Require one native work run, the exact created immutable version in its admitted skill-version map, a successful native source read or load, and an agent-authored saved receipt. Reload and verify the result, revision and original source. ## What Changed - Add `saved-skill-use` with one native work run and seven required checks. The first check is `saved-skill-assignment-bound`. - Use ordinary `desiredSkills: [canonicalKey]` assignment. Require the returned stored assignment to have `versionId: null`. This works with Beta skills off and reads or changes no experimental setting. - Bind the creation receipt, pre-dispatch source read and version file inventory. Require a managed skill with the exact company key. Require the run's authoritative `skillSelectionSnapshot: true` and admitted version map to match the created immutable version. - Keep the random receipt reference and numeric inputs only in the saved source. The task prompt names the skill without supplying its answer. - Require exact company, task, agent, run, turn and runner bindings for a complete successful native source receipt. Accept the deployed OpenCode read wrapper only when both the saved assignment and native execution input bind its runtime directory to the created immutable skill version. Also accept the first-party assigned-skill read with a provider-independent canonical key/version binding from both the saved API assignment and admitted native input. Require the exact full source, checksum, byte count and complete page. Keep the materialized directory-name guard specific to the OpenCode filesystem path. Reject claims, shell output, foreign or incomplete reads and wrong skill loads. - Add a narrow Codex reader for the retained controller `semantic_tool.input` and `semantic_tool.result` pair. Require the actual native provider, runner, session, call, turn and item; the admitted immutable key/version; the read-only `read_assigned_skill` operation; and its successful active-task receipt. Recompute the input digest and the exact complete seven-field read-result digest from the saved source. Raw dynamic or MCP names grant no authority. Keep the builtin OpenCode/filesystem reader body unchanged. - Require authoritative native commit before qualifying the saved receipt. Independently check its value and same-run document revision authorship. Reload and verify the result and unchanged source. Retain those existing readback receipts in the private checkpoint for independent verification. - Keep the exact raw campaign ID in the journal, API name, frontmatter and saved name. A lost successful creation response remains recoverable by name. - Register discovery, source fingerprints, closed checks and existing feature associations. Update the case and fixture docs. Feature-map files are unchanged. - Stabilize one existing shared sandbox lease regression with a scoped `Date.now` spy and `finally` restoration. Keep its exact timeout, no-destroy, no-replacement and recorded lease assertions. Change no production service code. ## Verification - **Genuine QA staging qualification passed on this exact `2fb654f7be0bac3bfeb4ec6b7a2eb7a06b8de7e4` head**: [workflow41763421118](https://github.com/paperclipai/paperclip-cloud/actions/runs/41763421118), [TestOps launch2257/result4034](https://paperclip.testops.cloud/launch/2257/tree/4034?treeId=0), [sanitized report](https://pages.paperclip.ing/evals-qa-smoke-41763421118-1/). All seven required checks and all five phases passed, with exactly one configured OpenCode/DeepSeek native run succeeded and committed. Owned cleanup closed. A separate source-pinned replay verified25 original-evidence controls and16 rejection controls without altering originals. Automatic #testops PASS was independently read back. Both earlier failed QA grades remain unchanged. - Current-head CI `41758727042` attempt 2 passed with 48 successful jobs and one skip. The coordinator authorized one unchanged-source failed-jobs diagnostic. Start and end timestamps prove three fresh executions (Runner package, server shard 7 and aggregate verification), 45 retained successful executions and one skipped non-execution. The prepared-context test passed in 2,534 ms. All 28 repository rehome tests passed. The original failed attempt remains sealed. Its timeout causes are still unknown. No source, timeout or assertion changed, and no reliability fix is claimed. - Fresh exact-head Greptile review is 5/5 with zero unresolved threads. All 58 current check-runs and the status rollup are terminal and clean. The native Paperclip review ended neutral and incomplete, so it is not a validated assessment. The PR remains mergeable after the dependency case merged in #18901. The source remains frozen. A coordinator-only qualification draft is prepared with approval false. A genuine QA staging pass is still required before daily enrollment. - Current head `2fb654f7be0bac3bfeb4ec6b7a2eb7a06b8de7e4`: 269 focused tests across eight files passed. These include 42 Codex producer, converter and false-positive controls. The canonical E2E typecheck passed with its own declaration prerequisites. Credential-free discovery lists 24 journeys. Fresh exact-head Greptile review passed at 5/5 with zero unresolved threads. CI `41758727042` attempt 1 failed. It completed 45 successful jobs, one skip and three failed jobs. The primary failures were one existing OpenCode prepared-context test at its unchanged 5-second timeout and three existing repository rehome tests at their unchanged 15-second timeouts. The third failed job aggregates those results. The timeout causes are unknown. The original attempt remains sealed. A direct passing twin ran the prepared-context test in 2,533 ms. All package, script, CLI, UI, workflow and root inputs match between the actual tested merges. Three server files differ, including `task-retry.ts` inside the broader rehome import graph; no full-server equality or causal fix is claimed. The later single diagnostic pass is recorded above. The native Paperclip review ended neutral with an incomplete execution, so it is not a validated assessment. No live QA pass is claimed. - The valid review on `e1a474` found that Codex displays its first-party tool as dynamic. Source inspection confirmed that its raw name has no authority. The retained controller semantic input/result pair supplies the real operation authority. Core `09162fe893b464dd2ccdafd14bd3ef1611202a86` and current source have identical relevant envelope, digest, result-unwrapping and immutable-read implementation regions. - Three real-producer positive controls failed against the old grader. All 42 now pass. The tests call the actual immutable-file reader, production protocol digest builders, Codex display converter and API redaction. They validate the event schema and cover omitted defaults, the exact runtime alias and Unicode bytes. Missing, duplicate, reordered, failed, foreign, MCP, unbound, partial-page, wrong-digest, wrong-key/version and nonce-only evidence fails. Generic dynamic output fails even with copied full text and invented read-only markers. - The original builtin reader body and source-only fixture function are byte-identical to `e1a474`. All seven checks, the one-run limit, 500-cent guard, ordinary Latest assignment and Beta-off behavior remain unchanged. The independent verifier is pinned to this new head. Its synthetic semantic receipt passes 24 negative controls and its filesystem receipt passes 16. Older helpers and both original failed QA seals remain unchanged. - Earlier integrated head `e1a47438b1c7e51a5bb2896a4ec9b6ed6560318b`: 376 focused checks across eleven files passed, including Saved, settings and Chat overlap controls. All seven canonical server type layers and exact-head E2E types passed. All four lease controls passed. Credential-free discovery lists 24 catalog journeys. No new staging run has occurred. - Normal integrations consume actual master `fbb7055b9020e505cd1bf903b1b1c301ba8f3eef` and actual Chat squash `d7283e875772ca8c684dc86e471e6b48d543b7b1`. Preserve all five Saved case/parser files byte-for-byte and retain settings and Chat registrations. The only manual resolutions were two additive handler/type anchors. All seven checks, one-run limit and 500-cent guard remain unchanged. The independent verifier is pinned to this exact head. - Earlier dbdf CI passed after one authorized failed-jobs recheck of a confirmed Spot-terminated Build runner. Timestamp comparison proves two fresh executions (Build and verify), 46 retained successful executions and one skip. The original failed attempt remains sealed. Its native Paperclip Review completed neutral with an incomplete execution and is not a validated assessment. - New master entries made the earlier dbdf PR conflict. The frozen qualification guard correctly blocked dispatch before any pin or receipt. Preserve its unapproved draft as history; it did not authorize a QA run. This normal integration resolves the observed conflict. Fresh current-head CI/review and one genuine staging pass remain required. - Earlier merged head `dbdf1f3cbeaacc7cf9a2a457618035c945bd09f0`: 227 focused checks passed across seven files. All seven canonical server type layers passed. The E2E typecheck passed using the declarations built in this same checkout on this exact head. All four existing clock and lease controls passed. Credential-free discovery lists 22 catalog journeys, including both saved-skill use and task lifecycle. These are local checks, not a live staging qualification. - Normal merge of master `8771606fd87f69ed4057779e3dce3a9bca8088e6` retains the reviewed task-lifecycle case and its Cursor fixture repair from #18782. Resolve six shared journey registrations by preserving both cases. The five saved-skill case files remain byte-for-byte equal to reviewed `ff3814`. All seven required checks, one-run limit, 500-cent bound and ordinary Beta-off Latest assignment remain unchanged. No feature-map delta was added. - The independent sealed-run verifier is now pinned to this exact merged head. Synthetic semantic evidence passed with 24 negative controls. Synthetic filesystem evidence passed with 16 negative controls. Foreign document authorship and missing reopen/source readback still fail. The old `ff3814` verifier and both original failed QA seals remain unchanged. - Common generated outputs were copied only after all 4,225 package, script and root inputs matched, installed lock/TypeScript hashes matched, and all 4,955 source output paths and hashes passed before and after copy. No server declarations or editable source/module links were copied. The server declarations were rebuilt locally. A full frozen workspace install disabled lifecycle scripts and used the same locked packages. No Rust build ran locally. - Previous `ff3814` CI `41733864320` exposed a real wrapped PostgreSQL uniqueness error in a recovery server test. That production defect is independently fixed and tested in #18885. The later supporting PR merged separately. This branch does not duplicate its production patch or regrade the original failure. Preserve the original failed result and the separate Spot interruption. New full CI and a fresh exact-head 5/5 review remain required. - Previously, **225 tests passed across six files** in the focused evidence, flow, contract, catalog and report set after the semantic-read correction. The prior 181-test proof and all original failed attempts remain retained. This correction changes only the saved-skill evidence helper, flow argument, their tests and case documentation. The shared clock repair remains unchanged. - E2E TypeScript check passed on Node **24.22.2**. - Earlier credential-free discovery listed 21 cases and exposes the ordinary saved-skill journey without a Beta prerequisite. These local checks make no provider or model calls. - Source inspection of Core `09162fe893b464dd2ccdafd14bd3ef1611202a86` confirms key-only selection becomes a stored null Latest selection. Managed execution resolves its current immutable version and records the admitted map in `contextSnapshot`. The public run API preserves that map independently of optional result presentation data. - Seven invalid admitted-version fixtures wrongly qualified without the new guard. The regressions now reject them before the read or document can qualify. Beta-off and Beta-on ordinary request controls use the production sync validator. Wrong assignments, missing null selection, source drift, creation-version mismatch and another skill's complete native output cannot pass. - Existing native commit, null/compact result data, read provenance, nonce, saved-result, reload and lost-creation-response controls remain required. Their original red results are retained. - The earlier experimental version-pin variant remains unqualified. Its missing-prerequisite regression and the denied target-setting attempt remain separate failure evidence. This case qualifies ordinary assignment and the version actually admitted by the server. - Normal merge of master `5f1b59ba2c9adfb40a681bf419a8b7e7837d1125` includes the reviewed inbox snapshot fix #18798 and App/Shell access-fixture fix #18799. The merge has no conflicts. All 17 journey source, helper, registration and documentation files remain byte-identical to reviewed head `eab260955ff43eeacfdafc9b75dfc9b116f0eb79`. - Previous CI `41672686614` failed two UI test shards with obsolete user-ID fixtures; its aggregate verify job also failed. The merged access-fixture repair addresses those failures. This is a new source CI run; the failed run is retained. - Dependency output reuse follows exact path and byte checks for 10,977 tracked build inputs and 3,411 generated outputs. The first copy guard detected 12 bundles regenerated after its initial manifest and stopped before any destination mutation. The stable replacement manifest passed all checks before copying. Final focused tests, E2E typecheck and discovery passed after the verified copy. No editable source or module link was shared. - An earlier integration found stale generated Cursor Cloud declarations. Canonical dependency builds refreshed them, then typecheck passed. The original typecheck failure remains recorded. - The initial report support failure came from a blocked `tsx` IPC pipe before report execution. The unchanged test passed with normal local IPC permission. The original failure and diagnostic remain recorded. - `git diff --check` passes. Full workspace recursive typecheck, unit run and build remain assigned to hosted CI and coordinator verification. Full CI `41679720041` passed on the prior `f3f288a54177f2b9a7b225b1d1ab35440dfc597c` source. A clean 5/5 review passed on `0f445cf1c622afb25a1af92398784209df542696`. Fresh CI and review are required after the shared clock fixture repair. - The first real QA staging attempt on `f3f288a54177f2b9a7b225b1d1ab35440dfc597c` failed `saved-skill-native-read-attributed`. Assignment, immutable admitted version and cleanup passed. The test stopped before document, reload and source-after observations. The original failed attempt and its complete evidence remain sealed. - The original successful builtin `read` event contained the full saved procedure. OpenCode returned its file wrapper with an unknown normalized operation, null target and a hashed runtime skill name. The old parser rejected that shape. A synthetic regression failed before this correction. The corrected parser recognizes that exact retained read in memory, with no change to the original grade. Wrong paths, copied shell text, missing starts, mismatched calls, failed/truncated reads and wrong version/name bindings still fail. A later staging attempt is described below. Neither failed original was regraded. - Original CI `41703535552` failed one shared server test and its dependent verification job. The lease-resume test expected an RPC timeout of 31,234 ms, but the runtime correctly supplied 31,233 ms after one elapsed millisecond. The test and runtime service were byte-identical to the previous passing source. The failed run remains recorded. - A scoped advancing clock reproduced that exact 31,233-versus-31,234 failure locally. A stable clock around only the acquisition call now preserves the exact original assertion. The repaired test and three related sandbox lifecycle controls pass on disposable embedded Postgres. No production timeout, lease preservation, cleanup or replacement behavior changed. - The first local reproduction stopped before test execution because six same-package native Postgres library aliases were missing. Their exact existing versioned libraries matched the canonical dependency. Restoring only those internal relative aliases allowed the separate deterministic red and green measurements. This prerequisite failure remains recorded. - The second real staging attempt on `02582aadd23fc128ae29daa69482113cf0f95ad1` also failed `saved-skill-native-read-attributed`. Authentication, setup, assignment, source-only input, collection and cleanup passed. It completed one native run. Later document, reload and source-after checks were not executed. The complete original remains sealed as failed. - That attempt used the ordinary first-party `paperclip_read_assigned_skill` tool. Its complete native receipt contained the exact canonical skill key, created version, `SKILL.md` bytes, SHA-256, byte count and null next-page offset. Core verifies the pinned manifest and file checksum before returning that receipt. The old grader recognized only filesystem and native skill-load output. An actual-shape regression failed before this correction. The corrected parser recognizes the retained read in memory. Wrong keys, versions, source bytes, hashes, partial pages, semantic failures, copied shell output and foreign or unpaired native calls still fail. No model prompt, product source or original grade changed. A live pass on this corrected source remains required. - CI `41708636059` on the previous `02582` source passed at attempt 3. The first two interrupted attempts remain recorded. AWS evidence identified Spot capacity termination for the second Rust runner. The targeted retry executed Rust and aggregate verification; 46 successful jobs retained their exact prior execution timestamps and one job remained skipped. New metadata IDs do not establish new executions. Fresh full CI and review remain required for this source. - Earlier CI `41730743279` passed on `1ee86838ea528edd8c60fb9ddd858faf3c83e783`. A later review finding correctly showed that the semantic read still depended on an OpenCode-only directory-name helper. That finding remains recorded. Three local Codex, ACPX Cursor and Cursor Cloud flow controls reproduced the rejection before this correction. - The semantic path now binds the canonical key and immutable version through the common native runtime skill selection contract. Its checksum, complete source, page and native call attribution checks stay unchanged. Missing, duplicate, wrong-key and foreign-version API/native assignment controls still fail for all three provider configurations. The OpenCode filesystem wrapper still requires its exact provider and materialized directory name. These local controls test the existing canonical receipt shape; they do not claim live provider qualification or add a guessed provider encoding. Fresh CI and review are required after this correction. - This public PR does not enroll the case in the daily plan or change protected QA configuration. ## Risks The target must supply a disposable native agent with normal company skill-write access and cleanup. It must preserve the authoritative admitted version map and provide a complete source read or named native skill-load receipt. Missing evidence fails the case even when the receipt document has the right value. The OpenCode wrapper path requires matching assignment and native input records for the same immutable version; it does not accept a generic targetless read. The builtin first-party path requires the same version binding and exact full-source receipt. Codex must provide its authoritative controller pair and a digest that commits to that exact complete result. Partial pages, unknown operations and semantic failures do not qualify. Local flow controls simulate the target API and browser boundaries. They do not establish live Cloud qualification. Experimental assignment pins need separate qualification. The journey changes only its owned skill, assignment and task. It changes no shared agent, setting, model, connection or budget. ## Model Used OpenAI GPT-6 through Codex with reasoning and code execution. The runtime does not expose a more specific model identifier or context-window size. Local checks made no product-model calls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ce7766dc11 |
fix(recovery): resume retries with remote workspace storage (#18910)
## Thinking Path > - Paperclip manages work performed by AI agents. > - Native recovery must resume an interrupted run under its saved ownership. > - Remote workspace storage records that ownership in `runnerProfileJson.remoteWorkspace`. > - Recovery recognized only the older workspace record and plugin task leases. > - A retained remote PID could therefore leave a due retry without a dispatch. > - This PR routes the new record through the existing remote recovery verifier. > - Checkpoint, lease, process and single-executor checks remain authoritative. ## Linked Issues or Issue Description **What happened?** After a native sandbox turn returned without a semantic result, same-run recovery was saved with a retry deadline. The run retained its sandbox PID and the new remote workspace reference. The recovery selector skipped it. Later task attempts could then fail on the unfinished workspace writer. **What did you expect to happen?** The due retry must enter existing remote checkpoint and ownership verification. Recovery must resume the original run once or expose the real verification blocker. **Steps to reproduce** Persist a retryable native failure with a remote PID, a prepared `paperclip.remote-workspace-run.v1` reference, an existing lease and checkpoint, then run heartbeat recovery. The added PostgreSQL cases reproduce this shape. **Deployment mode** Native Paperclip Runner in a sandbox with remote workspace storage. Related: #18841 handles legacy workspace recovery during storage migration. #18908 handles failed native preparation. This repair covers a later, provider-started retry. ## What Changed - Recognize the current remote workspace record when classifying process location and selecting remote retries. - Extend the existing database and real-process fixtures to cover the new record, including a retained PID without a process group. - Reuse existing restart recovery. Add no new retry loop, ownership bypass, migration or workflow change. ## Verification - Before the repair: three new behavioral cases failed. Two retries never dispatched; a missing checkpoint never reached its visible recovery hold. - After the repair: all 82 heartbeat recovery tests pass. - All 57 neighboring resumption-binding, restart-recovery and remote-probe tests pass. - Safety cases retain retry deadlines, live controller ownership, active execution, missing-checkpoint rejection, single dispatch and zero host signaling of the sandbox PID. - Local platform: macOS arm64, Node 24.22.2, PostgreSQL 18.1. The tests use disposable databases and an owned live process as the remote PID collision control. Provider execution is outside this routing regression. - Full `pnpm -r typecheck` and `pnpm build` pass at `34aa2d08d38e8afec7f8d98ce984a74551caeda6`. - Current-head GitHub checks: 53 successful; `ci / verify` passed. Paperclip Review completed neutral with a 5/5 assessment and no findings. Two Storybook jobs were skipped by scope; the external security scan was neutral. No failures, pending checks, or merge conflicts remain. - Greptile: 5/5 at the same commit, with no review threads. - The duplicate full local `pnpm test:run` was interrupted after current-head CI passed. Its first two chat shards passed (574 cases); it has no complete local full-suite verdict. This is separate from the passing targeted recovery suites and required CI test lanes. - Staging diagnosis used read-only run, finalization, workspace and lease evidence. The fixed revision has not been deployed. The original task's useful continuation remains unverified. - Initial local attempts stopped before test execution because generated SDK output and PostgreSQL library aliases were missing. Those setup failures are retained separately from the behavioral red/green evidence. ## Risks A wrong remote classification could send a run to the wrong ownership path. The existing verifier still checks the saved checkpoint, controller lease and target. Tests require one dispatch, preserve process identifiers, and prove that recovery never inspects or signals a coincident host PID. Legacy workspace and plugin task coverage remains green. This repairs missed continuation; it does not establish the cause of the earlier missing semantic result. ## Model Used OpenAI GPT-6 through Codex, with reasoning and tool execution. The exact model snapshot and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
27a825da2f |
feat(telemetry): track the server OS with the registered contract (#18897)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its first-party telemetry uses a generated event contract and typed helpers. > - The existing events do not identify the server's OS family. > - A proposed event is dropped until the generated contract registers it. > - This PR syncs the registered contract and emits the OS family when an enabled server telemetry client starts. > - The benefit is a way to compare Windows, Linux, and macOS installations while keeping machine details out of the event. ## Linked Issues or Issue Description **Problem or motivation** We cannot compare Paperclip installations by host OS. Counting task or install events does not provide this dimension. **Proposed solution** Send one `server.started` event with Node's raw `process.platform` as `os_platform` each time the server initializes its enabled telemetry client. Compare distinct install identifiers per platform, rather than restart event totals. **Alternatives considered** A proposal-only emitter would send no data. A separate later contract-sync PR would leave the feature incomplete. This PR includes the canonical generated SDK and promotes the emitter together. CPU architecture and OS versions are outside this feature's scope. **Roadmap alignment** This is a narrow telemetry addition to existing server startup, with no new user flow. The full contract copy also adopts the three existing connector proposals described in #13578; their schemas are already registered. The generated file stays an exact copy, so their helpers and tests must be promoted in the same change. ## What Changed - Export `trackServerStarted` and call it once when server telemetry initializes. Repeated initialization returns the same client and emits no duplicate event. - Copy the canonical generated telemetry SDK byte for byte from schema source commit `5369e6b`. Remove the startup proposal marker. Normal opt-out and CI suppression apply before initialization. - Promote the three connector events included in the canonical SDK. Derive enum types from the generated map and preserve raw values. Update their real-client transport and disabled-state tests. - Assign startup to the existing `operational_enum_count` retention class. Document OS privacy, container/WSL interpretation, and distinct-install counting. - Add real-client startup transport tests and server initialization tests, including file opt-out, environment opt-out, and CI suppression. - Keep startup running when telemetry state storage fails. Publish the client and start periodic flushing only after the startup event succeeds. On failure, clean up the client, disable telemetry for the process, and log a generic warning without the storage path or underlying error. ## Verification - `pnpm --filter @paperclipai/shared typecheck`: passed. - `pnpm --filter @paperclipai/shared build`: passed. - `pnpm --filter @paperclipai/shared exec vitest run`: 98 files, 1,055 tests passed. - `pnpm --filter @paperclipai/shared exec vitest run src/telemetry`: 6 files, 39 tests passed. - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/telemetry-startup.test.ts src/__tests__/shared-telemetry-events.test.ts`: 2 files, 19 tests passed. - Exact generated artifact comparison, `git diff --check`, and local secret/PII review passed. - Proposal inventory contains no startup or connector proposals. The repository extractor fails with TypeScript 7's missing compiler API; rerunning it with the installed TypeScript 5.9 compiler API passed. No extractor source change is included. - Initial local server typecheck attempts found missing workspace dependencies and a missing plugin SDK link in this fresh checkout. Dependencies and the local SDK link were repaired; `pnpm --filter @paperclipai/server typecheck` then passed. Full repository typecheck, tests, and build are covered by the PR CI rather than claimed as local passes. Greptile found that a telemetry state-file error could abort startup. New regression tests reproduced this on the previous head with both an injected permission error and the real state store blocked by a file. Both now pass, including no published client, no periodic flushing, no send, and no retry after failure. Shared telemetry tests and server typecheck were rerun after the fix. Full PR CI passed on `a41540b7899bc8a0aedbd78d72f8b69ad1a4b318`, including typecheck, build, all selected test shards, and the final verify gate. [Successful workflow attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41752039595/attempts/2). Attempt 1 was interrupted when the server (1/12) runner received a shutdown signal; its incomplete result was retained in the PR discussion. The same-commit shard retry and dependent gate passed. Greptile reviewed this exact head at 5/5 with all threads resolved. ## Risks This change starts sending one OS family value per server startup when telemetry is enabled. It sends no OS version, architecture, hostname, path, prompt, or machine identifier beyond the existing envelope. Containers and WSL report the runtime's OS. Event totals reflect restart frequency, so use distinct install identifiers for OS adoption analysis. The complete generated SDK also enables the previously registered connector proposals. Those events keep their existing approved payloads; no connection names, credentials, tool arguments, or results are added. Privacy review is requested for both newly active paths. Deploy the registered schema before merging this app change. Live production delivery has not been tested. No schema migration is needed. Revert this PR to stop the new startup emitter and restore the prior contract. ## Model Used - Original implementation: Anthropic Claude Opus 5.5, as recorded in the original commit. - Completion: OpenAI `gpt-6.1-sol`, xhigh reasoning, through Codex in T3 Code; shell, Git, and automated tests. Context window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the feature issue template - [x] The final change and description contain no private issue links or proposal markers - [x] The existing PR branch is preserved to retain its review history - [x] I have run the targeted tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandClaude Opus 5.5
|
925965a09b |
fix(task-retry): wait out brief chat authorization contention (#18895)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A user can click Retry on a task that is connected to an external
chat conversation.
> - That Retry prepares the chat retry inside one admission transaction.
The chat step takes a per-principal authorization lock with
`pg_try_advisory_xact_lock`.
> - When another chat transaction holds that lock for a short time, the
chat step throws `NativeChatReviewPresentationContentionError`. This
error is not an `HttpError`. The route returns HTTP 500, and the error
handler reports a server crash.
> - The contention is routine and short. The admission transaction does
not contact a provider, and it rolls back completely. The owning service
can safely try it again.
> - This pull request retries the admission a bounded number of times.
If the lock stays busy, the service returns a retryable 409 instead of a
500.
> - The benefit: the user's click usually succeeds without an error, and
Paperclip stops reporting expected lock contention as a crash.
## Linked Issues or Issue Description
**What happened?**
A hosted staging server reported
`NativeChatReviewPresentationContentionError: Chat response
authorization is temporarily busy; no provider delivery was attempted`
from `POST /api/issues/:id/retry`. The stack is `requestTaskRetry` →
`writeExecutionControl` → `prepareFailedChatRunRetry` →
`failedChatRetrySource`. The error came from the
`chat-identity:<company>:<principal>` advisory try-lock. The route
returned HTTP 500 for each occurrence.
**Expected behavior**
Short lock contention must not fail a deliberate Retry click. The
service must wait it out. If the lock stays busy, the service must
return a clear, retryable client error and must not report a server
crash.
**Steps to reproduce**
1. Bind a task to a restricted chat endpoint and conversation. Make the
latest task run a failed run.
2. In a different database transaction, hold
`pg_try_advisory_xact_lock(hashtextextended('chat-identity:<company>:<principal>',
0))`.
3. Call `POST /api/issues/:id/retry`. Before this change, the chat retry
step throws `NativeChatReviewPresentationContentionError` and the route
returns 500.
**Paperclip version or commit**
Reproduced on master `8771606fd87f69ed4057779e3dce3a9bca8088e6`. The
server release that reported it was
`d59791828482f213713ca55000c40e6c9c6c03e5`.
**Deployment mode**
Hosted staging, Node 24, PostgreSQL.
Related: #18845 and #18621 changed nearby Retry admission. They do not
handle this contention.
## What Changed
- `server/src/services/task-retry.ts`: `requestTaskRetry` now calls the
unchanged admission transaction (`admitTaskRetry`) in a bounded loop: 10
attempts, 100 ms apart. It retries only
`NativeChatReviewPresentationContentionError` and lock-not-available
contention (`isExternalChatWaitAuthorizationContention`, which matches
`55P03`). Every other error, including 403/404/409 decisions, propagates
unchanged on the first attempt.
- If the lock stays busy for all attempts, the service throws
`conflict(..., { code: "task_retry_busy", retryable: true })`. The UI's
existing 4xx handling resets the Retry button, and the error handler
does not report a 4xx as a crash.
- `server/src/services/task-retry.test.ts`: three PostgreSQL cases:
- Real advisory-lock contention. A second transaction holds the same
`chat-identity` lock that production checks, then releases it. Retry
succeeds and persists exactly one wake request. A replay with the same
idempotency key returns the same request.
- Persistent contention returns 409 `task_retry_busy` after exactly 10
attempts and persists no wake request.
- A 403 from the chat step does not retry.
The transaction body is unchanged. It was moved into `admitTaskRetry`
without edits.
## Verification
- RED: with the old `task-retry.ts` and the new tests, the contention
case fails with the exact production error
(`NativeChatReviewPresentationContentionError: Chat response
authorization is temporarily busy…`). The persistent case fails because
the error is not an `HttpError`.
- GREEN: `pnpm exec vitest run src/services/task-retry.test.ts` passes
all 39 cases against embedded PostgreSQL.
- `pnpm --filter @paperclipai/server typecheck` passes (all layers,
including tests).
- `src/services/heartbeat/workspace-preparation.test.ts` (another
`requestTaskRetry` caller): 165 of 167 cases pass. The 2 failures
(`remote_workspace_migration_source_changed` in "resumes an imported
path/file-url Git source…") also fail on unchanged master in the same
environment, so this change does not cause them.
- Runtime assumption (rule `runtime-dependency-boundaries`): the retry
is safe only if a failed attempt leaves no persisted state. The
contention test uses a real PostgreSQL advisory lock held by a separate
connection and checks that only one wake row exists after the failed
attempts. Each attempt is a separate transaction, so the loop holds no
row lock or advisory lock while it waits.
- The full repository `pnpm test:run` and `pnpm build` were not run
locally. I rely on CI for those.
## Risks
- Low risk. The worst case adds about 1 second of request time when the
lock stays busy, and then returns 409 instead of 500.
- An attempt can fail after an earlier step in the same transaction (for
example, workspace preservation registration). The whole transaction
rolls back, so the next attempt repeats those steps from a clean state.
Those steps run in the same transaction and do not contact a provider.
- If `requestTaskRetry` is called inside an outer transaction, each
attempt rolls back only to its savepoint. Locks that the outer
transaction holds stay held. In that case the loop ends with the same
409 after its bounded budget.
- No schema, API shape, or UI change.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), through Claude Code with
tool use (shell, file editing, test execution) and extended reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; see
Verification for the two pre-existing failures)
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
behavior or command documentation changes are needed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandPaperclip
|
057898530c |
fix(server): handle wrapped recovery action conflicts (#18885)
## Thinking Path > - Paperclip coordinates durable agent work and recovery. > - Independent workers can find the same task needs recovery. > - The database permits one active recovery action for that task. > - A losing insert must read the action that already committed. > - Drizzle wraps the PostgreSQL unique violation in its cause. > - The current classifier reads only the outer error and misses that conflict. > - This PR recognizes the same allowed conflict through a bounded cause chain. > - Both workers can then return the one durable action through the existing retry path. ## Linked Issues or Issue Description Refs: #18783. **What happened?** The saved-disposition-debt recovery regression failed in CI `41733864320`. Two recovery workers collided on `issue_recovery_actions_active_source_uq`. The fixture captured an unhandled PostgreSQL `23505` error. A separate canceled shard in that run lost its AWS Spot instance; that interruption is a different failure. **Expected behavior** Two service instances that race to create the same active action must return that one action. Unrelated database failures must still propagate. Existing transaction and ownership rules must stay in force. **Steps to reproduce** Hold a PostgreSQL SHARE lock on the recovery-action table. Start the same source-scoped upsert on two independent service instances. Confirm both real INSERTs are waiting in `pg_stat_activity`, then release the lock. The original code persists one action but rejects one caller with a `DrizzleQueryError`. Its outer code is absent; its `PostgresError` cause has SQLSTATE `23505` and the expected active-source constraint. **Paperclip version or commit** Supporting branch starts at master `54782706e1d83d92295246e499627930df03662d`. It now contains a normal merge of master `8771606fd87f69ed4057779e3dce3a9bca8088e6`. The classifier correction stays unchanged. The PR diff against that master still contains only the recovery service and its tests. ## What Changed - Inspect up to eight error causes with cycle detection. - Require SQLSTATE `23505` and one of the same two recovery-action constraint names on the same error. Support the PostgreSQL driver's `constraint_name` field. - Reuse the existing source-scoped retry path and its unchanged three-retry limit. - Add a deterministic real-database collision regression and no-retry controls for unrelated uniqueness, other SQLSTATEs and cyclic errors. - Keep the original recovery regression, assertions and timeouts unchanged. Keep the saved-skill journey separate in #18783. ## Verification - Review follow-up: the real collision test now checks the intended retry merge. The surviving action has attempt count 2, active status, the default unlimited budget, the expected owner, supplied evidence and supplied attempt timestamp. Its wake and monitor fields remain null. Existing identity and convergence assertions run first. - A disposable counterfactual returned the winner unchanged after the lost insert. It passed the original barrier, one-row, no-rejection and same-ID assertions, then failed the new attempt-count check with 1 instead of 2. The counterfactual was not committed. The production service was restored byte-for-byte. The actual service passed all seven focused race, no-retry and sequential controls. The affected canonical test-tree-2 typecheck passed against verified unchanged local declarations. - Parent `4f8abfe4f2244649219ef6ced3534634193f0b53` full CI `41746314291` passed all 48 selected jobs with one intended skip. Its 5/5 summary also had two duplicate unresolved reliability comments about the missing state assertions. This test-only follow-up addresses both. Fresh full CI and current-head review are required again. - The deterministic real-database regression failed before the correction. Both inserts reached the lock and exactly one active action persisted. The losing error was a `DrizzleQueryError` with a `PostgresError` cause, SQLSTATE `23505` and constraint `issue_recovery_actions_active_source_uq`. - **11 focused checks passed across two files on the integrated head** (181.72 seconds). This includes the real-database collision, no-retry controls and all four unchanged recovery producer/dependency regressions, including the original failed real-clock saved-debt case. Both callers return the one persisted action. - Current-head canonical server typecheck passed on Node24.22.2 with all seven layers: below, cycle, above, tests and test-tree-0 through test-tree-2. Canonical shared, SDK, DB and Runner TypeScript prerequisites passed. The first filtered installs lacked CLI and UI imports required by the new test layers. Those module-resolution failures remain recorded as setup failures. A full frozen workspace install with lifecycle scripts disabled supplied the same locked dependencies. No source or type-configuration workaround was used. No Rust or broad workspace build ran locally; full repository typecheck, tests and build remain hosted-CI gates. - The actual supporting master baseline reproduced the same wrapped23505 failure before the correction. Earlier configuration-resolution, local IPC and native-library prerequisites remain separate setup failure evidence. No test failure was reclassified as a pass. - First supporting CI `41739489620` was canceled before its build/test jobs completed. Its later automatic full-CI run `41739747947` failed. The recovery server shard passed. The Runner failure was the Cursor fixture case repaired upstream in #18782, now included by the normal master merge. That CI also observed an attachment-browser timeout, a repository-rehome timeout and a lost-runner connection. Their underlying causes remain unknown. A later pass will establish recovery, not explain those earlier failures. These original results remain retained. - Current `fbd392b9b65dbcf4ac078825096037ed0d490a0b` full CI [41751706634 attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41751706634/attempts/2) passed: 48 successful jobs and one intended skip. Attempt 1 remains retained: ten primary jobs lost their bound Spot capacity and the verify job failed as a dependent. A single ordinary failed-job recheck produced eleven fresh successful executions (ten primaries and verify); timestamps prove 37 successful executions were retained. The skip is not counted as execution. Current Greptile is 5/5 with both addressed reliability threads resolved and no unresolved findings. All 57 current check runs are successful, neutral or skipped, and Snyk is successful. This is CI recovery; it does not explain the separately retained older attachment and repository timeouts. - The original CI failure and both failed saved-skill QA attempts remain unchanged. This PR does not establish a live QA pass or deploy a production change. ## Risks Conflict classification must not turn unrelated errors into retries. The new controls require those errors to propagate after one insert attempt. The traversal is bounded and stops at cycles. Transaction, company scope, ownership, notifications, retry count and database constraints are unchanged. No schema or public API change is required. ## Model Used OpenAI GPT-6 through Codex with reasoning and code execution. The runtime does not expose a more specific model ID or context-window size. These checks make no model or provider calls. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Michael NguyenandPaperclip
|
58d1b21098 |
fix: separate fresh setup from archived personal connections (#18902)
## Thinking Path
> - Paperclip helps people manage AI agents and their service access.
> - Each personal connection belongs to one user.
> - Setup can recover an archived connection by its name and provider.
> - A fresh setup can therefore select another user's old identity and
fail its owner check.
> - This change creates a separate connection for fresh setup after
personal archival.
> - Explicit reconnect still requires the existing owner and preserves
its history.
## Linked Issues or Issue Description
**What happened?**
A fresh GitHub setup can return 403 after a personal connection with the
same name was archived. The request does not include a reconnect target.
Setup still selects the archived identity by name and provider.
**Expected behavior**
Fresh setup creates a separate personal identity. It leaves the archived
connection, credentials, grants, and activity unchanged. Explicit
reconnect rejects a different user.
**Steps to reproduce**
1. Create a personal GitHub connection in a test company.
2. Archive that connection and its application.
3. As another authenticated member, start fresh personal setup with the
same name and provider.
4. Observe the owner-check failure before this change.
**Paperclip version or commit**
Base:
|
||
|
DottaandPaperclip
|
e2dc2cc0be |
fix(native): retire idle shared-sandbox ingress owners before dispatch (#18893)
## Thinking Path > - Paperclip manages agents and their tasks. > - Native tasks can share one reusable sandbox workspace. > - A completed child can keep its provider process warm. > - A parent has its own logical session, but the sandbox has one ingress listener. > - Retirement by task identity leaves the child's listener in place. > - A new parent Runner then fails to bind that listener. > - This pull request retires a proven idle physical owner before the parent's distinct session starts. ## Linked Issues or Issue Description Refs #15353 and #17097 for listener configuration and warm transport rotation. Refs #18682 for owned process retirement. #18860 repairs a different resume-source failure. Searches found no existing repair for retained warm sessions on different tasks that share the same physical ingress allocation. **What happened?** A completed child and a later parent can use the same reusable sandbox. Warm-session retirement compares logical task ownership. It does not find the child's distinct task scope. The child keeps the fixed ingress listener, and the parent's new process fails with `runner_ingress_bind_conflict` / address already in use. **Expected behavior** A parent must start in its own logical session after the proven idle prior listener closes. Active, foreign or unverified owners must remain protected. **Steps to reproduce** Run the `retires the completed child listener` cases in `native-session-executor.test.ts`. The actual executor registers a warm child. PostgreSQL holds its committed, accounted result and clean expired lease. A distinct parent receives the same allocation and current workspace writer. The provider boundary uses real TCP listeners. The original executor reaches the second bind with the child still listening and fails with EADDRINUSE. The changed executor waits for the owned close, then starts the distinct parent session. **Paperclip version or commit** Reproduced on |
||
|
Devin FoleyandClaude Opus 5.5
|
ad5cd785b1 |
fix(wake-queue): keep promoted Retry runs scoped to their issue (#18876)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The wake queue starts the next run for a task when the task's
current run releases its execution lock.
> - A board Retry saves a receipt with only `{ issueId, taskRetry }`.
This receipt has no saved wake context.
> - When the release drain promotes that receipt, the promoted context
gets `taskKey` from the payload, but it does not get `issueId` or
`taskId`.
> - The new run takes the task's execution lock, but the database saves
it as a company-scoped run with no issue.
> - Low-trust workspace preparation then has no issue reference. The run
fails before provider work with "Low-trust execution issue is outside
the active trust boundary."
> - This pull request restores the issue scope of a payload-only receipt
during promotion.
> - The benefit is that a Retry on a low-trust task runs in its task
boundary and does not fail during setup.
## Linked Issues or Issue Description
**What happened?**
A user clicked Retry on a low-trust task while a run was busy. When the
busy run ended, the release drain promoted the saved Retry receipt. The
promoted run had `issue_id = NULL`, `scope_kind = company`, and a
context snapshot with only `taskKey`, `wakeReason: task_retry`,
`wakeSource`, and `wakeTriggerDetail`. Workspace preparation failed with
`low_trust_boundary_mismatch` (HTTP 422) in the setup phase.
**Expected behavior**
A promoted run for a task must keep that task's issue scope. This is
true also when the deferred receipt has no saved wake context. Workspace
preparation must then evaluate the task's own trust boundary.
**Steps to reproduce**
1. Use a task with a low-trust execution policy whose assignee is an
agent.
2. While a run on that task is busy, click Retry on the task.
3. Let the busy run end. The release drain promotes the deferred Retry
receipt.
4. The promoted run is company-scoped with no `issueId`. It fails in
setup with `low_trust_boundary_mismatch`.
**Paperclip version or commit**
`master` at
|
||
|
DottaandPaperclip
|
7b1c53f8f1 |
fix(recovery): continue verified preparation after lease interruption (#18892)
Use recorded system interruption and the existing execution, controller, stopped-lease, and saved-file proof instead of a shutdown-only error label. Preserve bounded retries, user Stops, task authority, and exactly one continuation. Keep the chat recovery fixture faithful to the preparation stage and terminal time written by the real lifecycle. Retain missing-field safety controls. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
db9be4c5a6 |
fix(workspaces): preserve safe one-shot failure messages (#18888)
## Thinking Path > - Paperclip manages agent work and keeps saved work safe. > - Remote workspaces use provider commands to build and copy a verified archive. > - Some providers return a one-shot command's combined output in stdout. > - The workspace transport checked only stderr for its two safe failure messages. > - This hid those messages behind a generic error on that provider path. > - This pull request checks both streams for the same exact messages. > - The benefit is a useful existing error message without exposing other output. ## Linked Issues or Issue Description **What happened?** Daytona one-shot execution returns combined output in stdout and leaves stderr empty. The remote workspace transport checked only stderr after a failed command. It therefore lost the existing full-disk and unfinished-Git-operation messages. **Expected behavior** Preserve those same two exact safe messages from either returned stream. Keep all other output private. Keep the existing stderr preference when both streams contain different safe messages. **Steps to reproduce** Run the real workspace capture helper with a conflicted Git merge or an injected ENOSPC write. Apply the provider's one-shot output contract to the real child result. Before this fix, all three new cases return only remote_workspace_operation_failed. Related provider contract: #10941. The duplicate search found no PR for this exact transport classification gap. ## What Changed - Check stderr and stdout for only the two existing exact safe failure messages. - Add real helper regressions for one-shot Git and full-disk failures. Keep the conflicted files and original saved result intact. - Add privacy, unknown-output, split-message, stderr-preference and healthy verified capture/restore controls. ## Verification - RED: three real helper cases fail on the unchanged transport. They cover a conflicted Git merge and ENOSPC during helper installation and archive creation. - GREEN: 73 tests pass in remote-workspace-transport.test.ts and remote-workspace-snapshot-archive.test.ts. - Server layered typecheck passes: `cd server && node scripts/typecheck-layers.mjs` with Node24.22.2. All seven source/test layers pass. - Full CI is requested with the full-ci label. Current-head CI and review are pending. - The local install used the frozen lock with scripts disabled. The maintained root SDK-link helper restored its skipped local postinstall links. The original prerequisite failure is retained. Generated package outputs were copied only after complete input, installed-toolchain and output-hash equality checks; server declarations were checked in this worktree. - The full workspace typecheck, test suite and build were not run locally. This two-file change uses focused real-helper tests and server typechecking; full CI supplies the broader checks. ## Risks - A failed command can now show one of the existing safe messages when the provider puts it in stdout. Unknown, prefixed, suffixed and private output still produces the generic error. - This change does not repair an archive failure or prove that a retained workspace was saved. It changes no capture, ownership, archive validation, stop, save or retry guard. No live provider command or recovery action was run for this PR. - No API, permission, schema, configuration, map or documentation contract changes are required. ## Model Used - OpenAI GPT-6 (Codex), with reasoning, tool use and code execution. The host does not expose the exact deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Nicky LeachandClaude
|
8fd917aedf |
fix(native): accept late provider notices in recovery (#18890)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep a provider conversation across task turns. > - When a later run gets a new task lease, recovery must prove that the prior run settled. > - Recovery required `run.terminal` to be the last committed event. > - A provider can send a late notification for a settled turn. The Runner then records a `provider.notice.recorded` warning after `run.terminal`. > - This false denial stranded a clean session. The notice stays in the journal, so each later run on that session meets the same denial. > - This pull request lets recovery accept a recoverable notice after the terminal event. All other new events still block recovery. > - The benefit is that a late warning no longer ends session continuity. A denial now also names the check that failed. ## Linked Issues or Issue Description No issue exists. I searched open and closed pull requests and issues for the error code, the event name and the function name. I found no duplicate. **What happened?** A run failed in about nine seconds with `runner_state_identity_mismatch: plugin_checkpoint_recovery_unproven`. No provider work started. The prior run had finished and was committed as succeeded. Its journal had one `provider.notice.recorded` warning, "ignored a notification for a settled Codex turn", 1 ms after `run.terminal`. A second session had the same warning about 61 seconds after a normally completed turn. **Expected behavior** Recovery accepts the session and continues the provider conversation. A recoverable notice does not start work, so it must not block recovery. **Steps to reproduce** 1. Start a warm native session in a plugin sandbox environment. 2. Finish a run so that a provider notice follows `run.terminal` in the journal. 3. Wait until the idle timeout stops the Runner task and releases the lease. 4. Start the next run on that session. It gets a new task lease. 5. The run fails with the error above. The notice stays in the journal, so a retry on the same session meets the same denial. The `local-receipt-late-notice` test mode in `native-session-executor.test.ts` reproduces this failure on the unfixed code. **Paperclip version or commit** Observed on `d8a9b3eb0c`. The same check is on `master` before this change. **Deployment mode** Cloud deployment, with a native Runner in a plugin sandbox environment. ## What Changed - `isSettledRemoteWarmControlState` accepts a `provider.notice.recorded` event after the terminal event. The notice must be recoverable and have `info` or `warning` severity. It must come from the same runner, run, session and lease. - Error notices, unrecoverable notices, notices from another source and new work still block recovery. - Checkpoint recovery and local quiescent recovery now use this function. They no longer have separate checks that require `run.terminal` to be the last event. - Each checkpoint recovery denial adds a specific reason after the existing code, for example `plugin_checkpoint_recovery_unproven: journal_not_settled`. The `runner_state_identity_mismatch: ` prefix does not change. - I reworded the existing comment on `isSettledRemoteWarmControlState` so that it stays correct. I added no new code comments. ## Verification - **Failure before the fix:** I added the tests first and ran them on the unfixed code. 38 tests failed. In the 4 `local-receipt-late-notice` cases, checkpoint recovery rejected with `runner_state_identity_mismatch`. In the `late provider notice` case, quiescent recovery rejected with `quarantine_owner_unverified`. The new unit test for notices also failed. The other 32 failures were assertions for the new denial reasons. - **Targeted tests after the fix:** 134 tests passed, with 987 skipped by the filter. - `cd server && pnpm exec vitest run src/services/native-runtime/remote-warm-session-recovery.test.ts src/services/native-runtime/native-session-executor.test.ts -t "settled remote warm control evidence|recovers a blocked|automatically recovers only a proven settled local session"` - **Type error found in review:** Vitest does not check types, and my first version passed these tests. Review of the diff showed an incompatible `identity` parameter type. I changed that type to `Record<string, unknown>`. - **After the type change:** `remote-warm-session-recovery.test.ts` passed (23 tests) and `pnpm run typecheck` in `server` passed. I did not run the 134 tests again, because this change affected only types. - **Real data, read-only:** I ran the new function on 285 session journals from a test deployment. The result changed for 2 journals. These are the 2 stranded sessions, and each has only provider notices after `run.terminal`. - **Not run locally:** I did not run the full test suite. I stopped a run of the wider native-runtime test files before it reported a result. CI provides the full regression coverage. - **Not verified:** no live run has reached `read_state` on a stranded session after this fix. The tests use a fixture for that call. ## Risks - **Stricter identity check:** checkpoint and quiescent recovery now also require the journal's saved identity to match. Recovery reads this identity from the same file, so only damaged state should fail this check. - **Wider goal snapshot rule:** these two paths now accept idle goal snapshots after the terminal event. Warm handoff already used this rule. - **Wider notice rule:** warm handoff and the settled projection check also accept recoverable notices now. A notice does not start work, and the checks for pending commands and unanswered tool input stay in place. - **Longer error text:** code that compares the full `plugin_checkpoint_recovery_unproven` message for equality would stop matching. I found no other reference to this code in the repository. - **Stranded sessions:** these sessions need no data repair. They can still fail with `checkpoint_not_settled` if the sandbox checkpoint is not clean. - **Migrations:** this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the diagnosis and approved the change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Dotta
|
82dd3bb702 |
Preserve unstarted queued turns during offline enrollment (#18887)
## Thinking Path > - Paperclip manages AI agents and their work. > - Existing-volume enrollment moves server files to durable object storage. > - Enrollment stops application deployments before it changes file authority. > - A task drain keeps unstarted turns queued in the database. > - The migration rejected every queued row, so a busy instance could never finish enrollment without discarding requests. > - This change preserves unstarted queue intent and continues to reject running or previously started executions. > - Users keep their pending turns and saved files through the conversion. ## Linked Issues or Issue Description **What happened?** A preserving enrollment found queued turns after task admission stopped. The workspace, instruction-file, and historical-log guards treated these rows as live writers. The drain also prevented the rows from starting, so waiting could not satisfy the migration guard. **Expected behavior** After every old deployment has stopped, maintenance can move files while unstarted queued turns remain unchanged. The replacement claims those turns through normal admission. Actual executions and retained execution ownership continue to block migration. **Steps to reproduce** 1. Queue a turn behind another task. 2. Start a task drain and let the current task finish. 3. Stop the application deployments and run offline enrollment. 4. Observe `active_execution_requires_drain` for the unstarted queued turn. **Paperclip version or commit** Reproduced on the migration contract through #18864. This branch starts at `ab2d2098a2764c511978d1c3e7a3fc5dfcef07f4`. Related: #18826, #18835, #18841, #18864. Searches for queued enrollment and queued migration found no duplicate fix. ## What Changed - Share one offline execution predicate across workspace, agent-file, and historical-log migration. - Reuse the existing pre-dispatch evidence predicate; queued turns with execution markers, saved checkpoints, source events, coordinators, or environment leases remain blockers. - Keep task bindings and normal live-migration restrictions unchanged. - Exercise normal replacement admission after deleting the old filesystem: the same queued run reads restored workspace bytes and instructions, completes its original wake, and cannot execute twice. - Document the offline boundary and queued-turn behavior. ## Verification - **111 tests passed** across the final 110-case migration suite and assembled queued-turn heartbeat proof, with real PostgreSQL and local S3. This includes workspace/log migration, queued ownership blockers, ordinary agent-file controls, and the actual bundled maintenance process. - The maintenance process preserves the queued row while archiving historical output. The new assembled heartbeat test creates a normal wake under drain, migrates without changing either row, deletes the original filesystem, and resumes through normal admission. It reads saved workspace bytes and the restored instruction bundle, persists one successful native result, completes the same wake, and verifies a second sweep does not duplicate execution. Only sandbox transport/provider execution are scripted; no live model or Daytona calls. - An earlier optional run-evidence worker integration invocation was skipped without its dedicated environment; it is outside the changed offline path. - Server typecheck, server build, and runtime bundle passed. - The first maintenance test exposed the separate agent-file guard; that guard now uses the same offline predicate and the test passes. - Initial local typecheck attempts lacked CLI dependency links in this isolated checkout. Restoring dependencies fixed setup without source changes. - Full repository checks run in CI. The first head lost spot runners in Build and serialized tests; one unchanged idle-deadline fixture crossed UTC midnight and failed its timestamp assertion. Final-head CI passed on run 41744678994 after retrying a spot-runner shutdown. Greptile reviewed the exact final head at 5/5; all inline findings are resolved. The separate Paperclip Review app remains queued. Hosted enrollment remains pending release qualification and compatibility verification for the existing provider configuration. ## Risks The offline predicate relies on Cloud stopping every old application deployment. It does not make online migration safe. Running executions always block. Requeued executions with retained ownership also block. No queued request, task binding, or recovery state is removed or rewritten. No database schema changes. ## Model Used OpenAI Codex, GPT-6 based, with reasoning and tool-assisted code execution. The exact deployed model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Devin FoleyandPaperclip
|
4e29f00139 |
fix(native): keep settled ACPX sessions across Runner upgrades and run scheduled resumes (#18879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps one ACPX provider conversation across the per-turn runs of a task. Each run cold-attaches the session that the previous run settled. > - The saved ACPX state binds the digest of the launch profile that wrote it. Every deploy ships a new provider pack, so that digest changes. > - After a deploy, the next run of such a task fails `run.attach` with `ACPX durable launch profile digest does not match runner startup`. Every same-run retry fails the same way. > - On one instance this happened seven times. Six tasks used their retry budget and ended blocked, or in progress with no runner. In the seventh, the scheduled same-run resume never ran at all: the task stayed `in_progress` with nothing running, and its monitor did not fire. > - This pull request re-binds a settled session to the upgraded launch profile, types the remaining mismatches so the server stops retrying them, and makes a scheduled remote resume survive the stale-lock sweep. > - The benefit is that a task continues its conversation after a deploy. When it cannot, it stops with a clear next action instead of stalling. ## Linked Issues or Issue Description Refs #18846, #18847, #18854 and #18860 (nearby native recovery changes). Refs #15795, an open change to the same stale-lock sweep function. It changes run teardown, not lock cleanup, so a second merge needs at most a small rebase. **What happened?** A per-turn native task used provider `acpx` with Claude, driver `acpx_runtime`, on a remote sandbox. 1. Run A succeeded and settled its provider session. The next run with the same build cold-attached it. 2. The instance deployed a new build. 3. A board approval woke run B in a fresh sandbox. Run B failed after about five minutes: `native_session_recovery_failed: Error: PRP command run.attach failed: {... "result":{"code":"command_execution_failed","message":"ACPX durable launch profile digest does not match runner startup"}}`. 4. The coordinator recorded `retryable_failure`, attempt 1, `native_session_interrupted`, recovery mode `exact_checkpoint_resume`, recovery state `resuming_session`, and a retry 30 seconds later. 5. Two seconds after the failure, the stale-lock sweep cleared the task's execution and checkout locks, because run B was `failed`. 6. Forty minutes later the attempt was still 1. The scheduled monitor had not fired. The task stayed `in_progress` with nothing running. **Expected behavior** A session that the previous run settled continues after a deploy. If it cannot continue, the run stops at once with a clear next action. A scheduled resume either runs or fails visibly. **Steps to reproduce** 1. Save a suspended ACPX session under one launch profile. 2. Change the sidecar bytes and the provider-pack authority at the same paths. 3. Cold-attach the next run in a new runner. Restore fails with the digest mismatch. 4. For the stall: leave a remote native run `failed` with a due `retryable_failure` coordinator, run the stale-lock sweep, then run restart recovery for that run. It returns `blocked: issue_execution_lock_changed` and writes nothing. The new Rust and PostgreSQL tests run these steps. **Paperclip version or commit** `master` at `b0c4e345267`. **Deployment mode** Native Runner, per-turn lifecycle, remote sandbox. The re-bind applies to every ACPX agent with an ordinary startup. ## What Changed ### Root causes - **Upgrade failure.** `AcpxLaunchProfile::canonical_digest` hashes the provider-pack authority, the sidecar command and arguments, each artifact path and SHA-256, and any candidate admission. For a remote sandbox the authority is the provider-pack manifest digest, which covers the Runner source revision and dist digest. So every deploy changes it. `AcpxDurableState` stores only the digest, and `validate()` rejects any difference. Only a Kimi candidate run binding had a carve-out. - **Silent stall.** The failure write keeps the task locks for `retryable_failure` and clears them only for `terminal_failure`. The stale-lock sweep treats any `failed` run as cleanable, so it cleared them. A remote retry goes through `claimNativeRestartRecoveries`. That claim requires `issues.execution_run_id` to name the run, so it returned `blocked: issue_execution_lock_changed` with no coordinator write and no deadline. The live recovery queue then dropped the entry. The local claim (`claimNativeSessionResumptions`) already reclaims a free lock, so only remote runs stalled. - **Monitor.** `noActiveNativeMonitorRun` holds a task's monitor while any native run for it has a coordinator that is not `committed` or `terminal_failure`. Run B's `retryable_failure` held it. That is by design; it ends when the resume runs. ### Runner (Rust) - `rebinds_settled_session` in `acpx_provider_backend.rs`. Restore re-binds a saved session to the current launch profile only when all of these are true: - The lifecycle is `suspended` and an identity exists. There is no active turn, pending runtime request, undelivered provider event, or unconfirmed provider exit. - The startup has no candidate admission. - The saved descriptor's sidecar command and arguments equal this startup's. - Any other digest mismatch keeps failing with the same message, plus `classification: "provider_state_incompatible"` in the durable command result. `DurableRunnerError` carries an optional closed classification for this. - A saved descriptor that names an agent release this build does not qualify (ACPX version, agent server package or version, or command digest) also gets the class. `validate_session()` and its message are unchanged; restore only adds the class. ### Runner (TypeScript) - `runnerRecoveryFailureClass` reads the classification from the authenticated `runnerCommandResult`. The Codex app-server driver (which also drives ACPX through runnerd) and `HarnessDriverBackend` pass it on as `failureClass`. - If replacement after a failed resume is not allowed, `executeNativeSession` throws `NativeProviderStateIncompatibleError` (message unchanged: `native_session_recovery_failed: ...`). If replacement is allowed, the existing path opens a fresh session in the same run, as before. ### Server - `nativeSessionFailureSourceCode` maps that error class to `native_provider_state_incompatible` with `instanceof`, not text. The disposition is permanent: `terminal_failure` at once, the task blocks, and the board owns the recovery action. Its next action says that a Retry continues in a fresh provider session with the same workspace. - `findNativeSessionResumeRun` also returns the barrier's `errorCode`. `rebindNativeSessionCheckpoint` does not resume a source with `native_provider_state_incompatible`. The existing fallback in `buildNativeExecutionWithCheckpoint` then starts a fresh normalized session with the full task bootstrap. - `sweepStaleIssueLocks` keeps both locks while a referenced run's coordinator owns a result-less `retryable_failure`, unless the task is done or cancelled. Its update also requires that no referenced run has a live status at the write, because a resume claim can return the run to running after the status read. - `claimNativeRestartRecoveries` reclaims a free execution lock for a `retryable_failure` resume, like the local claim does, when the checkout lock is free or held by this run and no other run for the task is queued, running or scheduled. It records `native.recovery.execution_reclaimed`. A lock held by another run still blocks. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` and `doc/execution-semantics.md`. ### Safety argument for the re-bind What the binding protects, and what this change keeps: - **Candidate admission** carries company, model, platform and expiry authority. A candidate startup never re-binds. The Kimi run-binding carve-out and all candidate rejections are unchanged. - **The agent that loads the conversation** stays exact. `validate()` runs on the saved descriptor before the digest check and requires this build's qualified agent profile: ACPX version, agent server package and version, and `commandDigest`. The saved identity must still match that profile and the provider mode. - **In-flight provider work** is never re-bound. A turn, runtime request, undelivered event or unconfirmed exit keeps the strict binding, so no old provider process or request continues under a new build. The next launch only loads a settled conversation. - **Launch shape.** A changed sidecar command path or arguments still rejects. Only the content at the same paths may change. - **Artifacts** are not trusted from the saved state. The current profile comes from the trusted controller at startup, and session start verifies every artifact's bytes and the restored provider identity, as before. What changes: a new provider-pack authority, or new sidecar or Node bytes, at the same paths no longer invalidates a settled conversation. The saved digest cannot show which part changed, so the rule does not depend on it. I did not add state fields to tell authority changes from artifact changes, because the rule would treat both the same. One existing assertion changes. `released_kimi_cold_restore_rejects_candidate_injection_and_changed_full_profile` expected a settled ordinary Kimi session to reject a changed authority, artifact digest, or saved digest. Those three cases moved to a new positive test. Command, arguments, every candidate grant field, a missing profile and run binding still reject. ### How a task in the reported state recovers after this deploys The task has old-digest saved state, a `retryable_failure` coordinator at attempt 1, and cleared locks. 1. Startup recovery, and the live recovery queue, run restart recovery for the due run. It reclaims the free execution lock and claims the run. The resume runs now. 2. Run B's failed `run.attach` is a durable outcome of B's exact run authority. If the resume replays that outcome instead of executing it again, it fails with the old, untyped text. The remaining attempts then fail the same way, and the run ends `terminal_failure` with `native_session_retry_exhausted`. The task blocks with a board recovery action. This takes about a minute. 3. If the task blocks, press **Retry** once. Retry selects run B as the source (it has process identity and controller evidence, so the #18860 skip does not apply) and rebinds B's checkpoint, which holds A's provider session. The new run cold-attaches under the new Runner, and restore re-binds the old digest because the session is suspended and quiescent. The conversation continues. Run B's failed attach never wrote the state file, so the state is still A's settled state. 4. Tasks that already exhausted their retries (blocked, or in progress after `native_session_retry_exhausted`) need the same single Retry. Their earlier attempts ran under the old Runner. Wait for the task to show blocked before pressing Retry. A Retry pressed while the resume is running joins that run. ## Verification - Rust, failing first: `settled_session_rebinds_to_the_upgraded_runner_launch_profile` failed on `master` with the production message. `unsettled_or_relocated_session_keeps_the_strict_launch_binding` failed on the missing classification. Both pass now. - `cargo test -p paperclip-runner-core --no-fail-fast`: all 41 test binaries pass. The library has 471 passed and 1 ignored. - `cargo fmt --check`, and `pnpm --filter @paperclipai/paperclip-runner typecheck` (TypeScript, generated contracts, `cargo check`): pass. - Server, failing first: `native-scheduled-resume-lock.test.ts` (PostgreSQL). On `master` the sweep cleared the locks and restart recovery returned `issue_execution_lock_changed`. Both negative controls (another run's lock, a queued run) already passed. All four tests pass now. - `pnpm --filter @paperclipai/server typecheck`: pass. - Runner vitest (`src/native-session-runtime.test.ts`, `src/backends`, `src/drivers/codex`, `src/drivers/provider-failure.test.ts`, `src/live/runnerd-codex-transport*.test.ts`, `src/control-plane/acpx-controller-recovery.test.ts`): 31 files, 1,011 tests pass. New: the typed error when same-session recovery is required, and the recovery class reader and pass-through. - Server vitest: the native executor, status corpus and provider-pack admission suites (1,268 tests) pass. New: the classifier makes the typed error permanent, and the same text stays retryable. Checkpoint selection returns `errorCode`, and an incompatible source starts a fresh session with the same workspace (unit and PostgreSQL). - Server vitest over the recovery area: stale-lock sweep, native resumption, process recovery, failed-workspace recovery, restart recovery (integration, diskless, preparation), controller recovery, startup retry, startup cleanup, recovered-owner follow-up, session resume, runtime selection and recovery deadlines. 926 passed. Four `heartbeat-process-recovery` tests failed in that loaded run. Three pass when I run them alone. `redacts opaque environment-bound credentials from Sentry diagnostics: result` fails the same way on unmodified `master` on this host (a local environment value is redacted). - Review round 1: a new sweep test returns the run to running before the clear. It cleared the locks without the write recheck and keeps them with it. The strict Rust restore test covers an older agent server release and an older ACPX release. The ACPX backend and durable runner unit tests, the lock, sweep, resume projection and controller recovery suites, `cargo fmt --check` and the server typecheck pass. In the second full cargo run, the two `stdio_mcp` relay tests timed out waiting 5 s for a child exit on this loaded host. They do not use the changed code and passed in the first full run. - After the rebase onto the current `master`: the new lock test, failed-workspace recovery, stale-lock sweep and session resume suites pass (189 tests). Server typecheck, `pnpm check:module-boundaries` and `git diff --check` pass. - CI on `929ef4508d2`: in the first run, `isolated-repository-rehome` hit its 15 s test timeout, `idle-completed-history` could not reach its fixture runtime service, and e2e shard 3 was cancelled by a runner shutdown. A rerun of the failed jobs passed. All 55 checks pass. - Not verified: a live remote sandbox deploy, and whether run B's same-run resume replays its recorded `run.attach` failure or executes it again. The Retry path is tested at source selection and checkpoint rebind (server) and at cold attach (Runner). It is not tested end to end through a sandbox. ## Risks - A settled session now continues under new sidecar bytes. If a sidecar change made an old ACPX record unreadable, session start fails as any failed resume does. It does not corrupt state, and the run follows its normal recovery policy. - The sweep now keeps locks for a failed native run with a scheduled resume. If that resume can never be claimed (for example the agent stays paused), the task keeps the lock until the coordinator settles or the task closes. Before, the sweep cleared it, and the resume still never ran. - If another run holds the task's lock, a scheduled remote resume still stays blocked and the coordinator stays `retryable_failure`. This change does not settle that case. It needs a separate decision about superseding the resume. - `native_provider_state_incompatible` is a new failure code. No UI or migration change is needed. Older servers never see it. - No schema migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use (shell, file edits, test runs). The runtime does not expose the context window or reasoning configuration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ab2d2098a2 |
test: verify sleep producer readiness and CI contracts (#18795)
Reviewed and verified at
|
||
|
|
ce332ecec0 |
fix(execution-admission): keep admitted project sets after settlement (#18871)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - #18746 added execution-admission storage: one admission row per attempt and one charge row per admitted project. > - Settlement deletes every charge row. The admission row keeps only the primary project and an opaque context revision. > - Thus, after settlement, the complete admitted project set is lost. > - A retained provider session still carries the references of its previous admission. Example: a task mentions A and B, the admission settles, and then a user removes B from the task text. The next admission must still count B, or start a fresh context. The primary project or the context token cannot supply B. > - This pull request keeps the complete admitted set on the admission row after settlement. Charge rows stay the only active capacity. > - The benefit is that later retained-session resolution can read the previous set. Settled history does not hold capacity or block project deletion. ## Linked Issues or Issue Description Refs #18746 (corrects its storage). Related: #2831, #14333. **Subsystem affected** Server execution admission (`server/src/modules/execution-admission/`) and the `packages/db` schema. **Problem or motivation** `settleAdmission` deletes all charges of an admission. No other storage records the complete project set. After settlement, the set cannot be recovered. Current task text is not a replacement, because it can change after admission. **Proposed solution** Add `execution_admissions.admitted_project_ids uuid[]`. It holds the sorted, distinct set. It is written once at admission and a trigger blocks changes. It has no project foreign key. `{}` is a known empty set. Null means unknown. Only settled rows can have null. Migration `0428` fills the set of unsettled `0426` rows from their charges, because those charges are still complete. Settled `0426` rows stay unknown. **Alternatives considered** - Keep charge rows after settlement with a `released_at` column and a partial active index. The project foreign key would then block project deletion permanently, or the key must be removed from active charges. One table would also hold two different facts. - A separate history table with one row per project. This is more rows, joins and constraints for data that is only read as a whole set by admission ID. - Encode the set in `context_revision`. That token is opaque and owned by the context owner. The database cannot validate it. - Use `heartbeat_runs.context_snapshot`. It is mutable and is per run, not per admission attempt. ## What Changed - Schema and migration `0428_retained_admitted_project_sets`: add `admitted_project_ids`, a check (null only when settled, no null elements, at most 100 entries) and an immutability trigger. The backfill fills unsettled rows from their charges and leaves settled rows null. The migration is safe to run again. Migration `0426` is not changed. - `recordAdmission` writes the set. A replay compares against the stored set. `settleAdmission` still deletes the whole charge set in one transaction and does not change the stored set. - Add `readAdmittedProjects` (company-scoped, by admission ID). It returns `known` with the set, or `unknown`. - Add a pure `retainedSessionProjects` rule: a retained session uses previous ∪ current. An unknown previous set returns `fresh_context_required`. No consumer uses it yet; C1b integrates retained sessions. - Update `doc/DATABASE.md` and `server/src/modules/README.md`. Not changed: no public cap writes, no claim or dispatch integration, no session integration, no provider activation, no `serialize` → cap 1 migration. Active counts still use only the indexed charge rows. ## Verification Red, on unchanged `master` code, with the new tests: - `server`: `npx vitest run src/modules/execution-admission/adapters/postgres.test.ts`. The new tests fail. With a temporary reader that uses the only storage `master` has (charges), the settled A+B admission reads `[]` instead of `[A, B]`. - `packages/db`: `npx vitest run src/execution-admission-migration.test.ts` fails, because no `0428` migration exists. Green, with this change: - `server`: `npx vitest run src/modules/execution-admission`: 27 passed. The new PostgreSQL test covers: A+B admission → settlement → set still readable, charges and counts zero, settlement replay keeps the set, a second attempt with a different set, primaryless and empty sets, company isolation, rollback, immutability, the check constraint, and deletion of a project that has only settled history (its ID stays in the set). It also checks that a project with an active charge still cannot be deleted. - `packages/db`: `npx vitest run src/execution-admission-migration.test.ts`: 2 passed. The upgrade test starts from the shipped `0426` schema with an unsettled A+B row, an unsettled empty row and a settled row. After the upgrade the sets are `[A, B]`, `{}` and null. A second run of `0428` changes nothing. - `pnpm --filter @paperclipai/db typecheck` and `check:migrations` passed. `node scripts/check-module-boundaries.mjs` passed. - Not run locally: the server layered typecheck. The local process was killed for memory (a 5.8 GiB limit shared with other jobs). CI runs it. This PR does not prove C2 claim races or C3 lifecycle authority. ## Risks - Migration: one `ADD COLUMN`, an `UPDATE` that only touches unsettled rows (none in production, because nothing writes admissions yet), a check and a trigger. It does not delete or rewrite data. - Settled rows created before `0428` stay unknown. A future consumer must start a fresh context for them. It must not treat them as empty or primary-only. - The stored set has no project foreign key. Company scope is checked at admission time, under the same project locks and the charge foreign keys. Size: about 70 production lines (schema 8, migration SQL 22, adapter 28, domain 15), 105 test lines and 24 doc lines. Generated metadata (`0428_snapshot.json`, `_journal.json`, removal of `0422_snapshot.json` by the standard pruning) is separate. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, embedded PostgreSQL tests). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: nickyleach <331803+nickyleach@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
11ac90e31f |
fix(build): typecheck server/src/__tests__ and fix every type error (#18739)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server has about 1,100 test and helper files in `server/src/__tests__` > - `server/tsconfig.json` excludes that directory, Vitest does not check types, and no script runs the tsconfig in that directory > - So a test can call a function with the wrong arguments and still pass, and no tool reports it > - This gap hid 2,168 real type errors, and some of them hid tests that did not test what they claim > - This pull request fixes all of these errors and adds the directory to the server layers from paperclipai/paperclip#18737 > - The benefit is that a type error in a server test now fails the typecheck, with no baseline and no backlog ## Linked Issues or Issue Description Refs paperclipai/paperclip#18737, which is merged and added the server layers. Refs paperclipai/paperclip#18738, which is merged and checks the test projects outside `pnpm -r`, so it is not a duplicate. Refs paperclipai/paperclip#18137, which records that this directory has no check. **What happened?** No tool checked the types of 1,074 of the 1,101 files in `server/src/__tests__`. I searched `package.json`, the workflows, and all tsconfig files. 27 files are in the program of another tsconfig, and these 27 have 0 errors. 11 of the 27 are in `helpers/maintenance/tsconfig.json`. No script ran that project when I measured this. paperclipai/paperclip#18738 now runs it. **Expected behavior** The server typecheck checks these files, and a type error fails the check. **Steps to reproduce** 1. On the base commit, run `tsc -p server/src/__tests__/tsconfig.json --noEmit`. It reports 6,536 errors and needs 12.3 GiB. 2. Add a type error to any file in `server/src/__tests__`. 3. Run `pnpm --filter @paperclipai/server typecheck`. It passes. **Paperclip version or commit** I measured the error breakdown below on commit `8dfd632452`, with TypeScript 7.0.2. The branch now includes master up to paperclipai/paperclip#18865. **Deployment mode** Not applicable. This issue applies to development and CI. No production source file in the server changes. ### Error breakdown Each row adds one change to the row above. | Step | Total errors | Change | Cause | | --- | --- | --- | --- | | Config as committed | 6,536 | | | | Add `src/types/*.d.ts` | 4,447 | -2,089 | Config. The Express request types were missing. | | Set `rootDir` to the repository root | 2,436 | -2,011 | Config. | | Set `allowImportingTsExtensions` | 2,181 | -255 | Config. Vitest accepts `.ts` imports. | | Add `ES2024.Promise` to `lib` | 2,138 | -43 | Config. Node 24 has `Promise.withResolvers`. | | Set `allowJs` for `.mjs` imports | 2,148 | +10 | Config. It removes 8 `TS7016` errors and shows new real errors. | | Set `declaration` to false | 2,142 | -6 | Config. Tests emit nothing. | | Count only files in `server/src` | 2,058 | -84 | Other packages: 53 Daytona, 29 Runner test, 2 UI. | So 4,478 of the 6,536 errors (69%) came from config or scope, and 2,058 are real type errors in 314 files. The 29 Runner test errors are also real, which gives 2,087 errors. Tests that master added after this measurement had 81 more errors, so this pull request fixes 2,168 errors. This table classifies the first 2,058 errors. | Kind of real error | Errors | | --- | --- | | Wrong argument or assignment type | 1,023 | | Stale or partial fixture objects | 525 | | Property that the type does not have | 317 | | Missing narrowing of null or unknown | 101 | | Implicit any | 60 | | Missing name or module | 16 | | Other | 16 | ### What the type errors hid Most errors were stale fixtures and loose mocks. These errors hid real test problems, and this pull request fixes them: - **A test that never ran.** `instrumentation.test.ts` required `sdk-trace-base`, which is not installed. So 3 span tests were skipped. They now run and pass. - **Assertions that could not fail.** - `issue-thread-interactions-service` used a company ID as an issue ID. - `status-card-evaluation-work` compared `[]` with `[]` in a rollback check. - `slack-conversation-lifecycle` read `item.issueId`, which does not exist. - `email-channels` used a permission key that does not exist. - `heartbeat-project-env` imported `LOW_TRUST_REVIEW_PRESET`, which is not exported. The value was undefined. - **Postgres processes that did not stop.** 11 test files called `stop` on the test database helper. That method does not exist, and the helper method is `cleanup`. - **Arguments that the service ignored.** - `methodKey` is now `connectionMethodKey` in the Daytona, Unikraft, 1Password, and Railway tests. - 108 calls passed `"owner-user"` to a parameter that expects an options object. - 9 chat wizard fixtures passed `managementContext` as `source`, so they ran `existing_app` and not `manifest`. - 2 tests passed a `{ userId }` actor that the function ignores. - Inserts into `environments` passed a `companyId` column that the table does not have. - **Problems in tests that master added during this work.** - `sandbox-creation-operations` read `database.connectionString` when an external test database was set. In that mode `database` is undefined, so the fixture would stop with a TypeError. It now uses the external URL. - `idle-completed-history` set a Dot operation to `dispatched`. That is a Muse status, and the Dot type does not have it. The fixture now uses `admitted`, which is a real in-flight Dot status. ### Possible product problems, not changed These may be real problems in production code or types. I did not change them, because this pull request does not change production behavior: - `decisions-service` `service.create` does not validate the lenient `cancel_issue_tree` input. - `createConnection` ignores `input.connectionPurpose`. - The `createConnection` route schema default stores `{}` for `transportConfig`. - `prepareManagedAiRuntime` accepted the invalid value `mode: "connection"`. - `createBinding` accepts invalid `projectionClass` strings outside class 3. - `ensureManagedProjectWorkspace` forwards an untyped `repoRef` by spread. - `routineVariableSchema.label` is optional, but `RoutineVariable.label` is required. - `ToolRunContext.projectId` may be too strict. - `withdrawInteraction` reads `issue.status`, which its type omits. - `issueService.list` and `reconcileNativeFinalizations` omit fields that they return at runtime. - `createGitHubExternalObjectProvider` handles `tokenProvider: null`, but its type forbids this value. ## What Changed - Fixed the 2,168 type errors. The diff changes 322 test and helper files. - Corrected `server/src/__tests__/tsconfig.json`. Editors and the typecheck now read the same options and report the same errors. - Added three `test-tree` layers to `server/scripts/typecheck-layers.mjs`. They read the server layers as declarations, and any error fails them. A hash of the path selects the shard, so a new file does not move other files. - Added `server/scripts/typecheck-stubs/daytona-sdk.d.ts`. Server tests reach Daytona plugin source, and the root install does not have `@daytonaio/sdk`. - Changed two imports in `ui/src/lib/issue-execution-policy.ts` to use `.js` extensions, because server tests import this file. - Made `declarationsConfig` and `--declarations-only` skip the `test-tree` layers, because these layers emit nothing. Other projects read only the source layers. Added a test for this. - Updated `doc/DEVELOPING.md`. ## Verification Run these commands: ```sh pnpm --filter @paperclipai/server typecheck pnpm exec vitest run server/scripts/typecheck-layers.test.mjs ``` Results on macOS (arm64), Node 24, 16 cores. Each check ran after the merge of paperclipai/paperclip#18738 unless the row says otherwise: | Check | Commit | Result | | --- | --- | --- | | Server typecheck, `PAPERCLIP_TYPECHECK_LOW_MEMORY=1`, all 7 layers | `6e45a08573` | Exit 0, 0 errors, 62 seconds, peak 3.52 GiB | | `node scripts/typecheck-test-projects.mjs --low-memory` | `a3306bd13e` | Exit 0, 0 errors, 42 seconds, peak 3.25 GiB | | The same script with `--reuse-declarations` | `1b878aabc0` | Exit 0, 0 errors | | `typecheck-layers.test.mjs` and `live-recovery-work.test.ts` | `01db5634e4` | 90 passed. These include real compiler runs for a pass, a type error, and an invalid tsconfig. | | `tsconfig.declarations.json` after a full server typecheck | `0fb1e4fb0a` | It references only `below`, `cycle`, and `above`. No generated config names a `tsconfig.layers.json` file. | | Owned runtime test in `idle-completed-history` | `e8f2f448ac` | Passed, with the fix that master later replaced | | `idle-incoming-wake`, `plugin-idle-manager`, and the changed `chat-channels` test | `6e45a08573` | 30 passed | | Runner `native-execution.test.ts` | `eab4dcc1f4` | 114 passed | | The 19 server test files that changed since `eab4dcc1f4`, and `typecheck-layers.test.mjs` | `259737b700` | 4,275 of 4,277 passed. The 2 failures are in `chat-channels` and are in the list of 7 below. | The peak is the largest single process, from `/usr/bin/time -l`. Peaks for the server typecheck were between 3.42 GiB and 3.56 GiB across 8 runs. Peaks for the test projects were 3.25 GiB and 3.37 GiB across 2 runs. I did not measure a cgroup. **Runtime check of the fixes.** Types do not prove that a test still works. So I ran all 309 changed server test files, 11,929 tests, before the rebase onto paperclipai/paperclip#18737: | Result | Tests | Detail | | --- | --- | --- | | Passed | 11,913 | Includes the 2 tests and the 54 `dot-runner` tests that passed in a later run, as described below. | | Failed | 7 | The same 7 tests also fail on the commit before the fixes. | | Skipped | 9 | These tests have their own skip conditions: `sentry` 5, and 1 each in `bundled-plugin-registration`, `customer-success-cloud`, `onepassword-connection`, and `workspace-runtime-https-live-exercise`. | The 7 tests that fail with and without this change, on my machine: - 3 tests in `chat-channels` and `email-channels` stop with `ENOENT` on a `SKILL.md` file. The path is in the parent checkout, above the git worktree that I used, so I expect that the worktree is the cause. I did not prove this. - `workspace-instance-cleanup` compares `/private/var/...` with `/var/...`. This is a macOS symlink difference. - 2 tests in `git-reproducible-managed-checkouts` get `unknown_base_git_files` and expect `unpushed_managed_checkout`. I did not find the cause. - 1 more test in `chat-channels`, `materializes direct external-chat finals once`, gets `undefined`. I did not find the cause. CI is the check for these 7 tests. `ci / verify` passed on `eab4dcc1f4`. Failures during this work: - My fix to `agent-runtime-connector-access` changed an invalid permission key in the setup, but not in the assertion. The runtime check found this. I corrected the assertion, and the file passes. - `heartbeat-plugin-environment` failed one wait in the full run. It passed three times alone, so I think that load caused the failure. It can fail again under load. - `dot-runner` skipped all 54 tests in the full run because `cargo` was not on my `PATH`. It then passed 54 of 54 with the toolchain on the `PATH`. - My first `lib` value removed the default DOM types, and the error count went up by 375. I added the DOM libraries. - My Daytona stub with bare `any` left 48 errors, because generic helpers then inferred `unknown`. I added shapes that return promises. - One fix to `agent-secrets-routes` used `class_3_static_lease`, and the test failed. That class allows only specific paths. I reverted to `unclassified`. - `ci / General tests (server (8/12))` failed on `259737b700` because of the flaky port in `idle-completed-history`. paperclipai/paperclip#18853 fixed this test on master, so this pull request keeps that fix. **Audit of the diff.** I counted these patterns in added and removed lines, to find fixes that only hide an error: | Pattern | Added | Removed | | --- | --- | --- | | `@ts-ignore` | 0 | 0 | | `@ts-expect-error` | 0 | 1 | | `.skip` | 0 | 0 | | `as any` | 9 | 9 | | `: any` | 11 | 2 | | `as never` | 8 | 2 | | `as unknown as` | 35 | 4 | | `Promise<unknown>` | 16 | 5 | | `expect(` | 128 | 113 | No hunk removes more `expect(` calls than it adds. Three removals of test content were intentional, and I checked each one: - `muse-control` removed `signalToken`. Line 1012 already asserts that it is absent. - `merged-pr-confirmation-sweep` removed legacy aliases. Other tests cover them. - `heartbeat-workspace-branch-containment` removed a `finalize` branch that no test row can reach. Some partial fixtures stay unchecked through `as unknown as` and `Promise<unknown>`, mostly in `issue-agent-mutation-ownership-routes` and `issue-comment-reopen-routes`. **Accuracy of the check.** One program over the tree and the three shards reported the same errors at the same positions. Memory, as the peak of the largest process on one thread, was 3.93 GiB for one program and 2.43 GiB for the largest shard. I measured these two values before the fixes. Not run or not proven: - I did not run the full `pnpm test:run`, or a root `pnpm build`. I expect CI to run the full suites. - I did not run the 4 GiB container check, or the full `scripts/typecheck-low-memory.mjs`. The peaks above come from macOS, so they are not a Linux cgroup measurement. - I did not run the UI typecheck after the merges. - `workspace-runtime-https-live-exercise` is skipped, so its `cleanup` fix did not run. ## Risks - **A large diff in test files.** 328 files change. A pull request that edits the same tests can get merge conflicts. New tests on master can also add type errors before this merges. Any such error fails this check and needs a fix here. - **A higher cost for changes to source types.** A change to a type can add errors in test files that the author did not touch. The author must then fix them. This is the purpose of the check, but it is new work. - **Changed test behavior.** Some tests now run paths that they did not run before, such as the 9 chat wizard fixtures and the 3 span tests. They pass here, but they can be more sensitive to timing or to the environment. - **The Daytona stub can go stale.** The stub is written by hand. If the plugin uses a new SDK member, the typecheck reports an error until someone updates the stub. - **The release path.** `release-verify.yml` and `scripts/release.sh` run `pnpm -r typecheck`, so they now run this check. - **The tsconfig change.** The test transform reads `server/src/__tests__/tsconfig.json`. The options that I changed are for type checks, and the runs above passed. - **A gap that remains.** Tsconfigs in `packages/paperclip-runner` and in most sandbox provider plugins exclude `*.test.ts`. This check does not read most of those test files, so they can still drift. That work is separate from this pull request. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with reasoning, tool use, code execution, and subagents. The runtime does not show the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (7 tests fail locally, with and without this change. See Verification.) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
dcc0878b32 |
perf(agents): save healthy account settings without a runtime probe (#18690)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The agent runtime page saves account, model, and environment settings. > - The save handler starts a computer and runs a live account test before it writes the edit. > - A slow computer or provider can make a healthy account save take tens of seconds. > - Saved account health, access, installation, and compatibility can be checked without a runtime test. > - This pull request uses those checks for ordinary configuration saves. > - Legacy adoption, task adoption, and Copilot model checks keep live verification. > - The benefit is a fast save while Test and execution retain runtime checks. ## Linked Issues or Issue Description **What happened?** Saving a new AI account selection on an existing agent can take 20–30 seconds. Model edits can take the same path. The configuration PATCH acquires a computer, prepares account credentials, runs a provider/runtime probe, and releases the computer before it saves. **Expected behavior** A healthy installed account can be saved after current access and compatibility checks. A save confirms persistence. Test verifies the runtime. An unverified legacy account must still pass live adoption checks. **Steps to reproduce** Run the PostgreSQL regression with a 20-second controlled delay at computer allocation. On the original route, a successful save takes 20,191 ms and calls allocation and the runtime probe once. With the first patch, the same fixture takes 51 ms; with the environment-guard correction, it takes 37 ms and calls neither operation. Each timing has one sample. This is a local fault experiment with real HTTP and PostgreSQL. It is not a hosted latency measurement. Related: #17760 addresses binding comparison and prospective probe environment resolution. This change deliberately removes the live probe from ordinary healthy-account saves. It keeps live checks for legacy and task adoption. #17965 fixes the result of an explicit managed Runner test. Neither change removes this save-time wait. #18687 optimizes a different Apps configuration read. ## What Changed - Replace the internal probe boolean with explicit none, legacy, and always modes. - Use legacy mode for ordinary configuration saves. Keep current account health, access, installation, compatibility, environment, audit, and revision checks. - Keep the environment company check before the live-probe branch. Keep the live task-adoption transaction and Copilot model checks. - Add twelve real PostgreSQL save-route regressions, including company-bound environment checks. Keep the existing inherited-sandbox adoption fixture for legacy and task adoption. - Cover healthy and legacy settings changes separately. Keep provider rejection and reconnect tests on the legacy save path. - Update the specification and feature/access maps to distinguish saving from runtime verification. The configuration route owns this policy. The account service still owns account access and health. The existing environment test service still owns live probes. No new endpoint, migration, dependency, background job, or provider protocol is added. ## Verification Base: `3f8ebfff8aa9febd0733c6f65b07485dcadfd0e2`. - Original route with initial regressions: three failures and two passes. Healthy saves reached computer allocation. The controlled 20-second baseline saved successfully but failed the no-allocation assertion. - Initial save/adoption/Copilot selection: 21 passed. The name filter excluded 141 unrelated tests. The seven new cases cover persistence, revision and activity records, model edits, roster edits and stale retries, missing installation, revoked grants, legacy verification, and controlled allocation latency. - Explicit Test-route suite: 55 passed. The first attempt reached the 30-second import hook limit. The same suite passed with a 90-second hook limit. - Saved harness preflight suite: six passed. Account revocation, compatibility, probe failure and target release checks remain. - `node scripts/check-maps.mjs` and `git diff --check`: passed. - Review revision: both full save/account suites passed, 96 tests total. The controlled save took 37.08 ms with zero computer allocations and zero provider probes. The five new environment checks passed after the three foreign-environment cases failed against the first patch. - Local full `pnpm -r typecheck`: unavailable because cargo is missing. - Local full `pnpm build`: unavailable at the verified Copilot runtime prerequisite. Server typecheck preparation also reaches missing cargo. - Local full `pnpm test:run`: stopped after concurrent broad checks exceeded the container memory limit. No full-suite pass is claimed. Direct server TypeScript checking and its bounded retry were also killed by the container memory limit. The first remote Typecheck + Release Registry check passed; the next head still requires fresh CI. Reproduce the controlled timing: ```sh cd server PAPERCLIP_CONFIG_SAVE_PROFILE_DELAY_MS=20000 pnpm exec vitest run src/__tests__/agent-configuration-save.test.ts --hookTimeout 90000 ``` The fixture keeps real Express, authorization, account selection, PostgreSQL persistence, revisions and audit writes. Only remote allocation and the provider probe are controlled. The default tests do not contact a provider. Successful configuration saves no longer depend on either controlled operation. A later explicit Test or task must still verify the actual runtime. Review follow-up: the first Greptile review found that the new healthy-account path skipped the environment company guard. The corrected route checks the effective environment before it decides whether to probe. Five real PostgreSQL cases cover foreign, same-company and global environments, inherited defaults, and an existing account binding. Against the first patch, the three foreign-environment cases returned 200 and failed their expected 403 assertions; the two allowed cases passed. Initial fixture attempts hit unique environment constraints; the fixture now uses distinct sandbox rows and names. The first CI server shard found 11 tests that required a live probe for every save. Those expectations are changed to the documented healthy/legacy split, with additional healthy-selection cases. The build and Runner jobs were cancelled after this test failure. Their logs show no separate compile or test verdict. No cancelled check is counted as a pass. Latest commit `6257e9e67f289e1c650a8b4c2b818fcca37e2800`: [CI run 41542980091](https://github.com/paperclipai/paperclip/actions/runs/41542980091) passed typecheck, build, all test shards, Runner checks, and E2E. All 55 check runs are complete: 52 passed and 3 were skipped by their workflow conditions. The commit status check also passed. [Greptile review](https://github.com/paperclipai/paperclip/pull/18690#issuecomment-6590555746) is 5/5 for this commit with no new findings. The earlier environment ownership thread is resolved. These remote results complete the checks that the local tool and memory limits prevented. The earlier local failures remain recorded above. ## Risks A healthy saved account can become invalid at the provider before the next runtime use. Ordinary saves no longer detect that through a live test. They still reject unhealthy, revoked, incompatible, or uninstalled accounts in current storage. Explicit Test and execution retain provider authentication and project-authentication checks. Legacy-account adoption, task connection adoption, and Copilot account-specific model checks keep their existing live validation. There is no migration. Reverting the route change restores the earlier synchronous probe behavior. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. This runtime does not expose the exact model ID, context-window size, or reasoning setting. Those values are not inferred. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites; full-suite limits are above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
049c1eacf5 |
Preserve historical native logs during ephemeral enrollment (#18864)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Ephemeral web servers need retained run history in shared storage.
> - Some old native runs have surviving local logs but no completion
digest or verifiable native source coverage.
> - Enrollment must preserve those bytes without calling them a complete
native transcript.
> - This pull request adds a read-only archive path to offline
enrollment.
> - Users can still read old output after the original server filesystem
is removed.
## Linked Issues or Issue Description
**What happened?**
Offline enrollment blocks on interrupted runs and unfinalized historical
native logs. The current importer requires canonical native-event
coverage even when the original local file survives.
**Expected behavior**
Preserve the exact surviving file in shared storage. Keep the original
run status, completion metadata, and source events. Do not allow writers
or native repair to modify archived history.
**Steps to reproduce**
Use an S3-configured instance with a terminal native run whose local log
lacks a final size or digest. Stop all app deployments and run ephemeral
maintenance. Before this change, interrupted status or unproven native
coverage prevents enrollment.
Related preservation work: #18826, #18835, #18841. This is a bounded
migration fix, not a new live logging protocol.
## What Changed
- The run-evidence migration owner can archive unfinalized native local
files during offline enrollment.
- Require the offline migration environment, zero active or queued runs
across the instance, and expired controller/finalizer leases. Check
global quiescence at each page boundary. Recheck each run's ownership
and metadata before publication.
- Verify stable local file identity and metadata, any recorded
size/digest, uploaded prefixes, and remote SHA-256.
- Store raw history in a separate `<run>.archive` journal with source
cursor zero and backend `s3_archive`. Leave completion fields and native
events unchanged.
- The existing log store reads archives but rejects begin, append,
enqueue, flush, and finalize. Native source repair does not run for
archives.
- Keep the default background importer's coverage checks. Continue to
reject incomplete mirrors and corrupt files.
- Add real PostgreSQL/S3 and packaged maintenance coverage. Correct a
pre-existing large-chunk fixture: the current verifier can prove exact
compactor output, so the negative case now uses a different canonical
source.
## Verification
- Passed on final head
|
||
|
DottaandPaperclip
|
683fa4725e |
fix(evals): refresh native runtime and pin reviewed catalog repairs (#18780)
## Thinking Path > - Paperclip lets people inspect and manage their agents’ work. > - The timeline must report canonical ownership history, and OpenCode tool failures need bounded diagnostics. > - The measured candidate also had independently reproduced question-navigation and oracle defects. > - This change promotes the reviewed production fixes and pins one immutable, qualified source composition. > - Original queries, model, counts, clocks, privacy controls and qualification holds remain intact. ## Linked Issues or Issue Description Refs #18775, #18777, #18779, #18711, #18713, #18718, #18699, #18798, #18799. **What happened?** The measured candidate predates merged native-runtime fixes and contains fixture defects. The timeline omitted recorded reassignment/removal history. Finish-input errors lacked a closed correlation and shape witness. **Expected behavior:** project recorded ownership transitions without inventing facts, distinguish successful native execution from failed verification steps, and retain content-free finish diagnostics. **Steps to reproduce:** repeat an unchanged assignee while editing another field; then reassign or unassign it. Only recorded transitions should appear. Invalid OpenCode finish input should produce a correlated rejection witness without values or schema text. Related #18699 adds gated company OTLP diagnostics. This change uses the existing local callback; it adds no exporter or capture policy. ## What Changed - Promote canonical timeline creation, reassignment and removal projection from #18777. Initial child ownership requires a recorded fresh-creation receipt; delegation requires a readable recorded parent. Unknown legacy facts fail closed. User actors/owners require active company membership; agent actors/targets must exist in that company. Spans, events and edges share that admission filter. Remove the introduced all-history absence query and use guarded indexed UUID lookups. - Add the shared/OpenAPI `unassigned` event kind, preserving newer main contracts and issue-route/service code through reviewed semantic joins. Existing UI consumers remain generic; no new visual marker is claimed. - Add bounded OpenCode finish diagnostics: schema digest, hashed call identity, closed shape enums and validation category. A throwing sink does not change admission or cancellation. - Pin immutable candidate `fd5e3b1e5a379524c0d1651ed8273e75ffd86541`, composed from combined #18777 source `b734393eff0d7ef85b7ad8abd556d2a66ca3a4e6` plus the reviewed MCP diagnostic delta. It preserves actually merged #18775 `97c07d149b5670088faf15317dcf603c75e2af87` and #18779 `a352c057d475067b4fe67375f06785287dfbe0ce`, and merged runtime fixes #18711, #18713 and #18718. - Qualify narrative handling for adjacent repair/verification sentences, intervening native-failure wording, and negated repair objects. Questions, denials and hypotheticals retain their controls. Preserve Previous question beside the qualified Entity type import; separately restore the already-called `internalResultLink` goal helper binding. That import is a runtime fixture repair, not a type-only change. - Include the narrow older-baseline CI typecheck closure and causal heartbeat test observers in the candidate. All four default typecheck projects execute. Main already includes the protected upstream tooling and newer heartbeat fixture, so neither is overwritten with an older candidate file. - Preserve all 244 identities, original queries, model, native/human counts and clocks, including the explicit prior Studio accounting exception. The full catalog stays on its candidate branch; the 196/244 gate and historical SDK runtime holds remain. ## Verification The primary’s f478 retry repeated the real composer-hidden five-second assertion (1 failure,18 passes,1 skip). Investigation established a setup readiness gap: an outer route response can delay mounting the chat thread beyond the assertion. The successor adds the existing real-thread `aria-busy="false"` readiness barrier before the unchanged composer assertion. The original120-second scenario and digest, unread, source-link, history, and reply checks are unchanged. The full owning browser scenario passed naturally in10.2s and with its first real chat response held6000ms in12.0s. This synchronizes fixture setup; the underlying AWS late-mount latency remains unproved. **Current main gates passed:** exact head `823dab49fcd8470140c178771e5c0c6b78725e97` has [successful normal CI](https://github.com/paperclipai/paperclip/actions/runs/41724620555), including both required aggregates, fresh Greptile 5/5 and zero unresolved review threads. Paperclip Review is neutral with a completed substantive 5/5 and no actionable defect. The current optional Superagent Security Scan is neutral/inconclusive: Security Graph context retrieval returned HTTP503, so its assessment did not start. That result supplies no security PASS credit. The primary source PR retains its separate current-head gate before Root sequences normal protected merges. - Reviewed MCP controls: 130 on main and 129 on the older candidate. Meaningful parent comparisons failed before implementation/correlation correction. Controls cover actual offered schema identity, null/omitted admission, bounded rejection categories and throwing-sink cancellation. - Exact unchanged production postimages retain 63 Postgres/public-route controls, nine plan controls and the focused index-plan proof. Owner OpenAPI passes 32 controls; main’s newer contract passes 33. Earlier setup/sandbox failures remain recorded; unchanged database controls were not replayed. - Narrative owner controls pass 216, retaining all prior 207 controls. The independent peer passes 144 balanced cases; earlier 108 and 39-case matrices remain retained. Actual consumer parent failures reproduced the review findings before repair. - Main’s server NoEmit passed on its protected baseline; only two pin files changed afterward. The candidate’s prior full server and four-project checks passed. All server/package/script/root-lock inputs remain byte-identical; the changed Runner project’s guard passes on the current source. Compatibility driver controls pass 47 and six owning suites pass 82. The one explicitly reviewed Docker-only unit omission concerns an unimported image-build feature; all four intended project guards still execute. - Pin/adapter/contract controls pass 47. All 242 expanded declarations, the two legacy dispatch rows, and all 244 singleton private policies retain their authenticated identities. Descriptor, launcher and private-protocol bytes remain exact; declaration loading invoked zero fixture callbacks. - All 506 SDK input blobs match the meaningfully compiled `6abe5739` source vector. Prior real shared/SDK builds measured declaration/map changes and zero runtime-JavaScript changes; a compiled consumer accepts the new kind and rejects an unknown kind. The reviewed registry remains unchanged. Default source proof, 51 SDK controls, 20 custody controls and 16 UI consumer controls remain qualified; runtime/compiler/parser/Zod/canary pins and held runtime grades are unchanged. - The full Mac heartbeat run on prior observer source retained **58 passes / six failures in 802.78s**. Two nonassignee controls subsequently pass focused checks. Four late timeout causes remain unresolved locally; no flakiness/infrastructure cause or complete local pass is claimed. [Actual AWS CI](https://github.com/paperclipai/paperclip/actions/runs/41712023737/job/126182620050) ran this exact heartbeat file with **64 passes in 107.209s**. Its server/package inputs are unchanged in the current narrative-only successor. The source-identical f478 heartbeat file also passed all64 AWS tests in105.812s; source inputs remain unchanged. Current main ordinary CI passed; the primary source PR must independently close its latest-head ordinary CI gate. - Historical main [run 41713049555](https://github.com/paperclipai/paperclip/actions/runs/41713049555) recorded six synchronized runner-shutdown failures and cancellations; the shutdown cause is unknown. A separate chat passive-control assertion observed one `db.transaction` call where zero was expected. Finite local probes passed 2+2 but did not establish a cause or late stack. No external publication, cleanup cause, source fix or flakiness is inferred. The primary’s separate composer-visibility CI failure also remains an inconclusive historical result; no guessed fix is credited. No old-head retry was used. - Earlier UI/inbox fixture failures and the CLI startup timeout remain historical. UI/inbox repairs are now upstream in #18799/#18798 and main preserves those files verbatim. The CLI timeout cause remains unknown. Broader local repository/Rust qualification is not claimed; fresh hosted CI supplies broader coverage. - All four actual repair merges, an exact merged-tree/diagnostic source join, and fresh controller/source/private-policy/active-run guards precede the ten-case TestOps measurement. No paid or model eval was launched for this PR; the original three passes/seven failures remain unchanged. ## Risks The next ten-case model result is unproved. Earlier MCP timeout and continuation-shape causes remain unknown; diagnostics provide future evidence without relaxing validation or rewriting grades. Prior local/CI failures are retained, including the unresolved chat assertion and local heartbeat timeouts. Initial ownership requires canonical in-window audit facts. Missing legacy facts remain unknown. The additive `unassigned` contract is synchronized across shared/OpenAPI/server consumers, with no new UI marker. SDK runtime holds and the broader 196/244 qualification gate remain. Normal protected merges require fresh checks and resolved feedback; no CODEOWNER or branch-protection bypass is authorized here. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: repository analysis, code changes, source validation, and local test execution. The context-window size is not exposed by this harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — current focused/type/pin controls pass; the prior full Mac heartbeat failures remain disclosed and receive no local PASS credit - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — current main normal CI and required aggregates passed; optional neutral reviews and separate primary gate are disclosed above - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — exact current main head; full thread pagination verified - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0ab3ce7a03 |
fix(workspaces): preserve legacy recovery across storage migration (#18841)
## Thinking Path > - Paperclip manages durable tasks and agent work through the control plane. > - An ephemeral web server needs existing workspace files in shared storage. > - Failed historical runs can still own recoverable sandbox files or unfinished coordinator state. > - Moving local files must preserve those obligations without replaying the old agent turn. > - Migration journals the exact source and fences old copyback; the recovery owner saves verified sandbox files separately. > - New user input can continue only after matching preservation receipts and independent execution-stop checks. > - Accepted results retain their finalization owner and must have its durable workspace receipt before storage moves. ## Linked Issues or Issue Description Builds on merged #18826 and #18835. **What happened?** Existing workspaces with failed native copyback could not migrate to remote storage. Simply changing their storage authority would let later recovery target a retired local directory, lose child-task attachments, or leave a recovered task unable to accept a new turn. **Expected behavior** Preserve local files, exact sandbox provenance, task bindings, checkpoints, and unresolved history. Recover surviving sandbox files into immutable task-scoped attachments after migration, and let a new authorized user turn use those files without replaying the old execution. **Steps to reproduce** Create a terminal native run with a prepared copyback descriptor and a retained sandbox. Migrate its local workspace, remove the old host filesystem, recover the sandbox, and submit a new user turn. Also exercise a child task reusing its ancestor's workspace and an expired observed coordinator. ## What Changed - Journal exact legacy run, task, descriptor, and unfinished coordinator bindings during offline migration. Descriptor-free runs retain the complete declared lease/allocation/directory identities; publication locks and revalidates them. - Publish a run-scoped fence that prevents automatic restart or copyback into the retired local authority. - Release only the exact journaled expired coordinator lease under the offline deployment fence and locked revalidation, recording the prior owner in recovery history. Preserve phase, checkpoint, result, and run status. - Save recovered sandbox files through the existing immutable snapshot owner, retaining allocation and provider-stop checks. - Attach recovered files using the current task's identity when a child reuses an ancestor workspace. - Accept matching migration and snapshot receipts for workspace-repair admission and explicit new-user continuation. Changed coordinator bindings and invalid receipts remain blocked. - Require the finalizer's successful workspace_finalize operation before moving a workspace whose accepted result has finalized copyback; the existing owner can then finish assessment/status effects. - Document unresolved workspace-policy, task-ownership, and missing-provenance cases as separate owner actions that block detachment. ## Verification - 569 tests across migration, retained recovery, and explicit continuation passed on |
||
|
DottaandPaperclip
|
451d06e2fc |
fix: resume interrupted Runner preparation and confirm failure report delivery (#18853)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns run recovery and task continuation after a restart. > - A shutdown can interrupt Runner preparation before any provider work starts. > - Workspace recovery can save the files but leave that task behind a legacy execution hold. > - Retry then accepts a request without starting a successor. Startup also leaves the task blocked. > - This pull request verifies the interrupted preparation, preserves its files, and admits one bounded continuation. > - It also keeps a dropped Sentry capture eligible for a later report. ## Linked Issues or Issue Description **What happened?** A hosted task remained blocked after a graceful server shutdown interrupted Runner preparation. Its remote workspace was recovered. Retry returned an accepted request, but no successor started. Startup did not resume it. The task also had no matching Sentry event. The recovery cause is verified. The exact cause of the missing historical Sentry event is unknown. A separate regression with the real Sentry SDK proves that processing backpressure can drop a capture while the task cache marks it as reported forever. **Expected behavior** Retry must continue eligible interrupted work from its saved workspace. Startup must schedule one bounded retry when durable evidence proves that no provider started. A later task read must be able to retry an unconfirmed Sentry report. **Steps to reproduce** 1. Interrupt Runner preparation during a server shutdown before native runtime selection. 2. Retain and recover its remote workspace under the legacy recovery hold. 3. Press Retry, or restart after file recovery has completed. 4. Observe that the task remains blocked without a successor. For the reporting case, hold the first Sentry event in its asynchronous enrichment step. Fill the SDK processing buffer. Capture a second task failure, release the first event, and read the second failed task again later. **Paperclip version or commit** Reproduced against source commit `4d31588dea`. The hosted incident served `6ef37398c8d799c5142a3a6590ecc70ad352b663`. **Deployment mode** Hosted Linux server with remote workspaces and external PostgreSQL. Local regressions use the source checkout on macOS arm64, Node 26.4.0, PostgreSQL 17.10, and the audited optional Sentry SDK 10.71.0. Related public changes: #18541 added task-state failure reports. #18678 retained remote exports before Retry. #18803 fenced remote workspace writers before continuation. The small backend-port fixture correction follows #18739. It is included here because the same existing setup defect blocked CI. ## What Changed - Verify pre-native shutdown interruption against the immutable adapter claim, controller ownership, run events, native records, stopped leases, and exact workspace recovery receipts. - Retire only the server's unchanged bootstrap failure block. Admit explicit Retry through the existing continuation path. - Discover completed workspace recovery at startup and schedule one retry under the existing failure budget. Let saved input own its continuation. - Apply the existing safe replacement dispatch checks to preparation retries. A later human block, competing execution, dependency, pause, approval, or budget gate must still prevent dispatch. - Keep execution evidence available through the bounded run projection. - Confirm Sentry task-report delivery from a successful SDK HTTP response. Keep unconfirmed captures pending for 60 seconds before a later task read can retry them. Both caches remain bounded and start no timer. - Document recovery and report delivery semantics. - Align existing handoff/budget assertions with the guarded preparation retry reason. Keep the owned-runtime fixture outside reserved proxy ports, with bounded allocation attempts. ## Verification - Red before the recovery fix: all three new Retry/startup cases failed because no successor was created. Green after the fix: all three passed with real saved workspace archives and PostgreSQL. - The pre-merge revision passes all 384 tests in the six affected recovery, run-control, run-state, retry, and dispatch suites. - The late-block regression failed before its repair. Its four focused recovery/dispatch cases then passed, including final dispatch cancellation. - Review found a second stop check that still rejected verified local and lease-free preparation. Both new cases failed before that repair. All 17 interrupted-preparation regressions passed on the pre-merge revision. - The Sentry regression failed with the real SDK before the fix. All 91 affected reporting tests then passed. The transport stays in memory. No test sends events to Sentry. - Full repository typecheck passed with `node scripts/typecheck-low-memory.mjs`. The full `pnpm build` and module boundary check passed. - CI found untyped delivery-hook parameters when the optional Sentry SDK was absent. Explicit callback types fix that build. The SDK-absent server build and all 91 SDK-present reporting tests pass. - Server typecheck passes with the SDK installed and the real Sentry SDK contract CI job passes. Greptile reviewed the pre-merge commit `103e175c44` at 5/5 with no open findings. - The local full-suite attempt initially overlapped the rebase and stopped at test-line selection. Its unchanged-head rerun passed the first 309-test chat shard. I stopped the old-head run before applying review fixes. Current-head CI owns the remaining full-suite verification. - CI exposed two old retry-reason expectations. The handoff and retry-budget checks pass with the new guarded reason. A separate existing process-adoption fixture chose port 42953 in the reserved 42000–42999 proxy range. The fixture now rejects reserved ports, following #18739. Its real process/adoption/termination test passes locally. - The final Runner package job [initially failed](https://github.com/paperclipai/paperclip/actions/runs/41718507919/job/126205912033) one unchanged Codex credential case after it could not acquire its port quorum, then timed out. The complete unchanged 42-case credential suite passes locally. One unchanged CI retry passed; the original competing port owner was not identified. No Runner implementation or credential test was changed. - Required CI passes on `103e175c4490e1bdf804c64a8635b0e4568545ae`, including `ci / verify` and `ci / e2e`. That revision has Greptile 5/5 with no open findings. The optional Paperclip Review check on that earlier head remained queued, waiting for its assigned agent. - Merged current `master` without replacing published commits. The retry gate retains both shutdown interruption checks and the new workspace preparation timeout policy. All 311 tests in the five combined recovery, retry scheduling, completion, and native startup suites passed after the conflict resolution. Server typecheck passed. Final head `005d380b00d45a2393c4a2a190eab27611168d3c` includes the next conflict-free master update. All 17 interrupted-preparation regressions passed again on that final head. Greptile reviewed that exact head at 5/5 with no new findings. Required CI passes on that exact head, including `ci / verify`, `ci / e2e`, typecheck, build, the full test shards, and the real Sentry SDK isolation check. The optional Paperclip Review remains queued, waiting for its assigned agent; it is not a required branch check. - On the final head, six test jobs were cancelled after their spot runners received shutdown signals at the same time. Another server shard passed 2,419 tests, but its unchanged 103-record cleanup backlog case exceeded the 15-second limit. The unchanged full cleanup suite then passed all 29 tests locally. One unchanged retry of only the seven unsuccessful jobs passed. The [original cancellation and timeout logs](https://github.com/paperclipai/paperclip/actions/runs/41724898038/attempts/1) remain in the workflow history. No cleanup implementation or test was changed. - The signed-in hosted UI reproduced the failed Retry before the fix. The fixed path is covered through the real HTTP Retry route and assembled heartbeat/recovery services. This PR has not been deployed to the hosted incident. A live repaired Retry remains unverified until rollout. ## Risks - Automatic continuation must not repeat provider work. It requires positive preparation evidence and rejects retained provider, process-launch, coordinator, result, live-controller, unsafe lease, or mismatched workspace evidence. - Repeated recovery calls must create one successor. Tests cover concurrent requests, repeated startup visits, pending Retry, retry exhaustion, and changed task ownership. - Sentry delivery state stays local to the process. A restart or cache eviction can report a failure again. An unconfirmed event needs a later task read to retry. - No schema or API contract changes are required. ## Model Used OpenAI GPT-6 through Codex. This session does not expose a more specific deployment ID or context window size. Reasoning, code editing, test execution, and browser diagnostics assisted the change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2920d88034 |
fix: preserve cancelled runs through native finalization (#18857)
## Thinking Path > - Paperclip helps people manage AI agents that do work. > - Native runs can accept a result before workspace file finalization ends. > - A user can Stop the run while file finalization is still pending. > - The direct finalizer replaced that cancelled run with the earlier provider status. > - The restart projection already preserves a cancelled run. > - This pull request preserves cancellation in the direct projection too. > - Saved results and normal task decisions remain available. ## Linked Issues or Issue Description **What happened?** An accepted successful result can wait for workspace finalization. If run-only Stop completes first, the run and its source wake become cancelled. Late finalization then changed the run to succeeded, cleared the cancellation error, and replaced its finish time. The wake remained cancelled. **Expected behavior** Preserve the winning run cancellation, finish time, errors and audited acknowledgement. Keep the accepted provider result and complete file finalization. The server task arbiter still decides task status from the accepted result. A valid result can finish the task even though the execution was stopped. **Steps to reproduce** Accept a native result through the control-plane port. Hold an actual workspace-finalization operation. Stop the run through the heartbeat service with a board request identity. Release the operation and call the direct finalizer. The new database regression exercises this sequence. Related work: #18804 settles exact claimed native wakes. This change preserves a wake that Stop already cancelled. #15212 and #12301 address the opposite race in legacy cancellation. They do not change this native finalizer projection. ## What Changed - Evaluate run status and errors against the current locked database row. Preserve a cancelled run when a late accepted result finishes finalization or when workspace/status-decision retries exhaust. Merge finalization failure metadata into the current row so concurrent receipts survive. - Keep finalization receipts and accepted results. Preserve existing server task-arbiter behavior and accounting fields. - Add real database tests for successful and failed accepted results, delayed workspace work, and Stop after the task decision but before run projection. - Check cancelled wakes, audited acknowledgement, no successor, restart projection and idle recovery. Update execution documentation. ## Verification - Before the production change, the actual producer sequence failed: the cancelled run became succeeded, its error code became null, and its finish time advanced. This failure remains saved privately. - The focused projection, cancellation-request and Stop-accounting suites pass all 67 tests. These include the same delayed-finalization sequence, both task status orderings, and complete and late accounting receipts without another provider turn. - Independent review found a second terminal writer in retry exhaustion. On the unchanged previous head, two real database cases changed a cancelled run to failed or succeeded after workspace or status-decision retries. A third case let actual board Stop win after the permanent workspace failure admission snapshot and still changed cancelled to failed. All three now pass. Existing uncancelled exhaustion controls still pass. No mock supplied the canonical cancellation state. - Initial review found that routine regression notes do not belong in the feature maps. Both map additions were removed. Production and test code are unchanged by that correction. - Runner TypeScript build and all server typecheck layers pass. Map, module-boundary and diff checks pass. - An additional direct typecheck of the normally excluded projection fixture reports two existing notification mocks whose transaction parameter is typed as `object`. The same check with the previous test file reports the same two errors. The new regression cases add no type errors. The standard server typecheck remains green. - The previous head passed required CI after one targeted retry of jobs that received the same runner shutdown signal. Those original failures remain recorded. The exhaustion correction passes fresh required CI on exact head `84404d34b9d982d8bca24b63ae2046051f5169b8` (workflow 41726120483, attempt 1). Both required aggregate gates pass. Greptile reviewed this head at 5/5 with no new comments; all review threads are resolved. An independent source review found no blocking issue in the correction. Runner TypeScript and all server typecheck layers pass locally; the exact production server and Runner build passes with the server stamp bound to `84404d34b9d982d8bca24b63ae2046051f5169b8`. The full required CI workflow passes. - This is local fixture evidence. It does not establish hosted recovery, a remote provider stop, or a deployed version. ## Risks - Accepted result finalization can still finish the task after run-only Stop. This is existing server policy; cancelled run API write authority stays revoked. - The fix protects cancelled execution history. It does not invent provider termination, saved-file or accounting proof. - No schema, endpoint, retry budget, provider call or new background worker changes. ## Model Used OpenAI `gpt-6.1-sol`, Extra High reasoning. The model used code execution, repository tools and test output. The host does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
b0c4e34526 |
fix(native): resume from the owner after an unstarted failed follow-up (#18860)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep one conversation across task turns. > - A follow-up turn resumes from the newest earlier run that holds a provider checkpoint. > - A follow-up can fail during admission, before provider work, and still hold the checkpoint that it inherited. > - A later Retry then resumes from that failed run. Controller recovery finds no controller checkpoint there and fails. > - This pull request makes resume source selection skip a failed follow-up that proves that it never started provider work. > - The benefit is that a board Retry resumes from the last committed owner, and the task can continue. ## Linked Issues or Issue Description Refs #18846, which lets a follow-up start after a recovered per-turn owner. Refs #18854 (same-run retry of failures before provider work) and #18718 (controller projection refresh). **What happened?** Run 1 was a per-turn remote sandbox run. Recovery committed its result, but it saved no Runner suspension receipt. Run 2 was a follow-up that was admitted before #18846. Admission failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`, before any provider work. Run 2 has these properties: - Status `failed`, and the result has `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. - The coordinator is `terminal_failure`, with no result, no lease owner, and no controller process. - No controller checkpoint rows and no controller recovery pointer. - A `sessionCheckpoint` bound to run 2 with a null cursor. The provider session in it is copied from run 1. A board Retry then started run 3. Resume source selection treats any checkpoint as a progress barrier, so it selected run 2. Controller recovery then failed with `native_controller_recovery_unsupported:metadata_format`, because run 2 has no controller recovery pointer. The task stayed blocked. Every later Retry selects the newest failed follow-up and fails the same way. **Expected behavior** A Retry must resume from the last committed owner (run 1) when the newer follow-up provably never started provider work. If the follow-up might have started provider work, or holds its own controller evidence, it must stay the source. **Steps to reproduce** 1. Commit a per-turn remote owner through recovery, without a suspension receipt. 2. Admit a follow-up, and make its admission fail with `prior_owner_terminal_state_indeterminate`. 3. Settle that follow-up as a failed bootstrap: `providerWorkStarted: false`, `terminal_failure`, and no result. 4. Admit a Retry. It selects the failed follow-up and fails with `native_controller_recovery_unsupported:metadata_format`. The new regression test runs these steps against PostgreSQL. **Paperclip version or commit** Reproduced on `909d86b59a`. **Deployment mode** Native Runner with a per-turn lifecycle on the built-in remote sandbox transport. Source selection is shared by all native providers. ## What Changed - `findNativeSessionResumeRun` skips a newest barrier that is an unstarted inherited follow-up and selects again without it. The selector repeats this inside the existing 32-run window. - A run is skipped only when all of these are true: - It is a native run with status `failed` and a `bootstrap` receipt with `providerWorkStarted: false`. - Its coordinator is `terminal_failure`, with no result, lease owner, or controller process. - It has no process identity, provider event, controller checkpoint row, run result row, or controller recovery pointer. - Its checkpoint is bound to this run and session, and it has no cursor. - After the skips, every skipped run must name the same backend, driver, and provider session as the selected source. Otherwise the selector returns the original barrier. - Runtime selection and controller recovery both call this selector. Both now use the committed owner. Admission then reaches the #18846 path, which retires the old authority and starts a new Runner. - Add four cases to `native-recovered-owner-followup.test.ts`, with one negative control for each guard. - Document the rule in `doc/execution-semantics.md` and the test in `doc/testing-native-remote-continuity.md`. ### Why source selection - Source selection made the wrong choice. The failed follow-up has no provider progress. Its checkpoint is a copy, and the provider never saw its turn. - One shared selector feeds runtime selection (checkpoint rebind, previous turn, runner instance) and controller recovery. A fix in only one of them would let them disagree about the source. - Controller recovery keeps its rule: it never searches backward when the selected run is incomplete. The selector skips a run only with positive proof, so an incomplete run with any evidence of work still stops recovery. - A fix in the executor could not help. The executor sees only the source that the selector gives it. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts`: 22 tests pass. - Before the change, the two new retry tests failed with `native_controller_recovery_unsupported:metadata_format`. - After the change, the retry selects the committed owner and is admitted. It is also admitted after two failed follow-ups. - Eight controls keep the failed follow-up as the source: process identity, provider event, controller checkpoint row, a receipt with `providerWorkStarted: true`, a coordinator with a controller process, a run result row, a different provider session, and a non-null cursor. - I disabled each guard in turn. Each change made its matching control fail. - These suites pass: native session resume, resume projection, controller recovery (unit and integration), native startup retry, runtime selection, execution workspace service, native status corpus, task workspace binding, and all 1,088 native executor tests. The container and Runner binary cases skip locally. - Server typecheck, `node scripts/typecheck-low-memory.mjs`, `pnpm check:module-boundaries`, and `git diff --check` pass. - I did not run the full `pnpm test:run`, `pnpm build`, or the opt-in Linux Runner container cases locally. CI runs them. - The tests start at the selector and executor admission. They do not exercise the Retry route, the task retry queue, or a live sandbox provider. ## Risks - A wrong skip could resume from an older checkpoint. Each guard requires positive evidence of no provider work. A skipped run must also name the same provider session as the selected source. If any check fails, the selector returns the original barrier. - Only `failed` runs qualify. Cancelled, interrupted, and timed-out runs keep their existing handling. - The skip adds one proof query per candidate. It runs only when the newest barrier is a failed native run whose checkpoint is bound to itself and has no cursor. - The rebind now uses the owner's tool-contract fingerprint. If the tool contract changed since the owner ran, the Retry starts a new provider session. The provider history in this failure mode is already unavailable. - No migration or configuration change is required. ## Model Used - Anthropic Claude Opus 5.5 (model ID `claude-opus-5-5`) in Claude Code, with extended thinking, repository tools, shell and code execution, and local test runs. The runtime does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a80eec1164 |
fix(recovery): continue Retry after completed file preservation (#18845)
Reuse completed file preservation for task Retry and show bounded admission explanations, including newer saved messages and identifier routes. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f187de99aa |
Preserve domain deadlines across owned idle sleep (#18656)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
0a1e4c3d0f |
fix(workspaces): retry pristine remote setup timeouts (#18852)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task needs a workspace before its model can start. > - Fresh workspace preparation can reach its transport deadline. > - Failure handling records that timeout as a generic setup failure and stops the task. > - An unpublished workspace has no saved task work to repeat. > - This change routes that narrow failure through the existing bounded retry scheduler. > - Saved work, retained executions, and normal admission gates keep their existing protection. ## Linked Issues or Issue Description Refs #18827, which preserves remote timeout diagnostics. This change builds on that merged repair. Related #16727 concerns controller-managed clones and optional partial clones; this change keeps remote clone contents and deadlines unchanged. **What happened?** A fresh remote workspace reached its 300-second command limit before model execution. The task then stopped with `setup_failed` and no scheduled continuation. A separate full clone completed in 137 seconds on a disposable Daytona sandbox. The original delay remains unexplained. This change repairs the missing recovery path; it does not claim to remove the underlying delay. **Expected behavior** A fresh, unpublished workspace preparation timeout can use the existing two automatic retries. Retry scheduling remains durable and company scoped. Stop, pause, dependencies, budgets, task ownership, saved work, and retained native execution must still prevent unsafe work. **Steps to reproduce** 1. Start preparation for a fresh remote workspace with no saved version or published allocation. 2. Make its initial directory/clone command return a transport timeout. 3. Observe that the base code records a generic setup failure without bounded recovery. 4. With this change, the owner validates current workspace authority and preserves a specific bootstrap failure for the normal retry scheduler. **Paperclip version or commit** Base: `e0276e0166`, including #18827. **Deployment mode** Native Paperclip Runner with sandbox workspace storage. Verification uses real PostgreSQL, filesystem commands, and a separate live Daytona transport probe. ## What Changed - Recognize only the initial preparation timeout while the same company, writer, and epoch still own a pristine workspace. - Keep restore, additional repository, provisioning, save, changed ownership, dirty workspace, and retained native execution failures outside this recovery policy. - Persist the typed failure before using the existing retry scheduler, admission gates, and attempt budget. - After successful terminal lease cleanup, reclaim an unpublished bootstrap-only reservation even when a provider returns no stop receipt. Preserve receipt requirements for saved/published work, unknown execution, and native identities; do not synthesize termination evidence. - Prevent generic recovery from granting another budget after exhaustion. Keep a specific recovery notice. - Add regressions for persistence, concurrent scheduling, restart, exhaustion, Stop, pause, dependencies, fresh allocation, lost replies, and saved output after cold restoration. - Document the recovery contract. ## Verification - Red: the real-PostgreSQL workspace regression first failed because the timeout had no recovery classification. The completion regression then failed because it became `setup_failed`. A retained-native negative failed before the admission guard. The review follow-up reproduced a failed successor reservation after production lease release returned void. - Green: all 127 transport/archive/real-PostgreSQL workspace cases pass after the review repair. All 12 receipt/void handoff cases pass against the final query. The 141 complete completion/retry/recovery cases and the fresh-allocation selection case pass. Safety cases cover concurrent scheduling across restart, the two-attempt cap, Stop/pause/dependency gates, native identities, saved/published files, active leases, failed cleanup, and cold-restored output. - `node scripts/typecheck-low-memory.mjs` and `pnpm build` pass at `ba96cff771707415e4a5e893c8b3a97499ead25b`. `git diff --check` and the module-boundary check pass. The full local test wrapper completed all 1,835 chat cases across six partitions, then was interrupted during its serial general-server phase after the complete current-head CI suite passed. Its interrupted run is not a full local-suite pass; its log is retained. The owned process tree and database children exited. - The [complete GitHub Actions run](https://github.com/paperclipai/paperclip/actions/runs/41718108375) is successful at this exact head, including all server/workspace/chat partitions, all nine serialized groups, all eight browser shards, build, typecheck, native Runner, package, Rust, and release-policy checks. The first Rust worker shut down during cache restore before tests; one retry passed without source changes. Expected Storybook skips remain skips. - Earlier failures remain accounted for: the first typecheck found two incomplete process-result fixtures, now corrected. An intermediate completion run exposed a test double that returned no promise and a teardown timeout; the complete 141-case suites subsequently passed. The first-head CI had four worker shutdowns and one worker communication loss. A three-second monitor settlement wait failed in CI and locally, then all 21 cases passed unchanged at base and head. An earlier CI retry found a real-process history adoption assertion failure; that exact case passed locally at base and head. Both server groups pass in final-head CI. These timing causes remain unknown. [Earlier CI evidence](https://github.com/paperclipai/paperclip/actions/runs/41713712522). - Live supporting evidence: unchanged production preparation code completed a full repository clone in 137 seconds using Daytona SDK 0.203.0, Node 24.22.2, Git 2.39.5, and Linux x86_64, with 1 CPU, 1 GiB memory, and 1 GiB disk. The approved diagnostic account could not access the original fleet snapshot, so the probe used public `node:24-bookworm`. Both probe allocations were deleted and absence verified. Full task acceptance at the fixed revision remains unverified: the original fleet snapshot and an explicitly configured disposable Codex subscription fixture were unavailable. No CLI login or hosted subscription was copied. - Greptile is 5/5 at the final head with no actionable findings. The Paperclip review finding is repaired, replied to, and resolved. The separate **Paperclip Review** check remains queued. Its task failed workspace export after publishing the first assessment; the staging service was then observed showing a workspace recovery page after an update failure. Subsequent browser rechecks timed out, including a fresh tab, so the normal recovery control could not be used. Reconcile that existing export after service access returns, then let the saved current-head review request run. That external review is the remaining check; this PR is not yet fully green. - The original hosted task was last verified blocked. This PR has not been deployed and the task has not been retried by this work. Reviewers can run `pnpm test:run`, `node scripts/typecheck-low-memory.mjs`, and `pnpm build`. The remote-workspace integration suite additionally needs an owned disposable PostgreSQL database named `remote_workspace_*test` through `PAPERCLIP_REMOTE_WORKSPACE_TEST_DATABASE_URL`; otherwise that suite is skipped. ## Risks Each retry keeps the existing preparation deadline, so persistent slow clones can consume the bounded budget before intervention. A retry receives fresh allocation admission; it does not replay the command against the old allocation. The workspace owner checks persisted authority before classifying the failure, and normal admission still rechecks that authority. Historical generic setup failures do not gain automatic replay. No database migration, provider-pack update, model fallback, credential change, or deployment is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser tools. The host does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
909d86b59a |
fix(native): retry runs that fail before provider work (#18854)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs start the provider through Runner. `session.open` starts the provider session before any prompt. > - When `session.open` fails, the server records the generic `native_session_interrupted`. Recovery evidence then reports `ambiguous_state`, so the first failure blocks the task with no automatic retry. > - The failure is often temporary. On one instance an agent's provider account hit its usage limit. For about 14 hours every later run failed at `session.open`, and 47 tasks stayed blocked. Outside limit windows, the same error also occurs once and a later retry succeeds. > - The server already has a bounded same-run bootstrap retry for failures that are proven to come before provider work. It did not recognize this failure, and it did not consider a known provider limit. > - This pull request classifies a definite `session.open` failure as a pre-session failure. Bootstrap retries back off and wait out a known provider limit. The task is blocked only after the shared budget is used, with a clear reason. > - The benefit is that a temporary startup failure or a provider limit no longer strands tasks on the first failure, and every existing replay guard stays in force. ## Linked Issues or Issue Description Refs #18847. That pull request keeps the provider cause on `session.open` rejections and names typed categories (for example `classification=provider_rate_limit`). This change does not depend on it. With #18847, a typed limit at `session.open` uses the same cooldown path, because the server reads the cause with `providerFailureFromError`. Refs #18622, which added the `ProviderFailure` cause. Refs #13993, an open change about legacy adapter startup backoff; it does not touch the native recovery path. **What happened?** A native run (`paperclip_runner`) failed at `session.open` before any prompt: `ACPX sidecar command session.open was rejected (retryable=false, classification=session_ensure_failed)`. The server recorded `native_session_interrupted`. Recovery evidence had no checkpoint and no provider event, but the cause was not in the pre-session list, so the mode was `ambiguous_state`. The coordinator went to `terminal_failure`, a board-owned `active_run_watchdog` action was created, and the task was set to `blocked` on the first failure. During a provider usage limit, every later run of the agent failed the same way, so each task was blocked. **Expected behavior** A run that provably failed before provider work retries automatically, with backoff and a bounded budget. If the agent has a known provider limit, the retry waits for the cooldown. The task is blocked only when the budget is used, and the recovery action explains why. A run where provider work may have started is never replayed. **Steps to reproduce** 1. Make Runner reject `session.open` for a native run (for example a provider at its usage limit). 2. Observe the coordinator: `terminal_failure`, `recoveryMode: "ambiguous_state"`, `providerSessionEstablished: false`. The task is `blocked`. 3. The new `native-startup-retry.integration.test.ts` reproduces this with a real database and the executor. **Paperclip version or commit** `master` at `5877d751c0e`. **Deployment mode** Native Runner with an ACPX Claude provider. The fix is in the server and is not specific to one deployment mode. ## What Changed - `nativeSessionFailureSourceCode` returns a new cause, `provider_session_open_rejected`, when Runner reports that `session.open` failed or was rejected. Those outcomes are definite, and the command runs before any prompt. An `indeterminate` outcome stays `native_session_interrupted`. Earlier, more specific causes (for example a missing runtime package) keep their classification. - `nativeProviderRecoveryEvidence` treats the new cause as pre-session. The durable evidence still decides. A bootstrap retry happens only with no provider session, no provider event and no checkpoint. Otherwise the mode stays `ambiguous_state` and the task is blocked as before. - New module `native-runtime/native-startup-retry.ts` plans each bootstrap retry. It is called only for `bootstrap_retry` mode. - Backoff: the existing `infrastructureRetryDelayMs` (about 30 s, then about 60 s, plus or minus 20 %). The jitter derives from the run and attempt, and it is persisted with `nextAttemptAt`. - Limit cooldown: evidence is a `rate_limit` or `quota_exhausted` provider failure on the current failure (the #18847 shape), or on an agent run that started in the last 24 hours. A `native_provider_usage_limit` or `provider_quota` error code also counts. The cooldown is the provider's `retryAt` (or `retryNotBefore`), or one hour after the failure when no reset time is known. A later successful run of the agent clears older evidence. The cooldown is capped at 24 hours. The retry runs after the cooldown plus the backoff delay. - A failed evidence read is logged and falls back to plain backoff, so it cannot stop the failure from settling. - The budget does not change: three provider attempts, shared with checkpoint resumes and replacements. When it is used, the coordinator records `native_session_retry_exhausted`. The task is blocked, and the board-owned action says: "Provider startup failed on 3 attempts before any provider work began. Automatic retries are stopped." When a limit is known, the message includes the expected reset time. - While a retry waits, the coordinator detail, the recovery action evidence and the scheduled wake record `providerCooldownAt`. The next action names the reset time, and the task projection shows "Retry scheduled". - `PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS` (one hour) moves to `system-recovery-state.ts`. The legacy provider-quota recovery default now refers to it. The value does not change. - A chat retry of an exhausted `provider_session_open_rejected` run is accepted like the other pre-provider causes. The existing proofs still apply. - `doc/execution-semantics.md` describes the backoff, the cooldown and the exhausted state. ## Verification - Red: before the fix, the first new integration case failed. The coordinator was `terminal_failure` with `recoveryMode: "ambiguous_state"`, `originalFailureCode: "native_session_interrupted"` and `providerSessionEstablished: false`, and the task was `blocked`. Four more cases failed the same way. The two never-retry cases passed before and after the fix. - Green: `npx vitest run src/services/native-runtime/native-startup-retry.integration.test.ts` in `server` (embedded PostgreSQL, the real executor, and the real resume claim). 8 cases pass: - a rejected `session.open` is retried with backoff (attempt 1 in 24 to 36 s, attempt 2 in 48 to 72 s), and the task stays `in_progress` with an agent-owned action and a `resume_native_run` wake; - after three attempts the task is blocked, the action is board-owned with the clear reason, and the resume claim refuses another attempt; - a recent limit failure holds the retry until one hour after that failure, and the claim refuses it before the cooldown; - a provider reset time from an earlier run, or from the startup failure itself, sets the cooldown; - a distant reset time holds the retry for at most one day; - a later success, another agent's limit, and an expired limit do not delay the retry; - a durable provider event, or an indeterminate `session.open` outcome, still blocks the task with `ambiguous_state` and no retry. - `native-session-executor.test.ts`: all 1,087 tests pass, including a new classification case for `failed`, `rejected`, `indeterminate` and a different command. - `chat-channels.integration.test.ts -t "requires terminal-safe native ownership for an exact chat retry"`: 23 pass, including the new `session_open` case. That case fails without the list change. - Related suites pass: native session resumption, native provider failure, capacity and overload integration tests, provider failure classification, native status arbiter corpus, native restart recovery. - `npx tsc --noEmit -p server` passes. `git diff --check` passes. - [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/41713988964) is green on head `32561aa335d`. This includes `ci / verify`, `ci / e2e`, typecheck, build, all server, chat and workspace groups, all serialized server suites, and all Runner checks, which run the native-runtime server tests. In the first attempt, eleven jobs stopped because their runners received a shutdown signal; no test reported a failure. A rerun of those jobs passed. - A local run of the whole `server/src/services/native-runtime` directory was stopped before completion because it runs serially for a long time. CI ran those files. - Runtime dependency: the new work is one indexed query (`company_id`, `agent_id`, `started_at`) with `LIMIT 20`. It runs only when a bootstrap retry is planned or exhausted, not on the normal path. No new timer or polling: the wait uses the existing persisted `nextAttemptAt` deadline and the existing resume dispatch. - Not verified: a live provider account at its usage limit, and a live remote sandbox that waits through a full cooldown. The full repository suite was not run locally; CI ran it. ## Risks - A task that waits for a cooldown stays `in_progress` with its execution lock, for up to one day. This is the existing same-run retry state. Stop, reassignment, pause and an explicit Retry keep their current behavior through the existing claim checks. - During the wait the failed run keeps its environment lease in the same way as any same-run retry. A remote lease with an unexported prepared workspace is retained by the orphaned-lease sweep until the retry runs. Other active leases of the failed run are released after the usual five minutes, and the retry acquires a new one. - A `rate_limit` failure without a reset time (for example a short HTTP 429) delays only startup-failure retries of that agent, by up to one hour. Normal runs are not delayed, and a later success clears the evidence. - A startup failure that needs user action (for example a sign-in failure) now uses two automatic retries, about 90 seconds in total, before the task is blocked. Before, it was blocked at once with an ambiguous reason. - The agent's other queued runs are not held during a cooldown. Each one fails at `session.open` without provider work and then waits for the same cooldown. An agent-wide admission hold has no clean extension point in the current queue, so it is not part of this change. - Re-arming consumed scheduled monitors is out of scope. This change makes it matter less: a run that consumed a monitor wake now retries itself on the same run. A monitor that a run consumed before its retry budget was used stays consumed. - No schema or API change. The new error code appears on runs and in run error codes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with extended reasoning and tool use (shell, file editing, test execution). The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
80c308f872 |
fix(runner): admit follow-ups after recovered per-turn owners (#18846)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep one conversation across task turns. > - After a per-turn sandbox turn, the controller saves a Runner suspension receipt or a failover backup. > - A full sandbox disk can stop that save after the turn is complete. > - Recovery then commits the result, but the controller has no proof that the Runner suspended. > - Each follow-up rebuilds the old controller journal from the database and rejects it again. > - This pull request lets admission retire that old authority when the prior owner is provably finished. > - The benefit is that the task continues with its saved workspace and does not stay blocked. ## Linked Issues or Issue Description Refs #18705 (checkpoint save on a full sandbox disk). Refs #18829, #18718 and #18343 (related admission and controller recovery). Refs #18811 (a related warm continuation wait). **What happened?** A per-turn native run in a remote sandbox finished its turn. The post-turn `harness_state.backup.persist` and `session.checkpoint.persist` steps failed because the sandbox disk was full. The controller got no Runner suspension receipt and no failover backup. Workspace recovery later saved the files. The run committed its result and ended as `succeeded`. A board comment then started a follow-up in a new sandbox. Admission failed after about two minutes with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. Execution recovery then blocked the task. The cause is in durable-root admission. `restorePerTurnRunnerdController` rebuilds the prior controller journal from the database checkpoint. `verifyPriorRunnerdStateForSessionScope` finds a terminal owner in the same scope with no suspended lifecycle and no retained warm owner. It returns `terminal_state_indeterminate`. Admission quarantines the root and fails. The database checkpoint does not change, so every later admission rebuilds the same journal and fails again. A board Retry after that failed follow-up does not repeat the same error. Runtime selection picks the failed follow-up as the resume source, because it carries an inherited provider checkpoint. Controller recovery then fails with `native_controller_recovery_unsupported:metadata_format`. The task stays blocked. This pull request prevents that failed follow-up. It does not change source selection (see Risks). **Expected behavior** When the prior owner is provably finished, the follow-up must start. It must keep the saved workspace. It must not start while the prior owner can still be alive, while terminal delivery or provider cleanup is unsettled, or while quarantine holds evidence from a different owner. **Steps to reproduce** 1. Complete a per-turn remote sandbox run. Commit its result through recovery and release its controller lease. 2. Keep its database controller checkpoint and recovery pointer. Do not save a suspension receipt or a failover backup. 3. Admit a follow-up in a new sandbox with no local controller projection. 4. Admission rebuilds the old journal, quarantines it, and fails with the error above. 5. Admit the same run again. It rebuilds the journal and fails the same way. The new regression test runs these steps against PostgreSQL. **Paperclip version or commit** Reproduced on `e0276e0166`. **Deployment mode** Native Runner with a per-turn lifecycle, the built-in remote sandbox transport, and the ACPX provider. The admission path is the same for other per-turn sandbox providers. ## What Changed - Add `retireFinishedPerTurnRunnerdOwner` to durable-root admission in `native-session-executor.ts`. It runs only when the existing verifier returns `terminal_state_indeterminate`, and only without a restart claim or a retained warm owner. - The function requires all of these. Otherwise the existing quarantine and rejection apply. - Per-turn lifecycle on the built-in remote sandbox transport, with a `created` or `replacement` sandbox. A resumed sandbox, a warm session, and a plugin task keep their existing paths. - No local Runner lifecycle evidence for the old owner. The session root contains only known authority entries. - A journal for that owner with its terminal delivered and only idle goal snapshots after it. All commands are completed. Each tool input has a completed result. No warm transition or staged attachment exists. - A terminal prior run in the same task. Its coordinator is `committed` with a result, has no lease owner, and has no recovery state. - A stopped prior controller, from `evaluateNativeControllerTakeover` with `requireStoppedController`. A durable controller is fenced by its released database lease, as in remote warm recovery. - Each quarantined root for the scope passes the same journal and local Runner evidence checks for the same owner. A retry rebuilds only the database journal, so quarantined Runner evidence must stay decisive. - When the proof holds, the function reads the journal again, moves the old authority into `continuity-breaks/<time>-<uuid>` in the session root, and logs one line. The follow-up then starts a new Runner authority. The provider session follows the existing resume-or-replace policy. - Share the list of continuity archive entries with provider replacement (`archiveContinuityState`). Provider replacement behavior does not change. - Add `native-recovered-owner-followup.test.ts`. It uses real PostgreSQL, a real encrypted controller checkpoint, and the production source selector and checkpoint rebinder. - Document the rule in `doc/execution-semantics.md` and the test in `doc/testing-native-remote-continuity.md`. ### Design choice I considered two approaches: 1. Start a new Runner authority for the scope at admission (this pull request). 2. Make the recovery path that committed the prior run also settle its Runner and controller projection. I chose the first approach for these reasons: - The provider history stayed in the unreachable sandbox. A settled projection cannot make it resumable. Admission would pass, but a replacement sandbox would then fail on the missing backup, and a new sandbox would only try to resume a session that is gone. - A settled receipt written by recovery would claim a suspension that the Runner never wrote. Other checks rely on that receipt as Runner evidence. - Several paths can commit a result. Only runs that commit after a settle-at-commit fix would benefit. The admission fix also unblocks tasks that were committed before it. - Admission already owns this decision. It already has the scope claim, the journal, and the database rows. ### Safety argument - **No live prior owner.** The coordinator has no lease owner, and the controller must be proven stopped. The follow-up runs in a different sandbox than the old Runner. A resumed sandbox keeps the existing path, which reads the old Runner's own lifecycle there. - **No unsettled terminal delivery or effects.** The controller acknowledged the terminal and completed every command, including the suspend command. Every tool input has an answer, and the coordinator committed the result. The unreadable Runner outbox can then hold only post-terminal telemetry. - **No unsettled cleanup.** Any local Runner receipt, including an unsuspended one, keeps the hold. A coordinator recovery state keeps the hold. - **No ambiguous quarantine.** A quarantined root from a different owner keeps the hold. A quarantined root with Runner lifecycle evidence, unknown entries, or an unsettled journal also keeps the hold. - **No lost evidence.** The move keeps the old authority in the session root. Existing retention expires it like other continuity breaks. A journal change during the database reads cancels the move. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts`: 12 tests pass. - Before the change, the four admission tests failed with the exact error above. This includes a re-admission that rebuilt the journal from the database and failed again. - After the change, `created` and `replacement` sandboxes are admitted. A re-admission is admitted. A task with an earlier quarantined copy of the same owner is admitted. - Eight controls keep the hold: uncommitted result, live controller lease, unexpired controller lease, recovery state, pending command, unanswered tool input, a different owner in quarantine, and an unsuspended Runner receipt in quarantine after a retry rebuilds the root. - I disabled each new guard in turn. Each change made its matching control fail. - An exploratory run of the same fixture before the change answered the Retry question above. The second follow-up selected the failed follow-up as its source and failed with `native_controller_recovery_unsupported:metadata_format`. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts`: all 1,087 tests pass. - Related suites pass: `task-workspace-binding`, `native-controller-recovery` (unit and integration), `native-harness-backup-expiration`, `native-session-resume`, and the restart and diskless integration suites. Their container and Runner binary cases skip locally. - `node scripts/typecheck-low-memory.mjs`, `pnpm check:module-boundaries`, and `git diff --check` pass. - CI passes at `879bb6a5c7`. Two attempts lost jobs when CI runners received a shutdown signal; the third attempt passed every job. - I did not run the full `pnpm test:run`, `pnpm build`, or the opt-in Linux Runner container cases locally. CI runs them. - I did not test a live sandbox provider or a live model. I could not inspect the original checkpoint contents. If a stranded run's journal does not show a delivered terminal and completed commands, its follow-up stays held. ## Risks - A follow-up after this path cannot use the lost provider history. The provider session then follows the existing resume-or-replace policy, and the task context comes from Paperclip. A checkpoint that requires the same session, such as an unfinished goal, still fails at provider recovery. - Post-terminal Runner telemetry in the lost outbox, such as a late usage report, stays lost. It cannot carry a workspace effect. - A task that already has a failed follow-up from this error can still be blocked on a board Retry. Source selection picks that failed run and fails with `native_controller_recovery_unsupported:metadata_format`. This pull request does not change source selection. A same-run retry of the failed follow-up is admitted. - The change is limited to per-turn runs on the built-in sandbox transport. Warm sessions, plugin tasks, resumed sandboxes, and restart recovery do not change. - No migration or configuration change is required. ## Model Used - Anthropic Claude Opus 5.5 (model ID `claude-opus-5-5`) in Claude Code, with extended thinking, repository tools, shell and code execution, and local test runs. The runtime does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
da77c1bcfa |
fix(runner): align ordinary document key contracts (#18824)
## Thinking Path
> - Paperclip helps people manage AI agents that do work.
> - Native agents use tools to read and write task documents.
> - A document key is a stable identifier. The service trims it and
converts it to lowercase before it checks the key.
> - The tool schemas accepted filenames and long keys that the service
rejects.
> - This pull request makes ordinary document tool schemas match the
existing service contract.
> - Agents can now see the key rules before they call the tools.
## Linked Issues or Issue Description
**What happened?**
The schemas for `read_document`, `write_document`, and
`list_document_revisions` accepted `reconciliation.md` and keys longer
than 64 characters after normalization. The task document service
rejects these values. The descriptions did not explain that the key is
an identifier rather than a filename.
**Expected behavior**
The model catalog and the live and scenario tool definitions must
describe the existing document key rules. Values accepted after trimming
and conversion to lowercase must remain compatible. Historical
case-document keys use a separate contract.
**Steps to reproduce**
Validate `{ "key": "reconciliation.md" }` against the previous
`read_document` schema. It passes. Call the document service with that
key. It rejects the key before it reads the database. The added
regression repeats this comparison for all three tools.
## What Changed
- Apply the same document-key pattern to `read_task_document` and
`write_task_document` at the API bridge. Keep their existing raw-input
limit of 100 characters; ordinary tools keep 120. Export the shared
schema through the existing Runner package and server vendor boundary.
Route permissions, normalization and document locks stay with their
current owners.
- Add one shared schema for ordinary task-document keys. Keep the
existing 120-character raw input cap and the 64-character normalized
identifier limit.
- Preserve uppercase input, ECMAScript trim whitespace, and the Kelvin
sign, which converts to ASCII `k`.
- Apply the schema to the model catalog and the three live and scenario
definitions. Regenerate the catalogs, seeded fixture identity, and
fixture manifest.
- Test schema parity against the real task document service. Keep
historical case keys and the backend validator unchanged.
## Verification
- Current-head required CI is green at
`e67d3f1f49393660f0c70b3d1d136e6f86af821d`: 56 successful checks, four
intentional skips, and two neutral checks. [CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/41715924252)
passed after one targeted rerun of five jobs interrupted by the same
runner shutdown; the original interruption is retained. Fresh Greptile
and Paperclip reviews are both 5/5 with no unresolved review threads.
Hosted recovery remains unverified until automatic deployment and
replay.
- Normal master refresh to `e67d3f1f49393660f0c70b3d1d136e6f86af821d`
includes the already merged external-object startup fixture correction.
The previous required canary failed downloading
`mdast-util-gfm-table@2.0.1` with npm E404; later registry metadata and
tarball HEAD both returned 200. A separate general shard failed the
ten-thousand-future-objects startup assertion. Both original failures
remain recorded; neither was bypassed or retried. At the refreshed
baseline, 122 document/bridge cases and the exact large-object startup
case pass locally. Six document cases skip by platform. Fresh required
CI and reviews must pass on this head.
- The current-head Paperclip review found the same mismatch in the
task-scoped bridge. Before the correction, the real advertised schemas
failed 22 parity cases while 87 controls passed. After the correction
and a normal master refresh, the document-key and bridge suites pass 122
cases, with six platform-specific skips. The tests compare advertised
JSON Schema, runtime Zod validation, and the actual normalized document
service. They retain the bridge raw 100/101 boundary and historical
case-key controls.
- Runner TypeScript build, all server typecheck layers, and the actual
production Runner and server build pass. The final server build stamp
binds commit `80c880709c5f5630799fb8a48d54ff11ebc16610`. Full
current-head CI and fresh review are required after the correction. The
earlier CI run was externally cancelled and rerun; its cancellation
remains recorded.
- The regression failed in 33 of 64 cases before the fix. The remaining
cases passed as controls.
- The fixed regression passes all 67 cases, including every ECMAScript
trim character.
- The four focused Runner contract suites pass: 116 tests. They cover
all three tool surfaces and existing case behavior.
- Generated catalogs and manifest checks pass. Module-boundary and diff
checks pass.
- All server production typecheck layers pass. The test layer passes
after correcting the new test’s Ajv import. The full production server
and Runner build passes at
|
||
|
|
4bdcd1cffc |
fix(recovery): release holds for unstarted dispatch cancellations (#18838)
Release false recovery holds for legacy dispatch cancellations whose retained evidence proves execution never started. Keep audit history, current admission gates, review-wait Retry, and retained workspace obligations. Share the execution-absence proof across cancellation policies and make the dispatcher error-code policy exhaustive. Add PostgreSQL coverage and stabilize the browser history fixture by waiting for pause cleanup before seeding it. Fixes #15483 Refs #18085 Co-authored-by: commitperclip[bot] <284812584+commitperclip[bot]@users.noreply.github.com> Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2a16dcf629 |
fix(workspaces): preserve shared directory trees during enrollment (#18835)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A volume-free web server needs existing task files in durable object
storage.
> - Historical workspace records can share a physical folder or contain
other task folders.
> - A task-level migration hold cannot fence every alias, and capturing
a parent recursively duplicates nested task data.
> - Offline enrollment imports children first and verifies their durable
coverage before omitting them from a parent archive.
> - Snapshot retention preserves those child versions for as long as a
covering parent depends on them.
> - Existing files, task bindings, and archived states remain intact
through the storage migration.
## Linked Issues or Issue Description
Builds on merged #18826.
**What happened?**
The per-workspace importer could capture a shared project folder while
another workspace record still owned the same path. It also recursively
included runtime-created nested task workspaces in each parent import.
Packaged enrollment rejected existing workspace data instead of invoking
its migration owner.
**Expected behavior**
Require offline maintenance for overlapping paths. Import nested tasks
before their parents, verify every excluded directory against its saved
snapshot, preserve dependencies through retention, and resume
interrupted imports without replacing existing remote authority.
**Steps to reproduce**
Create a project workspace and another task workspace below its
`.paperclip/worktrees/` directory. Try importing either while ordinary
writers are running, then run offline enrollment. Interrupt a parent
import after upload and resume it.
## What Changed
- Add physical-overlap admission checks and an offline entry-point
guard, including global running and queued executions.
- Import pending records in leaf-first pages through the existing
migration owner.
- Verify nested snapshots by downloading, restoring, and comparing the
original source before and after parent upload; check transitive
dependencies.
- Publish parent dependency receipts under child snapshot row locks.
Retention pins those children and revisits them when the parent is
deleted.
- Index imported-child dependency lookups with a partial GIN index; a
PostgreSQL query-plan regression covers 2,500 unrelated snapshots.
- Propagate cancellation through capture and object storage operations
and preserve existing retry journals.
- Wire workspace migration into packaged maintenance and document its
preservation contract.
## Verification
- The original overlap and duplicate-parent regressions failed before
implementation.
- 73 migration and retention tests pass against real PostgreSQL and
files, including interrupted uploads, resumed batches, equal-path
aliases, changed or missing child data, nested descendants, and active
or queued executions.
- Six S3-compatible storage integration tests pass, including migration
through the maintenance owner and restoration of both parent and child
after deleting the original filesystem.
- The standalone process case was initially skipped because the worktree
had no bundle. After rebuilding the server and runtime bundle, that test
passed against PostgreSQL and S3: authenticated ready/export receipts,
preserved decision secret, blocked application routes, and clean SIGTERM
shutdown.
- Server typecheck, server build, runtime bundle build, and
module-boundary checks passed. Full repository CI passed on
|
||
|
DottaandPaperclip
|
b5dac661e9 |
refactor(runner): clarify workspace-relative artifact destinations (#18816)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native agents can copy an authorized saved artifact into their
execution workspace.
> - The installer accepts workspace-relative destinations and rejects
absolute paths.
> - The API documentation states this rule, but the model-facing tool
description and destination field did not.
> - An agent could therefore attempt an unsupported absolute path before
using a supported relative path.
> - This pull request states the existing rule and gives a relative
example in the tool contract.
> - The benefit is clearer instructions with the same containment and
file identity checks.
## Linked Issues or Issue Description
Refs #18494, which introduced verified artifact installation.
**What happened?**
The tool description said the destination belongs inside the workspace
and its parent must exist. It did not say that the destination must be
relative. The installer correctly rejected an absolute path, including
one lexically inside the workspace. A relative retry could succeed.
**Expected behavior**
The model-facing tool definition must explain the same
workspace-relative path rule as the existing API documentation. It
should show a relative example and state that absolute paths are denied.
**Steps to reproduce**
Inspect the exported download_artifact definition and destination JSON
schema before this change. Compare them with the installer guard and
doc/runner-api-tools.md. The description omitted the relative-only rule
even though the installer enforced it.
## What Changed
- State the workspace-relative destination rule, source.zip example and
absolute-path denial in the tool description.
- Add the same guidance to the destination field metadata emitted in its
JSON schema.
- Keep parsing, installation, authorization, byte size/hash checks and
all path containment guards unchanged. The existing documentation
already states the intended rule.
## Verification
- The first required Linux run failed in the unchanged Kimi model-route
setup, before its HTTP assertion. The failure remains recorded. After
#18828 merged, this branch was refreshed by an ordinary merge of master.
The same five HTTP cases pass on that baseline. Local artifact suites
report 20 platform-specific skips on macOS; the earlier required Linux
jobs passed all 19 artifact cases and the artifact authority case. Fresh
exact-head CI and review passed at
`c7b3226c745caaaedad569369b71383a9a6e89ee`.
- Inspected the real exported tool definition and JSON schema after the
change. Both now expose the relative-path guidance and example.
- A source-only probe used the real schema and installer with synthetic
bytes and a stubbed command runner. Absolute in-root, absolute
outside-root, traversal and reserved paths remain denied before
dispatch. A relative path reaches the stub runner. This is a
contract/path-guard probe, not real byte-installation proof.
- No functional red is claimed. The original unsupported tool attempt
remains a retained failure even though the relative retry recovered. The
exact provider-seen descriptor and the model's counterfactual behavior
are not proved.
- Server typecheck, module boundaries and whitespace checks pass.
Production server build also passes at exact head
|
||
|
DottaandPaperclip
|
5877d751c0 |
feat(server): bind idle credential eligibility to incoming wake proof (#18639)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
0b87308d5d |
fix(native): defer warm continuation during predecessor finalization (#18811)
## Thinking Path > - Paperclip manages agent work and preserves its saved results. > - Native warm sessions can continue on a retained remote Runner. > - A completed predecessor can still have a live finalization lease. > - The continuation must wait for that owner before it can start work. > - Two admission checks treated that temporary lease as ambiguous state. > - This pull request uses the existing deadline-based wait for both checks. > - The same business run can continue once, with its saved context and gates. ## Linked Issues or Issue Description **What happened?** A compatible remote warm continuation can fail before provider work starts. Its predecessor has completed, but a finalizer still holds a live lease. The warm-session check reports `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. The earlier retained-sandbox check has the same problem on the command transport. **Expected behavior** Keep the unstarted continuation on its existing run. Wait until the recorded owner deadline. Recheck ordinary ownership, Stop, credentials and checkpoint evidence before execution. **Steps to reproduce** 1. Complete a remote warm turn and retain its controller records and provider session. 2. Give that completed predecessor a live finalization lease. 3. Admit a compatible follow-up with full local controller projections. 4. Before this change, admission fails permanently. It must instead persist a bounded pre-provider wait. **Paperclip version or commit** The PostgreSQL regression failed before production edits on `adb49a4434`. The current head is `5762dfbe4e`, based on master `e491b23eb5`. **Deployment mode** Native remote sandbox execution. No database migration or provider artifact change is required. Related: #18718 refreshes stale warm-controller projections. #18343 adds controller replacement recovery. #18554 restores suspended remote sessions. This change handles a live predecessor finalizer after those source checks. The shared historical error does not identify its failed predicate. This regression proves the finalizer race; it does not prove that every historical instance of that error had this cause. ## What Changed - Defer a compatible warm continuation while its completed predecessor has a live finalization lease. - Use the same wait in the preceding retained-sandbox read check. - Preserve the current run, provider attempt count and checkpoint. Reuse the existing resumption scheduler. - Add PostgreSQL cases for both finalization phases, lease expiry, company scope, incompatible configuration, unfinished commands, Stop and duplicate resumption claims. - Add an opt-in three-turn Linux Runner case. Update the child-controller fixture to own and register a local API listener. - Document the waiting behavior and acceptance case. - Repair two readiness barriers in the refresh-worker integration fixture after a required CI shard failed twice: wait for paginated startup and for active worker promises to settle before checking idle SQL. ## Verification - **Native regression RED:** before production edits on `adb49a4434`, the PostgreSQL boundary case received the permanent identity error instead of `NativeControllerRecoveryWaitingError`. - **Regression GREEN on the current head:** all **71** focused recovery, settlement, heartbeat safety and refresh-worker cases pass on `5762dfbe4e` with Node 24.21.0 and PostgreSQL 18. They cover incompatible identities, company/configuration mismatches, expired or missing deadlines, unfinished commands, Stop, retained files and duplicate resumption claims. - **Real process acceptance on the current head:** the three-turn `finalizer-wait` case passes in 58.01 seconds. PostgreSQL, TLS, a rebuilt Linux Runner and the deterministic Codex app-server fixture confirm zero attempts and no provider turn during the wait, one claim resuming the same run, retained history, a healthy third warm turn and final suspension. The sibling `warm-suspended` case from master also passes, in a fresh container, in 46.02 seconds. Both runs use `5762dfbe4e`; the artifacts were rebuilt from that checkout with the repository's pinned Rust 1.97.1 toolchain. - Acceptance used Node 24.21.0 and glibc 2.41. Earlier probes exposed the missing fixture API listener and the preceding retained-sandbox guard; both are corrected. A stale disposable listener and an incompatible libc image failed setup before the successful fresh-container run. These setup failures are not the original regression's RED. - Repository typecheck/build passed locally before the rebase. Server typecheck passes after the rebase. **Full CI passes on the current head:** 48 successful jobs and one expected scope skip, including typecheck, build, Runner checks, all serialized-server shards, general tests, E2E and the final aggregate. Module-boundary and whitespace checks also pass. - Broad local inventories are incomplete: six chat shards passed (1,834 tests), but untouched server/UI/adapter cases hit timeouts. The exact shared-base expiration case passes alone; a full-file rerun passed all 89 assertions but timed out in teardown. A Node 24 rerun timed out in another lifecycle case and teardown. A later PostgreSQL recheck could not allocate shared memory; this task's redundant broad controllers were stopped and their logs retained. - CI attempt 1 failed a Cursor Cloud cancellation deadline, an E2E cleanup cancellation with a PostgreSQL lock timeout, and the refresh-worker startup settling limit. One failed-only retry passed Runner and E2E; the same refresh startup case failed again. Two aggregate jobs had previously stalled in the hosted-runner queue and blocked the retry; that obsolete attempt was cancelled. No repeated blind rerun is being used. - **Readiness repair RED/GREEN:** real PostgreSQL 18 behind a loopback proxy with 20 ms delay per direction reproduces the exact startup error before the test change; the same probe passes afterward. An initial controlled probe hit the dedicated-database naming guard before tests ran and is excluded from RED evidence. The full-file check then exposed a separate false-idle barrier: a worker could be running while its scheduler had no timer. The fixture now also checks the existing active-work tracker. All **24 cases pass** with both repairs. The at-most-six scoped SQL reads and zero recurring idle-SQL assertions are unchanged; startup alone has an explicit bounded condition wait. No production behavior or timeout changes. - Greptile and Paperclip Review are both **5/5 on the current head**, with no open findings. Both reviews used source inspection; the runtime evidence above was obtained separately. The previous review workspace was recovered on the first production attempt after repairing its partial-clone export. Its preserved result and real follow-up both finalized successfully, saved snapshots and released their writers and finalization leases. Three current-head CI shards received hosted-runner shutdown signals and were cancelled. All three passed their one justified retry. The queued aggregate had to be cancelled to schedule that retry; the final aggregate also passed. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41706617909). - This acceptance does not qualify a live Codex model or Daytona API run of the native repair. No deployment or recovery of the original task is claimed. ## Risks - A wait must not grant a second executor. The current attempt stays at zero until the existing scheduler claims it. Expired or missing deadlines retain their existing hold. - Saved state must survive the wait. Admission still requires the same company, task, session, configuration and physical sandbox evidence. - Current Stop, budget, credential and permission gates remain with ordinary resumption. No polling loop is added. ## Model Used OpenAI Codex, GPT-6 family. The runtime does not expose the exact serving model ID or context-window size. Capabilities: reasoning, repository inspection, code editing, shell tools and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e0276e0166 |
fix(workspaces): identify remote command timeouts (#18827)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspaces prepare repositories and save files through sandbox commands. > - Each command has a timeout and a known operation. > - The transport discarded that information when a command timed out. > - A failed clone then appeared as the same generic error as other failures. > - This pull request keeps the operation and timeout in a fixed diagnostic. > - Operators can identify the failed stage without exposing command output. ## Linked Issues or Issue Description **What happened?** A remote workspace setup command reached its 300-second transport limit before provider work started. The workspace layer reported only `remote_workspace_operation_failed`. That message concealed the timeout and the affected operation. A later full clone on the same sandbox image passed in 107 seconds, so the original clone delay remains unexplained. This PR repairs the loss of diagnostic information. It does not claim to prevent the underlying delay. **Expected behavior** A transport timeout must remain a failure. Its diagnostic must identify the operation and configured limit. It must exclude raw output and must not automatically repeat a command whose remote outcome is unknown. **Steps to reproduce** 1. Return `timedOut: true` from the sandbox command boundary during workspace preparation or cleanup. 2. Observe that the base implementation reports only `remote_workspace_operation_failed`. 3. Apply this change. Preparation reports `remote_workspace_operation_timed_out: prepare (configured timeout 300000ms)`. Cleanup retains its separate 10-second limit. **Paperclip version or commit** The regression was reproduced on master base `367f29478d14003cc5c0f161b7ce8f3164993634`. The fixed current head is `be45ce6594254bd6b94023a550d8b655dd704bb0`. **Deployment mode** Hosted Paperclip with native Runner and Daytona sandbox execution. Local acceptance also uses disposable PostgreSQL and filesystem commands. Related work: #18403 introduced remote workspace authority. #18705 added a fixed full-disk diagnostic. I searched recent clone and remote-workspace PRs and found no duplicate timeout-reporting repair. This is a defect fix and does not add a roadmap feature. ## What Changed - Give each internal workspace command a closed operation name: prepare, repository, archive helper, capture, restore, or cleanup. - Preserve that operation and the configured limit in a typed error when the transport reports a timeout. Do not claim a measured elapsed time. - Keep other failure diagnostics and exact safe-message checks unchanged. - Preserve typed cleanup timeouts in the operation cleanup journal while retaining generic diagnostics for untrusted errors. - Add preparation, cleanup, and real-PostgreSQL lifecycle regressions. Document the timeout and the need to reconcile saved receipts before recovery. ## Verification - Red: both transport regressions failed on the base implementation. A separate real-PostgreSQL cleanup regression reproduced the review finding: the operation journal replaced a typed timeout with a generic error. - Green: all 62 remote transport and archive tests and all 50 real-PostgreSQL workspace lifecycle cases passed. The lifecycle suite ran with an explicit disposable database; it was not skipped. Command: `pnpm exec vitest run --project @paperclipai/server server/src/services/remote-workspace-transport.test.ts server/src/services/remote-workspace-snapshot-archive.test.ts`. - Safety: a timeout with exit code zero still fails. Raw stdout and stderr stay out of the error. A timeout cannot claim the fixed unfinished-Git diagnostic. No command retry occurs. Existing saved files remain unchanged. Existing archive, restore, receipt, symlink, credential-isolation, and full-disk controls pass. The cleanup lifecycle cases retain the acknowledged snapshot, preserve working files, avoid duplicate uploads, record only a safe typed timeout, and settle cleanup on the next normal reconciliation. An untrusted error that mimics the timeout prefix stays generic. - Live acceptance: production transport code ran against actual Daytona SDK 0.203.0 timeouts on the incident image, with 4 CPUs, 8 GiB RAM, and a 10 GiB disk. A controlled one-second SDK limit produced the generic baseline error and the fixed exact preparation timeout. The current `be45ce6594` code was rechecked for both preparation and cleanup. Each returned the typed error and its own configured limit (300 seconds and 10 seconds) while the controlled SDK fault remained one second. Each case made one command call. Each disposable sandbox was deleted before the next case. - The first live-probe attempt used an incorrect error-name check. It failed its acceptance check and remains recorded. The probe was corrected to use the same SDK timeout base class as production. Its first sandbox was deleted. This was a probe setup failure, not red evidence for the repair. - A separate same-image probe ran the real full repository preparation successfully in 107 seconds. The affected command path is unchanged except for error reporting. The original task was retried once through the product on deployed source `adb49a44346849e751548b2fb640693dbfad4b6f`. The requested memo and its first document revision persisted after reload and matched the downloaded attachment. The run succeeded, the task reached Done, its execution lock cleared, and final workspace save and result commit succeeded. The diagnostic PR is not deployed. - Recursive typecheck and build passed on `be45ce6594`. `git diff --check` passed. The first full-suite attempt stopped before tests because the offline install had skipped an embedded PostgreSQL native postinstall. The package's official symlink hydration fixed `initdb`. - The next Mac run passed 350 chat cases and hit one unchanged 15-second chat deadline. That case then passed in isolation, and all six full chat shards passed. The broader Mac matrix was later stopped as partial when CI results became available. No full single-command Mac pass is claimed. - Full Vitest file coverage uses the exact successful CI merge tree, `d264fd98e7fb6eff4a9e6b00d402f88894cc0e57`. Its canonical inventory has 2,899 files. Passing CI covered 2,057 files. The remaining 842 ran locally through the standard exact-file harness. The inventory and selection checks show no unassigned files. All four changed source/test files are byte-identical between the PR head and that merge tree. - The first remaining-file run passed 771 files. The database group passed 68 files and reported four deadline/readiness failures in three unchanged files. Those same three files passed all 12 cases when run sequentially on Node 24.22.2, matching CI's major version. Assertions, readiness limits, and test deadlines did not change. The first attempt used Node 26.4.0 with a parallel UI workload; this does not isolate the contribution of runtime version versus load. Both attempts remain recorded. - [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/41699549115) is green on current head `be45ce6594`: `ci / verify`, typecheck, build, all selected server/chat/workspace groups, Runner checks, and all eight browser shards passed. Greptile is 5/5 with no open threads. The prior P2 was resolved by removing new code comments. - Structured Paperclip review remains queued because its own previous accepted result exhausted workspace export recovery. A fresh current-head request is saved, but normal Retry returns HTTP 409 (`accepted_result_without_new_input`). Its advisory cleanup finding is fixed in `be45ce6594` and covered by red/green lifecycle evidence. This check is informational. The branch requires `ci / verify` and `ci / e2e`, and both pass. The independent review-service limitation remains recorded; this PR is not deployed. ## Risks Timeout error text changes. Consumers that compare the old generic message may need to accept the new diagnostic. No such production comparison was found in this repository. API shapes, deadlines, retries, ownership, credentials, and model selection do not change. A timeout still does not prove that the remote process stopped or that a save failed before publication. Existing recovery must reconcile that state. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code execution, and browser tools. The host does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1f55073f96 |
fix(cursor-cloud): support API-only controllers on managed stacks (#18310)
## Thinking Path > - Paperclip helps people manage AI agents at work. > - Cursor Cloud hosts execution behind its API. > - Product setup still expected a Paperclip Computer. > - Managed-only tenants therefore rejected valid setup. > - This admits the validated API-owned controller. > - Authority, Stop proof, budgets and cleanup remain enforced. ## Linked Issues or Issue Description Refs #18068 and #18069; both dependencies have merged. **What happened?** Managed-only setup still requested a Computer. Live tests exposed missing Cursor Stop proof and duplicate replay. **Expected behavior** Use Cursor’s API controller. Verify provider/controller termination and replay each log identity once. **Steps to reproduce** Enable native Runner/managed-only. Configure Cursor model/repository/key without a Computer, then Test, finish setup, run and Stop. ## What Changed - Derive API-only placement from validated native Cursor configuration; reject placement/command/workspace/home overrides and keep Kubernetes-only execution closed. - Skip Computers/checkout/workspace realization. Keep scoped tools, completion, private staging, budgets, leases and recovery. Persist placement; reject contradictions. - Align Cloud cards, Test, Save and Finish setup; hold unready drafts. Legacy keeps its Computer, credential and policy checks. - Verify Stop from exact API ownership, audited intent, provider-terminal proof, no Computer/process ownership and controller retirement. Recheck immediate/deferred retirement; wake existing recovery on durable evidence. - Clear warnings when the actual stopped target later has authoritative confirmation. Keep missing/other-run negatives and the existing predicate. - Publish safe status refresh after full-proof ACK. Retain bounded native/physical cancellation history; add no poller or weaker proof path. ## Verification Core merged as `3910352c923a`; Product merged as `27d35314be8e`. Public branch `5f86eb66b081` has tree `67344151d303`. Its exact-head Greptile summary is 5/5; the fresh audit found zero unresolved threads and complete pagination. The controlled workspace retry and required summary passed. All 59 current-head contexts are terminal and clean. Prior green-36 evidence keeps its identity. [Current CI](https://github.com/paperclipai/paperclip/actions/runs/41680511731) first failed workspaces-a 1/2: the CLI company-import/export beforeAll 30-second runtime-info/child-PID/HTTP readiness wait expired. All 431 UI files/4,714 tests and 285 other CLI tests passed (one skip). The cause remains UNKNOWN. Child output records startup recovery complete, but buffered CI prefixes do not establish its relation to the waiter deadline; runtime-file/PID/last-health state was not captured. Both unchanged local probes were RED. The second observed dependency preflight still pending at the deadline; observer overhead was not quantified. Neither proves the CI cause. After all functional jobs finished, only the queued metadata summary was cancelled; the other 48 records stayed unchanged. Attempt 1 retains 46 successes, the original failure, one skip and the cancelled summary. One same-head debug retry was admitted at 18:42 UTC on November 3. Its workspace job passed at 19:23 UTC on the same checkout: UI 431 files/4,714 tests and CLI 33 files/286 tests, including company import/export. Test caching was disabled. The new `ci / verify` summary passed; workflow attempt 2 completed SUCCESS. Required `ci / e2e` also passed. The final check at 19:53 UTC on November 3 found all 59 current-head contexts terminal and clean: zero pending and zero failing. The 47 inherited completed/skipped rows are not new executions. No code, assertion or timeout changed; success does not explain the original failure or erase the local REDs. CI checkout is `1ea9d8608ced`/tree `78cb1ec13d59` (parents `2e37c73c05fe`, 5f86); both workflows use 2e37. Its 20 extra paths add low-memory build/Rust task-memory. Scripts cohere; later matrix changes only labels, with no newer debug-cache steps. Static audit found no blocker. Supported API arguments omit the memory flag: no default monitor/instruction changes. Flagged external launchers can change Cursor instructions/digests and remain unqualified. The merge added 41 master-identical paths, retained the green-36 inbox hunk and found UI fixtures matching upstream. Startup/backup, recovery projection, sleep diagnostics and binary backup change production. Static audit found eight Stop/ACK owner blobs and 37 sleep-work SQL templates unchanged, with sleep fences/single-flight accounting intact. Exact precommit tree: 416/416 across eight server files, binary backup 1/1, server types PASS; repeated scopes are not summed. The deferred-retirement P1 reproduced three real-DB/owned-process failures; missing proof stayed unverified. Recovery passed 47 process cases/33 proof negatives. Warning repair binds actual targets to same-run proof. Validation passed 729 scoped UI cases, a separate 359-case thread run, 111 backend cases (84 CAS, 6 assembled, 21 activity/privacy), types/tokens. Proof validators/predicate are unchanged; no new poller or increased timeout. The d8d3 attempt stays FAILED: 45 passes, two workspaces-a failures, derived verify failure and one skip. Both shards shared three failures: two successful mocks omitted the account guard’s required userId. The two-line fixture repair changed no guard, assertion or timeout. Three failures reproduced; 54 access/auth tests then passed, including account-change/denial negatives, plus UI types/tokens. The 69285 attempt stays FAILED: 49 jobs, 46 passed, serialized shard 9 failed, its verify aggregate failed and one selection job skipped. The shard passed 575/576 cases. Its aborted-snapshot case expected nine computes but saw eight: a final HTTP 200 coalesced while seven ordinary reads had not proved the aborted compute settled. Earlier CI and the unmodified local baseline passed with a miss; no local RED is claimed. The repair awaits the snapshot, allows exactly one extra GET if coalesced, then requires a miss and retains nine computes/all HTTP429 assertions. No production code, hook, loop, sleep or timeout changed. Six inbox and seven cache/admission cases passed locally, plus server types; 30 unrelated cases were filtered. Green 36 passed 576/576 without taking the extra-GET branch; local runs also missed that schedule. Current CI event merge 1ea9 now passed 576/576 and naturally exercised the former failure: coalesced (count 2, 1ms), then exactly one fresh miss (count 1, 16ms), with strict nine-compute/HTTP429 assertions intact. No instrumentation or retry was added. Earlier compatibility passed scoped remote/inbox, routine, API-placement and UI tests, plus types/migrations/snapshots/maps/tokens. The 89-path adoption restored fleet routing without workflow overrides or added skips. Each result retains its source identity. Live qualification remains actual `9af1a5de42a2`: unchanged Artifact grader PASSED 17/17 and active-Stop grader PASSED 21/21. All three provider test resources were deleted canonically with GET404; post-delete download returned the exact 27 bytes. Histories stayed stable (Artifact 121 and Stop 67 canonical identities), with 21 company runs, zero active work and no accounting/cancellation/reservation debt or integrity findings. Saved settings reopened; no-prompt Test/Linux admission passed with `claude-sonnet-4-5`. No new Save is claimed. The first Stop sampling case stays FAILED on a 10.373-second gap. Unchanged retry kept all 24 samples: 35.076 seconds quiet, maximum gap 4.584 seconds, plus a separate CANCELLED/artifact-absent read after 180 seconds. Initial grading rejected locator spelling before functional checks; metadata changed only after same-file/inode/byte proof, with raw evidence/oracles unchanged. Original Question/Plan/reconnect and historical deleted-artifact results keep their identities. Git trees verified 11 history-only conflicts and six earlier mainline paths. Hosted 9af to the public BRANCH differs in 222 paths; 13,500 other mode/blob leaves match. Runner runtime implementations and six observer/driver pins match 9af on that branch only; inherited `drivers/acpx/runtime-host.test.ts` differs. No all-Runner/src, CI/latest-master runtime or image equality is claimed. Later static audits through `8ab89f41e499` found no concrete API-only blocker. That base adds completed-history sleep proofs and an MCP cleanup migration; protected Stop/ACK owners are unchanged. These later changes are outside this CI and have no live qualification here. Host duplicate-card and later recovery/UI repairs retain separate proof, not deployed 9af identity. Direct-key coverage requires absent aiConnection; API-only placement skips inherited remote-workspace disk-full paths, and config-read gates match their parent. Other retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; title/startup/custody/Stop/deletion unknown causes and bounded recoveries; superseded warning/50 duplicate identities; pre-repair ACK/Core CI failures. Fixture RED/GREEN passed 7/7 browser and 121/121 shutdown tests with a failing broken-unsubscribe control; stable WSP wrapper passed 128 after physical teardown failure. No full local-suite pass is inferred. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and behavior checked live; unpriced provider dollar cost stays UNKNOWN. The placement exception applies only to validated native Cursor API execution. Other local work remains denied under managed-only policy. Missing/stale Stop or cleanup proof stays pending; damaged archives remain failed. Explicit Legacy keeps its existing policy owners. ## Model Used OpenAI Codex/GPT-6 family: tools, execution, browser checks, delegated review. API ID/context window unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; retained failures are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
4d31588dea |
test(server): load model route setup before test deadlines (#18828)
## Thinking Path > - Paperclip uses native agents and local provider adapters to do work. > - The server model route must keep provider catalogs separate. > - Its tests use the real HTTP route with controlled service and adapter fixtures. > - Each test reset the module graph and imported the route again inside short test deadlines. > - Slow imports timed out. Unfinished work could then change mock call counts in a later case. > - This pull request loads the stable route graph during collection and keeps each case's fixture resets. ## Linked Issues or Issue Description Refs #18816 (affected required CI run). **What happened?** The pending Kimi model-route test exceeded its 15-second deadline in required Linux CI. The unchanged test file also failed locally: the first setup hook exceeded 30 seconds, later tests exceeded 15 seconds, and the last OpenCode case saw two calls instead of one after unfinished work reached the shared mock. **Expected behavior** Route import and transformation must finish before the case deadline starts. Each case must still test the real HTTP route with its own mock state, environment and profile values, and adapter registry cleanup. **Steps to reproduce** Run `pnpm exec vitest run --project @paperclipai/server server/src/__tests__/adapter-model-refresh-routes.test.ts` with a cold module graph. On a loaded worker, repeated module resets and imports can exceed the hook and test deadlines. ## What Changed - Register stable service mocks once and load the real route and middleware during test collection. - Remove per-case module invalidation. Keep all five original HTTP assertions, fixture resets, profile and environment restoration, and adapter unregister operations. - Keep the existing hook and test deadlines. Product source code does not change. No user-facing documentation changes are needed. ## Verification - The unchanged file reproduced the hook timeout, test timeouts, and leaked call-count failure locally. - All five original cases pass with the fix. The cold graph takes about 47 seconds during collection; the cases complete within the unchanged deadlines. - All five cases also pass at the same head in a shuffled order with seed `20261103`. - Module-boundary and diff checks pass. - Full repository typecheck, tests, and build run in required CI. No hosted product behavior is claimed from this test-only change. - Current-head CI attempt 2 failed in the typecheck harness because the trusted master workflow called `scripts/typecheck-test-projects.mjs`, which the older source head did not contain. The log is retained. A normal merge of master supplies the helper from #18738. All five original HTTP cases pass again on that merged baseline. Fresh CI and exact-head reviews passed at `86f0bf27267fe3f44ab813fb0a57ebbb94bae7fe`. - The refreshed run passed all server checks but one browser shard failed: `task-reload.spec.ts:63` waited five seconds for a task heading after the intentional bundle failure. The trace showed the saved task breadcrumb and properties, with the conversation still loading. The exact failure log and trace were retained. One targeted retry of that shard and its aggregate passed without source changes. This establishes recovery of the check; the original browser loading delay remains unexplained. - Final exact-head checks: 52 success, two skipped, and two neutral. Both Greptile and Paperclip reviews are 5/5. No review threads remain. ## Risks - Stable imports share module state across cases. The test retains explicit mock resets, adapter unregister, and profile/environment restoration. - The affected model route reads profile qualification and environment fixtures when it handles the request. These cases do not require a new module instance for each environment change. - No runtime or schema changes. No test deadline increase. ## Model Used OpenAI `gpt-6.1-sol`, Extra High reasoning. The model used repository tools, code execution, and test output. The host does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f836fec59e |
perf(budgets): avoid company lock for routine eligibility checks (#18830)
## Thinking Path > - Paperclip helps people manage AI agents at work. > - The budget service checks whether an agent can start work. > - Preliminary checks acquired the company accounting lock even with no budget policy. > - Independent checks then waited behind accounting writes for the same company. > - This pull request reads one consistent snapshot before it decides whether reconciliation is needed. > - Healthy checks avoid the writer lock. State changes and final paid admission retain their existing authority. > - The benefit is less company-wide contention during routine eligibility checks. ## Linked Issues or Issue Description Refs #17697 (the preliminary-check portion). Refs #17817 (related serialization in agent configuration). Related approach: #17966. That PR adds a precheck for policies that cannot block. This PR also avoids the lock for healthy active hard-stop policies and manual pause reads. It uses a repeatable-read snapshot and leaves public transactional admission locked. It does not add the inactive-policy spend filtering proposed in #17966. **Current behavior** Every public `getInvocationBlock` call takes the company writer lock. Healthy checks can wait behind an unrelated receipt or settlement operation. **Proposed behavior** Read current scope state, policies, exposure and relevant incidents in a read-only snapshot. Return directly when no transition is needed. Otherwise discard the snapshot and reconcile current state under the company lock. **Reason and benefit** Independent preliminary checks can finish while another transaction owns the company row. Final reservations and paid-work admission still serialize against current accounting state. ## What Changed - Separate preliminary eligibility inspection from mutating reconciliation in the budget service. - Share scope loading, pause decisions and soft-incident resolution conditions between the two paths. - Keep no-policy checks free of exposure scans and incident reads. - Add inspection and reconciliation spans to the existing operator-gated Observability path. - Add PostgreSQL contention tests and regressions for pauses, holds, incidents, recovering native accounting and company boundaries. - Use the shared fixture startup timeout. Per-test deadlines and the 500 ms lock timeout stay intact. - Update the implementation contract and tracing documentation. Add dated investigation and future-design notes. The future accounting architecture remains a proposal. ## Verification - 73 targeted tests passed across eight files on implementation commit `244db565`. No tests were skipped. The files cover budget service, enforcement scoping and delivery, live receipt attempts, budget history, run settlement, heartbeat accounting and budget tracing. - Real PostgreSQL tests hold the company accounting lock while routine checks finish. The same reader's authoritative admission instead fails with PostgreSQL lock timeout `55P03`. - Full workspace typecheck passed before the rebase: `node scripts/typecheck-low-memory.mjs`. - Full workspace build passed before the rebase: `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm -r --workspace-concurrency=1 build`. The Runner release binary was rebuilt from source. - `node scripts/check-module-boundaries.mjs` and `git diff --check` passed. - Initial local setup failed before assertions: cold fixture setup exceeded 20 seconds, then a missing SDK build was exposed. The fixture now uses the shared 90-second startup allowance. The SDK was built before the successful test run. An incomplete local Copilot artifact also blocked the first build; the pinned installer repaired it before the successful canonical retry. - The branch rebased cleanly onto master `83b5f7d486`. All 73 targeted tests then passed again on clean head `244db565ae206502b100c3cdf5cd86a0ae91b372` (eight files, no skips). The full local Vitest suite was not run; remote CI must verify the wider suite. - Initial remote CI failed three monitor-scheduler cases at the suite's three-second heartbeat-idle wait. The unchanged suite then passed all 21 tests locally (`pnpm exec vitest run src/__tests__/issue-monitor-scheduler.test.ts`, 70.13 seconds). The one failed-job rerun was cancelled after the review follow-up was pushed; it produced no pass/fail result. No scheduler assertions or timeouts were changed. On updated head `ba2ab2a0`, the unchanged monitor suite passed all 21 tests in 38.381 seconds in [server shard 10/12](https://github.com/paperclipai/paperclip/actions/runs/41702687492/job/126151059714). That shard passed 115 files and 2,677 tests, with no failures (one file and 20 tests skipped by existing suite configuration). - The independent review identified missing recovering-run coverage. The added contention case requires the locked fallback while another accounting writer holds the company row, then verifies the recovery block without pausing or cancelling recovery. The regression is in follow-up commit `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`. All 19 budget-service tests passed locally (no skips, 48.89 seconds), and `pnpm --filter @paperclipai/server typecheck` passed after the follow-up. - The updated CI run then hit a separate plugin-refresh fixture timeout at its final re-enable settle. The unchanged `external-object-refresh-work.test.ts` suite passed all 24 tests locally (52.02 seconds). Inspection identified a possible fake-clock race: its quiet helper can observe no scheduled deadline while reconstruction is still active, then encounter the existing 30-second reconstruction cooldown. The one same-head failed-job retry [passed](https://github.com/paperclipai/paperclip/actions/runs/41702687492/job/126158227876): refresh suite 24/24, shard 115 files and 4,125 tests passed (one file and four tests skipped by existing configuration). No assertions, timeouts, or product code were changed. - [Full CI](https://github.com/paperclipai/paperclip/actions/runs/41702687492) passed on exact head `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`: build, typecheck, Runner checks, all server/workspace suites and E2E. Both required aggregates are successful. All 59 checks are terminal: 54 successful, two advisory neutral and three expected skips. The PR is mergeable with a clean merge state. - Deployment latency and CPU effects have not been measured. - Greptile reviewed updated head `ba2ab2a0b001f7bd69e0f86fb89a842e79b0e2b8`: 5/5, no actionable findings or unresolved Greptile threads. - Independent Paperclip review on the updated head: advisory 4/5, no blocker, recovery gap confirmed resolved. One canonical maintainability advisory remains visible and is documented below. ## Risks - Preliminary inspection can precede a concurrent policy or pause change. It grants no paid dispatch authority. The final locked admission and reservation checks remain required. - Healthy configured policies add an incident lookup. An unheld blocking scope can compute exposure during both inspection and locked revalidation. Already-held scopes skip the first exposure computation. - Holds and incidents must retain their existing transition effects. The fallback reads them again under the company lock, rather than mutating from the earlier snapshot. - Eligibility inspection and reconciliation still maintain separate transition lists. Future reconciliation changes must update the inspection predicate and its contention coverage together; consolidating the effects model remains a nonblocking review follow-up. - No schema or API response change is required. This change does not relax the current budget enforcement contract. ## Model Used OpenAI Codex with model `gpt-6.1-sol`. Used reasoning, repository inspection, code execution and delegated review. The session did not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e491b23eb5 |
fix(runner): retire sandbox suspension receipts after admission (#18829)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Native Runner sessions preserve a conversation across task turns. > - Remote suspension records a local receipt for the stopped Runner identity. > - The built-in sandbox path now writes the same receipt as plugin tasks. > - Admission cleared that receipt only for plugin tasks. > - A later warm turn could reject a clean journal because the old receipt named an earlier run. > - This pull request clears the receipt after successful admission for both sandbox paths. ## Linked Issues or Issue Description Refs #18343 and #18554 for the shared suspension receipt producer. Refs #18718 for the related controller projection refresh. **What happened?** After a suspended built-in sandbox session resumed, its next warm turn could fail with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. Its journal named the completed resumed run. Its local suspension receipt still named the older run. **Expected behavior** Successful admission must retire the prior suspension receipt. The next compatible warm turn must reuse the admitted session and provider history once. A failed admission must retain recovery evidence. **Steps to reproduce** 1. Complete and suspend remote turn A. 2. Resume turn B through the built-in sandbox transport. 3. Keep the controller warm, with full Runner state owned by the sandbox. 4. Admit turn C. The old local receipt conflicts with B's journal identity. **Paperclip version or commit** Reproduced in the executor regression on base `367f29478d`. **Deployment mode** Native Runner with the built-in remote sandbox transport and a warm lifecycle. ## What Changed - Clear the local suspension receipt after a remote sandbox session passes provider recovery and durable admission. Apply this to built-in runners and plugin tasks. - Extend the existing three-turn warm admission regression to cover a suspended receipt and remote-only Runner state. - Keep an active-owner control that rejects admission and retains its receipt. Keep the existing foreign-owner, retirement, configuration, plugin, and unchanged-session controls. - Add a real Linux Runner continuity case that suspends A, resumes B, and keeps B warm for C. Register the test fixture's owned local API listener. - Document receipt ownership and the new continuity case. ## Verification - The new behavioral regression failed before the production change. Turn C raised the exact identity mismatch above. The active-owner control passed and retained its receipt. - All 1,087 native executor tests pass after the fix. - Both opt-in Linux process cases pass at the final commit: ordinary warm reuse and suspension followed by warm reuse. They use PostgreSQL, TLS, current-source Rust binaries, and a deterministic Codex fixture. Each preserves one conversation, starts exactly three turns, and reuses the warm Runner and provider on turn C. - Full repository typecheck passes with `node scripts/typecheck-low-memory.mjs`. Full production build passes with `pnpm build`. - All current-head CI checks pass at `a95a6fdfe8`, including the full test shards, browser shards, Rust checks, build, and typecheck. The policy job needed one retry after repository checkout timed out before checks ran. - Greptile and Ponytail each give 5/5 at the final head. There are no open review threads or findings. - The local full `pnpm test:run` attempt passed 351 cases in its first chat shard. The session interruption stopped it during the second shard, so it has no complete local verdict. The complete CI test run passes. - Module boundary checks and `git diff --check` pass. ## Risks - A receipt removed before admission could discard recovery evidence. Removal stays in the successful `onSession` callback. The negative admission control checks retention. - A stale owner must not gain authority. Journal, run, lease, and scope checks remain in place before the callback. - Historical contradictory or quarantined state still requires the existing verified recovery path. This change does not delete those records. - No migration or provider configuration change is required. ## Model Used - OpenAI Codex, based on GPT-6. The runtime does not expose a more specific model ID or context limit. Used reasoning, repository tools, code execution, and tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2035118526 |
fix(workspaces): preserve archived data during remote migration (#18826)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote workspace storage lets the web server run without a persistent volume. > - Existing deployments also contain archived and historical task workspaces. > - Retention can remove their folders while preserving encrypted restore points or Git reconstruction records. > - The importer rejected those records and only accepted the task's current workspace. > - This pull request imports verified saved data without reopening or rebinding tasks. > - The benefit is preservation of existing work during an explicit storage migration. ## Linked Issues or Issue Description Refs #18403 and #17472. **What happened?** An explicit remote-workspace import rejected archived records, historical task bindings, and removed folders with valid retained data. **Expected behavior** Import the recorded files or exact Git commit. Preserve archived state and the task's current binding. Refuse missing, expired, corrupt, or changed sources. **Steps to reproduce** Archive a task directory with a restore point, remove its folder, and run the remote-workspace import. Repeat with a Git restore point containing an unpushed commit and staged and unstaged edits. The previous importer reports an inactive workspace or missing source. ## What Changed - Preserve lifecycle state and current task binding while importing archived and historical records. Recheck the binding before publication and on resume. - Expose the task-directory owner's validated restore-point reader. Reuse it for ordinary restoration and migration. - Restore archived directories and Git data into private scratch space. Support saved Git recipes and exact legacy branch-deletion receipts. - Recover pruned unpushed Git commits from the encrypted bundle and its explicit prerequisites. - Keep independent snapshot download, restoration, source checks, writer guards, encryption, and atomic publication. - Document the supported import sources and the remaining tenant-enrollment gaps. ## Verification - Before production edits, two new regressions failed on archived status and historical task binding. - Final focused verification: 58 tests pass across remote-workspace migration and isolated task-directory restoration. - Real PostgreSQL, Git repositories, encrypted storage, and restored file bytes cover cold restoration after removing the old folder and source object, pruned unpushed history, index and worktree edits, corrupt/missing/expired sources, binding changes, and source-folder races. - A Git fixture initially lacked runtime ownership metadata. Correcting the fixture made the valid retained-source case pass; no production validation was weakened. - Full repository typecheck and build passed. Full `pnpm test:run` stopped in chat integration shard 2 after two failures: a retry-notice timeout and a subsequent fake-timer error. All four cases in that retry-notice group passed in a separate focused rerun without source changes; the complete suite has not passed locally. All exact-head CI jobs passed. The separate Paperclip Review app check is still queued; all existing review threads are resolved. Server typecheck passed after the review fixes. - Both review findings were reproduced: the real cleanup owner's verified-tip receipt was rejected, and a lost upload reply for a legacy Git source failed with `operation_content_changed`. The fixes accept the owner's scoped receipt metadata and rebuild reconstructed Git indexes from their staged trees. Both regressions now pass, including retained staged and unstaged edits. - Additional coverage restores from a fresh HTTP clone after deleting the original repository, imports a nested local checkout with encrypted settings, and rejects an unapproved nested source without publishing remote authority. - This PR does not claim a completed tenant enrollment or a new zero-downtime deployment proof. ## Risks Imports can fail when old receipts or repository objects are unavailable. They retain their migration journal and leave the original state intact. Operators must resolve pending writers and recovery before importing. Shared physical roots, nested worktrees, and unresolved sandbox recovery require separate migration work. This change does not remove those blockers or alter automatic retention. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. This session does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal task ID or instance-derived details - [x] I have run the focused tests locally and they pass; full typecheck/build passed and the full-suite failure is documented - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
fef86e32e9 |
feat(execution-admission): add project concurrency policy and admission storage (#18746)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Projects group tasks. Agents run tasks as heartbeat runs. > - Today, workspace mode and shared-workspace flags decide whether tasks run at the same time. A project cannot set its own limit on concurrent task executions. > - A correct project limit must count every project that a task uses, including @-mentioned projects. It must also keep the count when a project is archived or deleted, and when a run is admitted more than one time. > - This change cannot go into one pull request with claims and settlement. Thus this pull request adds only the policy semantics and the durable accounting storage that later claims will use. > - This pull request adds a new `execution-admission` module, three tables and the deletion guards. No route writes a cap and no claim path enforces it. > - The benefit is a tested storage contract (constraints, idempotency, rollback, indexed counts, lifecycle preservation) before any behavior changes. ## Linked Issues or Issue Description Refs #2831, Refs #14333 (related agent-level and adapter-level limits; not duplicates). **Subsystem affected** Server execution admission (new module `server/src/modules/execution-admission/`) and the `packages/db` schema. **Problem or motivation** A project cannot limit how many of its task executions are active. The current shared-workspace flags mix resource safety with capacity. A future project limit needs storage that records each admission attempt and every project that the attempt charges. Run IDs cannot be the identity, because a run can be released before dispatch and admitted again. **Proposed solution** Add a separate policy table that keeps "never configured" apart from "explicit no limit". Add admission epochs with one charge row per distinct project. Settlement deletes the charge set in one transaction. Later pull requests integrate claims, settlement, waits and the public control. **Alternatives considered** Columns on `projects` would leak an unenforced cap through every project reader. A project column or a `uuid[]` on `heartbeat_runs` cannot enforce per-project uniqueness or company scope. `budget_reservations` has no project set. A counter would be a second source of truth. ## What Changed - Added `project_task_execution_policies`. No row means never configured. A null maximum means an explicit "no limit". A check accepts only positive maximums. A composite foreign key keeps the project and company together. The row is deleted with its project. - Added `execution_admissions`. Each row is one admission attempt of a run: `(run_id, attempt)` is unique, and a partial unique index allows one unsettled attempt per run. It keeps the context revision, the nullable primary project (provenance, no foreign key) and the settlement identity. A check sets `settlement_id` and `settled_at` together. - Added `execution_admission_project_charges`. It has one row per distinct admitted project, unlimited projects included. Composite foreign keys keep the admission and the project in one company. The project foreign key has no cascade, so a charge cannot disappear with its project. `(company_id, project_id)` is indexed for active counts. - Added migration `0426_absent_guardian.sql` (renumbered from `0423`, then `0425`, as master added `0423`–`0425`). The generated SQL was reordered so the unique indexes exist before the composite foreign keys that need them. Snapshots `0419` and `0420` were pruned to keep five snapshots. - Added the `execution-admission` module (domain → application → adapters): - Domain: policy patch parsing and application (omitted keeps, null clears, partial updates keep other fields, malformed values are rejected), canonical project-set preparation (primary ∪ references, deduplicated, ascending, bounded with an explicit `too_many_projects` result), and the settlement replay decision. - Application: internal `readTaskExecutionPolicy` and `updateTaskExecutionPolicy` commands. No route, tool, CLI or import calls them. - Adapter: the project-set lock (`FOR NO KEY UPDATE`, ascending, one statement), the policy store, and the storage transitions `recordAdmission`, `settleAdmission` and `countActiveProjectCharges`. These transitions are not exported from the module entry point. Claims (later) must add the capacity check and the lifecycle evidence. - Project deletion returns `409 project_execution_admission_pending` while the project has a charge. Agent deletion returns `409 agent_execution_admission_pending` while its runs have an unsettled admission, and removes settled history. Company deletion calls a new module participant before it deletes runs and projects. - Documented the tables, the lifecycle rules and the proposed global lock order in `doc/DATABASE.md`. Added module ownership to `server/src/modules/README.md`. ## Verification - `cd server && npx vitest run src/modules/execution-admission` → 2 files, 26 tests pass (the Greptile finding on an empty primary ID added an assertion to an existing test). The adapter tests use embedded PostgreSQL and cover: - policy semantics, the maximum check and the company foreign key; - charges for every distinct project, a primaryless set and an empty set (no rows); - replay of one admission ID, conflict on a changed record, refusal of a second live attempt and the partial unique index; - settlement of the whole set once, replay, a different settlement, then attempt 2 for the same run; - company isolation of projects, runs and admission IDs, the duplicate-charge key, and rollback of an aborted transaction; - active counts per project, and an `EXPLAIN` that uses `execution_admission_project_charges_project_idx`; - archive keeps charges, `projectService.remove` returns 409, a direct project delete fails on the foreign key, and delete succeeds after settlement; - `deleteAgentDependencies` returns 409 while unsettled, and `deleteCompany` removes only that company's admissions. - `cd packages/db && npx vitest run src/execution-admission-migration.test.ts` → pass. It removes the new tables and receipt, seeds a legacy project and run, applies pending migrations, and compares the existing rows. - `pnpm --filter @paperclipai/db check:migrations` → pass. Five snapshots remain (`0421`, `0422`, `0424`, `0425`, `0426`; `0423` is SQL-only on master). - `node scripts/check-module-boundaries.mjs` → pass. - `packages/db` `tsc --noEmit` → pass. A scoped `tsc` of the module entry points and domain tests → pass. - Existing tests for the changed deletion services → 6 files, 161 tests pass (`resource-lifecycle-events`, `prompt-injection-prevention`, `agent-lifecycle/lifecycle`, `case-deletion`, `project-workspace-root-provenance`, `companies-service`). - Not run locally: the full `server` typecheck. `tsc` and `tsgo` were stopped by the out-of-memory killer on the 7 GB machine (exit 137, also single-threaded). The full `pnpm -r typecheck`, `pnpm test:run` and `pnpm build` were not run locally; CI ran them on head `520b69d553`: all gates pass, including Typecheck, Build and the 12 server test shards. After merging master (`1c9ca5829f`) into head `c088729361`, the migration check, migration upgrade test, module tests, module-boundary check and `packages/db` typecheck were rerun locally and pass; CI is rerunning the full suite. ## Risks - Migration risk is low. It creates three empty tables and indexes. It does not change existing tables or rows. - Behavior changes only when admission rows exist. Nothing writes them yet, so project, agent and company deletion behave as before. - Later claim work must take the project-set lock in the documented order. A claim that takes project locks after issue locks could deadlock with project deletion. - Settlement deletes charge rows, so a settled admission keeps its primary project and context revision but not its full project set. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, embedded PostgreSQL tests). Default context window and reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: nickyleach <331803+nickyleach@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d8a9b3eb0c |
fix(ui): show task identifiers in activity toasts (#18802)
Project the company-scoped identifier for authorized task events and use it in toast titles, fallback bodies, and links. Retain authorized delivery when the optional identifier lookup fails. Co-Authored-By: Paperclip <noreply@paperclip.ing> |