mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 17:51:58 +01:00
45e91f097cdfcb7c7cb7ed879dca47a5a207d856
1640
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
b4a66027b7 |
fix(server): deliver chat completions after nested workspace saves (#18665)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent Chat returns completed worker tasks to the original conversation. > - Delivery must retain its wake receipt when execution gates refuse admission. > - A workspace save can contain a nested snapshot operation. > - The old checks lost the receipt and treated that saved snapshot as unfinished work. > - This pull request preserves the receipt and recognizes a successful owning save. > - The benefit is that completed work can reach the chat through the normal queue. ## Linked Issues or Issue Description Refs #18652. Related: #17206 covers failed finalize recovery. This repair handles a successful finalize with an enclosed snapshot and does not change failed-save recovery policy. **What happened?** A delegated worker finished and saved its result. The original chat received no update. The dependency gate compared operation start times. A nested snapshot started after its enclosing finalize and therefore left a permanent hold. The completion wake also lost its exact idempotency receipt when the queue merged a refused wake into a generic dependency wait. The outbox reported `Completion wake returned no run or durable receipt`. **Expected behavior** A successful finalize covers a completed operation enclosed by that save when company, workspace, issue, and run match. Completion wakes that disable coalescing keep their own durable receipt even when admission waits. Dependency, budget, Stop, and pause gates still apply. **Steps to reproduce** 1. Hand a task from a chat to a worker and make the chat wait on that task. 2. Finish a workspace finalize that starts before its nested snapshot and ends after the successful snapshot. 3. Mark the worker Done and deliver its completion outbox entry. 4. On the parent revision, the dependency stays blocked and the completion receipt is absent. **Paperclip version or commit** Reproduced against `53f480d1b3` with real disposable Postgres rows and the normal queue/outbox services. ## What Changed - Honor `allowRunCoalescing: false` when recording execution waits. - Accept an enclosed successful operation only when the successful finalize has the same company, workspace, issue, and run and covers its whole recorded interval. - Preserve the existing rule that a strictly later successful workspace finalize clears earlier work. - Add failure controls for foreign runs, unfinished snapshots, failed saves, snapshots that finish too late, and later operations. - Exercise the real outbox and queue together. Hold provider dispatch at the existing drain boundary and verify server-recorded Done facts reach the admitted turn. - Document receipt identity and workspace-save coverage. ## Verification - Before repair: the two causal regressions fail and five safety controls pass. The missing exact receipt and empty wakeable-dependent list match the observed failure. - After repair: those seven cases pass without assertion changes. - The full affected set passed 288 tests and failed one existing scheduling test. Its unchanged Node24 baseline also failed. A closed-phase probe showed the fake adapter finished successfully at 3.453 seconds, after the incidental 3-second poll. The test now joins the existing execution promise and retains its status and dependency assertions under the unchanged Vitest timeout. The synchronized full dependency suite passes all 13 cases. - `pnpm -r typecheck` and `pnpm build` pass on pinned Node24/pnpm9. `pnpm test:run` is running. The combined real outbox/queue case passes. Fresh Greptile is 5/5 on `d7cee24b7032c7fdf171bfd1246cd90d41db1cba`. Exact-head normal CI and required `ci / verify` and `ci / e2e` pass. - The original scheduling failures on Node26 and Node24, the unchanged-source baseline failure, and the closed-phase probe are retained in the validation record. No model, deadline, production assertion, or paid attempt was retried to change the failure grade. - Current-head CI attempt1 passed all product and workspace coverage but failed the unchanged serialized retry-worker case at `issue-scheduled-retry-routes.test.ts:606` (first batch not observed before the poll deadline;410 other shard tests passed). The exact unchanged case passes locally on Node24 with real Postgres. The one failed-jobs-only retry passed on the same head; the original grade is retained. - The full local command completed all1828 chat integration tests. Its general stage remains running and has retained failures in unchanged `workspace-runtime.test.ts` (30-second local port-conflict fixture) and `muse-control.test.ts` (current binding authentication fixture). These are not counted as passing local full-suite evidence. Their current-head hosted coverage passed. - The separate advisory Paperclip Review app is paused and has no code finding. Its in-progress state is preserved and is not counted as a pass; it is not a required repository status check. - The live Cloud completion journey remains failed and unqualified. It must pass at the deployed fixed revision before daily-plan admission. No QA oracle, model, budget, or run limit changes are included. ## Risks - The workspace coverage query must not clear a save from another owner or new work after finalization. The guard checks the full interval and exact owner identity, with negative controls. - Refused noncoalescible automatic wakes retain separate receipts. Replaceable automatic signals still merge. No new polling or provider retry path is added. - No migration or API change. Deployment and live acceptance are separate remaining gates. ## Model Used - OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The exact deployment variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
cd12432947 |
fix(runner): recognize verified task document evidence (#18661)
## Thinking Path > - Paperclip manages AI agents and keeps their results attached to tasks. > - Native tools return links to saved task documents. > - Completion feedback asks agents to include those links. > - The file-delivery validator treated every such relative link as a workspace path. > - A valid document and CSV result could therefore fail completion when it cited the supplied document link. > - This change derives document links from the same verified publication projection for guidance and validation. > - Requested files still need their own verified delivery evidence. ## Linked Issues or Issue Description **What happened?** The native document tool and completion feedback produce a link such as `/EX/issues/EX-1#document-output`. When a task also requests a downloadable CSV, citing that link causes completion to reject it as a workspace-only file. This occurs even when the document is published and the file has a valid current-run delivery receipt. **Expected behavior** Accept a server-derived document link only when it names the current company/task and a currently attached revision with a matching native publication receipt. Keep the separate requested-file requirement and all foreign, stale, missing, and workspace-file guards. **Steps to reproduce** 1. Save a task document through the native `write_document` tool. 2. Publish a CSV through `register_deliverable`. 3. Ask completion feedback for the final-response guidance. 4. Cite its returned document link alongside the file receipt in the next completion report. 5. The old validator rejects the valid document link as a workspace-only file. **Paperclip version or commit** Current source checkout. The regression exercises the real native document producer and completion validator. **Deployment mode** Local source verification on macOS arm64 with Node 24.22.1, pinned pnpm 9.15.4, locked dependencies, temporary PostgreSQL, and a controlled remote file-reader boundary. No live provider or hosted deployment is included. ## What Changed - Derive the canonical document href in the existing company/task/current-revision publication query. - Use that same verified href in completion guidance and reference validation. - Keep file registration and hash/size receipt requirements separate from document evidence. - Cover evidence, artifact, and criterion references; foreign companies/tasks; bogus keys; local paths; and lost current-revision publication proof. - Exercise actual native document writes and CSV registration, including attachment download query and fragment links. - Update the execution contract and the document feature/access maps. ## Verification - Red before the production edit: native completion generated a real saved-document link, then rejected the same link alongside a valid published file. The failure was the workspace-only-file guard. - Green: 210 tests across native completion feedback, explicit output validation, and the existing native file-handoff suite pass. These use real PostgreSQL and native tool receipts; the remote file reader is controlled. - A saved document link alone still fails the explicit CSV requirement. A document whose current revision lacks its publication receipt is rejected. Foreign task/company links, missing keys, malformed links, and workspace paths remain rejected. - Full workspace typecheck, server typecheck, and production build pass. - `node scripts/check-maps.mjs`, `node scripts/check-module-boundaries.mjs`, and `git diff --check` pass. Map checks establish structure only. - Full `pnpm test:run` is running separately. There is no full-suite verdict yet. CI and review are pending. - Hosted acceptance and any recovery of existing tasks remain separate steps. This PR does not change final-response recovery or scheduling. ## Risks - An overly broad URL exception could admit foreign or unpublished output. The validator accepts only exact hrefs from the authenticated publication projection. - Document evidence could hide a missing requested file. A separate regression keeps that delivery gate effective. - The shared projection adds current task/company joins. Existing document feedback and publication tests cover those consumers. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with reasoning, code editing, shell execution, and automated tests. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
aa6c9c3f25 |
perf(storage): overlap streamed S3 multipart uploads (#18659)
## Thinking Path > - Paperclip manages AI agents and preserves their work. > - Large workspace and skill objects use the shared S3 multipart uploader. > - The uploader waits for each part acknowledgement before it sends the next part. > - Storage request latency therefore accumulates once per part. > - This change overlaps up to four streamed part requests. > - One source reader preserves byte order and backpressure. > - Failure cancels and settles active requests before cleanup. ## Linked Issues or Issue Description Related: #18082 added the shared streamed multipart uploader. #18644 fixes temporary disk use during workspace capture. It changes a separate stage. **What happened?** The shared S3 multipart uploader sends large object parts in sequence. The code waits for each `UploadPartCommand` before it consumes the next part. A controlled HTTP test confirms that only one complete part reaches the endpoint while its acknowledgement is held. **Expected behavior** Upload independent parts concurrently within a fixed limit. Preserve exact bytes, ordered completion receipts, source backpressure, cancellation, and full snapshot verification. **Steps to reproduce** 1. Upload a large stream through the S3 multipart provider. 2. Hold each successful part acknowledgement at a disposable storage endpoint. 3. Observe that the old uploader sends only one complete body until its acknowledgement returns. 4. Repeat with this change. Four complete bodies can reach the endpoint before the first acknowledgement. **Paperclip version or commit** Current source checkout. The affected uploader was introduced in #18082. **Deployment mode** Local source verification on macOS arm64 with Node 24.22.1, pinned pnpm 9.15.4, and locked dependencies. The HTTP tests use synthetic data and a disposable loopback endpoint. ## What Changed - Send up to four part requests at once through bounded streams. - Keep one source reader and retain the existing adaptive part-size policy. - Store successful receipts by part number before completion. - Cancel a stalled asynchronous source read without waiting for the source to produce more bytes. - Preserve the first error. Cancel sibling requests and settle them before multipart abort. - Cover exact bytes, out-of-order acknowledgements, backpressure, the concurrency limit, source errors, cancellation, and the real S3 HTTP client. - Document the transfer bound. Snapshot encryption, publication, and full object verification retain their existing behavior. ## Verification - Failing before the change: the held-acknowledgement regression expected four complete part bodies and received one. The same regression now passes. - All 15 focused multipart and storage-diagnostic tests pass. The real AWS S3 SDK sends four exact 8 MiB bodies through a disposable HTTP endpoint before any acknowledgement. Completion retains source order. - Controlled transfer comparison: one synthetic 64 MiB stream, eight 8 MiB parts, real S3 SDK and loopback HTTP, each acknowledgement delayed 100 ms. Original: 904 ms, peak one request. Fixed: 247 ms, peak four. Both transfers verified every part hash and completion order. This is local boundary evidence, not a hosted latency claim. - A second safety regression reproduced cancellation waiting indefinitely on a stalled asynchronous source. Abort-aware reads now settle within the existing bounded cleanup budget. - All nine existing encrypted snapshot storage tests pass after correcting missing library aliases in the local embedded-Postgres dependency. The failed setup attempt had no test verdict. - Full workspace typecheck and production build pass. The rebased server typecheck also passes. - The first full `pnpm test:run` attempt stopped during chat test collection with `No test found ... in lines ...` before a suite verdict. The chat test and line-selection scripts are unchanged from master. A retry after local Postgres dependency setup passed its first chat shards, then was stopped to avoid redundant host load after current-head CI passed every type/build/general/serialized/runner group. The unrelated narrow-banner E2E job failed its first attempt and passed one bounded retry; all current-head GitHub CI jobs are now green. There is no full local verdict. The separate Paperclip native review concluded neutral after reporting that work was paused; it supplied no substantive code review. - Commands: `pnpm -r typecheck`, `pnpm build`, and `pnpm --filter @paperclipai/server exec vitest run --configLoader runner src/__tests__/storage-s3-multipart.test.ts src/__tests__/storage-s3-failure-diagnostics.test.ts src/services/remote-workspace-snapshot-storage.test.ts`. - Deployment and an end-to-end hosted acceptance run remain separate steps. ## Risks - Four requests per upload increase concurrent storage traffic. Multiple simultaneous uploads multiply that bound. - Shared source consumption could reorder bytes. One producer and exact-content tests protect that boundary. - Failed siblings could outlive multipart cleanup. Cancellation and settlement tests protect cleanup order. - Capture and full read-back verification still take time. This change only targets serial multipart latency. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with reasoning, code editing, shell execution, and automated tests. The context-window size is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
49b3804830 |
fix(connections): load permissions and show AI account identity (#18653)
## Thinking Path > - Paperclip manages AI agents and their access to saved connections. > - The Permissions page loads account details and permitted agent choices. > - Company-wide AI accounts repeat retained installation checks for every candidate agent. > - This makes target checks grow with the square of the roster size. > - The browser waits without a deadline, and connection names do not identify the provider login. > - This pull request batches direct AI checks and bounds loading with recovery controls. > - Account details load separately and show a provider email or an explicit profile-permission limit. ## Linked Issues or Issue Description **What happened?** Opening a saved AI account can leave Permissions on “Loading AI account…” for minutes. The account itself can be connected. The existing capability batch excludes AI connections, so a company-wide install repeats checks for every installed agent for every candidate. Saved names and Paperclip ownership also do not establish which provider login a credential uses. **Expected behavior** Direct AI accounts reuse retained-target checks within one descriptive request. The page shows loaded account controls or a bounded failure with Retry. Permissions shows the provider email and organization when the selected credential permits profile access. An unavailable profile must not block maintenance or disconnect a working inference credential. Writes still authorize current permissions. **Steps to reproduce** 1. Save an AI account in a company with several agents. 2. Install the account company-wide. 3. Open its Permissions page. 4. Delay the account-list or grants response to exercise the loading failure. The maintained benchmark reproduces database work with synthetic local data. Refs #17945, which added batching for non-AI connections. Searches found no open PR for this AI-account loading fix. ## What Changed - Include direct AI accounts in the existing board capability batch. Referenced pool members, delegated actors, complex selectors, and mutation authorization keep their existing evaluator. - Add opt-in per-caller GET deadlines to the API client. A timed-out caller detaches the stuck coalesced request so Retry starts a fresh fetch. Other callers retain their own cancellation and results. - Apply a 15-second deadline to AI account lists and grants reads. The permissions view retries transient failures once and reports other failures immediately. - Share the parent Permissions grants query with AI account details. Keep loaded controls visible after a background error and offer Retry. - Add a grant-scoped identity GET and OpenAPI contract. The service checks current membership and credential audience before it reads the selected credential. It projects only the provider email and organization; provider IDs, tokens, and raw errors stay private. - Show account details independently of maintenance controls. Anthropic profile denial has an explicit message and preserves connected status. Transient profile failures have a separate Retry action. Observations are cached for five minutes under the company, connection, grant, and grant revision. - Extend real PostgreSQL and authenticated HTTP coverage, client/component regressions, map guidance, and the existing benchmark's AI mode. - Address the Apex file-size finding by moving gallery identity lookup into the existing app display helper module. `AppDetail.tsx` is 984 lines; lookup behavior is unchanged. ## Verification Local source checkout on macOS, Node 25.6.1, pnpm 9.15.4, and disposable PostgreSQL 18.1. Provider calls use fixtures. No hosted deployment or provider qualification is claimed. - Passed `pnpm -r typecheck` and `pnpm build`. - Passed all 87 tests in `connection-management.integration.test.ts` and all 33 tests in `connection-install-capabilities.test.ts`. - Passed 196 AI connection, identity-provider, and OpenAPI tests: `ai-connections.test.ts`, `ai-account-identity.test.ts`, and `openapi-routes.test.ts`. - Passed 107 focused UI/client tests: `client.test.ts`, `ManagedAiConnectionDetails.test.tsx`, `AiConnectionAccountDetails.test.tsx`, and `AppDetail.test.tsx`. - After the Apex extraction, passed UI typecheck/build and 106 component tests across `AppDetail.test.tsx`, `AppNotConnected.test.tsx`, `ManagedAiConnectionDetails.test.tsx`, and `AiConnectionAccountDetails.test.tsx`. - The authenticated HTTP tests compare complete grants responses across human roles, for tool and AI connections. They verify a real install mutation succeeds before revocation and fails afterward. - A company-wide personal AI fixture checks 20 targets instead of 400. Pool-member permission loss still denies the complete pool description. - UI states covered: shared cached grants, stuck account/grants fetches, one automatic retry, terminal failure, manual recovery, authorization failure, background failure with retained controls, and unmount cancellation. Client tests also cover stalled response bodies and independent coalesced callers. - Identity coverage includes actual bearer-key/JWT reads, private-user and cross-company denial before provider access, revoked credentials, provider 401/403/429/500, and grant-change isolation. A real local HTTP fixture stalls the response body, verifies the shared 15-second deadline, and keeps unrelated HTTP requests responsive. The test shortens the native deadline to 100 ms. Profile permission denial leaves the saved account connected. - Passed `pnpm check:token-gates`, `node scripts/check-maps.mjs`, `node scripts/check-module-boundaries.mjs`, and `git diff --check`. - The full local `pnpm test:run` started with the loading-only change. All six chat shards passed. It recorded one failure in the unchanged `heartbeat-retry-scheduling.test.ts` quota-rejection case. That exact case passed a focused rerun on the final head. The initial failure cause remains unconfirmed and its local log is retained; this is not a full local-suite pass. The redundant local run was stopped with SIGINT after final-head CI completed successfully. - [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/41515556725) passed on `ead8d276d4a2023614fff54b3e9a858f28c3178b`: typecheck, build, all 12 server shards, all 3 chat shards, workspace suites, all 9 serialized suites, all 8 end-to-end shards, Runner checks, and the aggregate verify gate. Security checks passed. Optional Storybook checks were skipped by workflow policy. GitHub reports the branch mergeable without conflicts. - Apex reviewed head `ead8d276d4a2023614fff54b3e9a858f28c3178b` at 5/5, confirmed the extraction resolves its earlier file-size finding, and has no unresolved review threads. The existing benchmark ran for the loading-only change now recorded in commit `5f725b36fa52002545241a5a8b7a10602f0cc0ce`, with `--purpose=ai --baseline-ref=026fd9e3452`. The identity follow-up leaves that batch source unchanged. Two warmups and five alternating samples exercise each of eight fixture shapes. Complete positional capabilities, reasons, and operations match the baseline in every sample. | AI fixture | SQL calls before → after | Target reads before → after | | --- | ---: | ---: | | 1 agent, company install | 9 → 9 | 1 → 1 | | 32 agents, company install | 5,248 → 164 | 1,024 → 32 | | 100 agents, company install | 50,400 → 504 | 10,000 → 100 | | 100 agents, partial install | 13,175 → 503 | 2,575 → 100 | These counts measure the real service over local PostgreSQL. They do not predict hosted page latency. To reproduce the standalone benchmark, use a checkout of commit `5f725b36fa52002545241a5a8b7a10602f0cc0ce` and run the command below. Its isolation check intentionally rejects the later identity follow-up's additional runtime changes. ```sh node --import ./server/node_modules/tsx/dist/esm/index.mjs scripts/benchmark-connection-install-capabilities.ts --purpose=ai --baseline-ref=026fd9e3452 ``` Run the final-head regression suites from the repository root: ```sh cd server pnpm exec vitest run src/__tests__/connection-management.integration.test.ts src/__tests__/connection-install-capabilities.test.ts src/__tests__/ai-connections.test.ts src/services/ai-account-identity.test.ts src/__tests__/openapi-routes.test.ts cd ../ui pnpm exec vitest run src/api/client.test.ts src/components/ai-connections/ManagedAiConnectionDetails.test.tsx src/components/ai-connections/AiConnectionAccountDetails.test.tsx src/pages/apps/AppDetail.test.tsx ``` ## Risks Descriptive permissions reuse retained facts within one request. A concurrent change can leave a displayed choice stale; the next request and every write still authorize fresh. Pools with referenced members retain their existing cost. The account and grants clients now bound each caller's wait. A browser abort does not cancel an authorization read already admitted by the server. The independent profile request uses a fixed Anthropic origin, a bounded body, no redirects, and a 15-second deadline. Inference-only credentials can return no email; that is reported without guessing from a Paperclip owner or changing connection health. Profiles are fetched only for the caller's authorized selected grant. No ambient CLI identity, persistent identity cache, background polling, or billing-identity change is added. Other API reads keep their existing behavior. No migration is required. Staging has not been redeployed. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh reasoning, code editing, execution, and test tools. The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
33fc7fcddc |
fix(db): clean up owned PostgreSQL test servers (#18627)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Database, server, and CLI tests use real PostgreSQL fixtures. > - Each fixture must own its server until PostgreSQL has stopped. > - The old teardown returned after five seconds even when its server remained alive. > - A startup failure could then mark the host unsupported and skip database coverage. > - This pull request waits for owned shutdown and makes supported-host startup failures fail collection. > - A shared-server command reduces process creation while keeping each fixture's database isolated. ## Linked Issues or Issue Description Refs: #7756 for related development-process ownership. That development-server issue remains outside this test-fixture fix. Related PR: #16799 collects macOS startup diagnostics. This PR changes test shutdown and failure behavior. **What happened?** Repeated test runs can exhaust macOS PostgreSQL semaphore resources. The fixture abandoned slow shutdown and swallowed stop errors. The support probe converted ordinary startup failures into skipped integration coverage. **Expected behavior** Tests must stop their PostgreSQL servers, retain files if stop fails, and report resource exhaustion as a failure. Fixtures on a shared server must not close another database's clients. **Steps to reproduce** Start a fixture whose PostgreSQL stop exceeds five seconds, or make a supported-host startup fail with semaphore exhaustion. The former helper returned from teardown or reported unsupported coverage. The new regression tests exercise both paths without exhausting host resources. **Paperclip version or commit** Source checkout based on current master. **Deployment mode** Local source development and CI tests. ## What Changed - Add explicit ownership for test PostgreSQL children. Wait for fast shutdown, then request immediate PostgreSQL shutdown if needed. Keep cleanup idempotent and fail when exit cannot be confirmed. - Publish reusable templates only after clean shutdown. Do not publish a crashed or immediately stopped cluster. - Stop owned children on POSIX worker signals and process exit. Preserve test directories when shutdown fails. Remove conflicting dependency exit hooks. - Retry port collisions only. Fail collection for resource exhaustion, unavailable configured servers, and other startup failures on supported platforms. Preserve skips for explicitly unsupported platforms. - Move administrative-server support into the common fixture used by database, server, and CLI tests. Close clients at database scope and migrate cloned templates before use. - Add a directly invoked shared-server runner to own one test server and run suites against separate disposable databases. Preserve child exit status and support an existing disposable server. - Unify cancellation: stop and wait for the test process group before PostgreSQL teardown. Add nested-process coverage, real worker-exit coverage, shutdown regressions, shared-database isolation checks, and setup documentation. Replace ignored heartbeat completion polls with the existing execution drain before asserting persisted outcomes. - Finish owned cluster-directory cleanup after signal-driven cancellation. Preserve prepared templates across shared-fixture tests. Remove stopped directories asynchronously, using native removal on POSIX with a Node fallback. - Align database and CLI fixture hook budgets with the existing 30-second server budget. This lets the ten-second shutdown protocol finish and report errors. The idle-sleep suite uses the existing 90-second fixture budget for its unusually large final directory cleanup. ## Verification - The semaphore-exhaustion regression failed against the old helper because it resolved to `supported: false` instead of rejecting. - All 37 focused tests pass on the final runtime code (`e4de00ab40`). They include real worker exit, SIGTERM, nested-process SIGINT/SIGTERM cancellation with PostgreSQL available during child cleanup, failed template shutdown, unsupported platforms, stale SQL templates, and directory removal. A child exiting with status 7 makes the shared-server runner return status 7 after cleanup. - Full local `pnpm -r typecheck` and `pnpm build` pass on the final runtime code (`e4de00ab40`). Subsequent changes only adjust server-test lifecycle waits, the idle-sleep cleanup budget, and documentation; the server typecheck also passes with those final tests. - A fresh shared-server run on `90b65fd8ad` passed all 68 database files and 297 tests. Earlier preparation runs had two ten-second teardown timeouts and a stale-template failure; the final helper migrates cloned templates and permits the full shutdown budget. These earlier failures remain accounted for here. - The local full repository attempt stopped on a Discord receiver-lease test after 338 passing chat tests. That exact case passed in isolation. This full-suite attempt is not a pass. - On `90b65fd8ad`, [server shard 6](https://github.com/paperclipai/paperclip/actions/runs/41496495253/job/125452949369) and its one unchanged retry reported two 15-second cached-input heartbeat timeouts and a 30-second idle-sleep teardown timeout. All ten cached-input cases then passed locally with their original deadlines. A local parallel run of the full files reported eight behavioral/hook failures and two suite teardown timeouts (281 tests passed). - Serial idle-sleep runs passed all 225 test assertions but overran the 30-second final hook. Tracing measured client closure in 4 ms, PostgreSQL exit after 5 seconds, and synchronous directory removal taking another 35 seconds. Asynchronous Node removal and native removal also exceeded that hook under the full workload. Cleanup now remains asynchronous and awaited, and only this heavy suite's final hook uses the existing 90-second fixture budget. No PostgreSQL shutdown deadline or test assertion was weakened. The serial run then passed the complete idle-sleep file, including all 225 tests and teardown. Its heartbeat file passed 63/64 cases, exposing an ignored three-second completion poll. After replacing that poll with an execution drain, another timer case exposed the same pattern (63/64 passed). All remaining ignored completion polls now await execution drain without changing test deadlines. The complete heartbeat file then passed all 64 tests. The intentional asserted lock-contention poll is unchanged. The final server typecheck passes, and a process audit found no PostgreSQL server left by the completed local runs. - On final head `2cf316d3f571e4516890cba743ecffeb9703cce3`, the previously failing server shard 6, database lane, full typecheck, and build passed. The [first CI attempt](https://github.com/paperclipai/paperclip/actions/runs/41506358136) then lost self-hosted runners in the canary and serialized-server lanes, with additional jobs cancelled. The server-2 lane had one 30-second timeout in the unchanged workspace-recovery HTTP Retry test (2,282 other tests passed). That exact test passed locally with its original deadline (28.78 seconds including imports). One unchanged retry of only failed and cancelled jobs completed successfully on the exact final commit (workflow attempt 2). The workspace-recovery timeout case and every interrupted lane passed; successful first-attempt jobs were retained. The additional Paperclip Review check remains queued, waiting for its assigned agent; it has no unresolved GitHub review threads. Greptile completed review on this exact final head at 5/5, with zero unresolved review threads. All four original review findings and their threads are resolved; additional Windows shutdown, cancellation-directory, and template-reuse findings are fixed. - `pnpm check:module-boundaries` and `git diff --check` passed. Post-run process checks found no PostgreSQL servers left by the completed verification runs. ## Risks - Immediate PostgreSQL shutdown can require crash recovery. This fallback applies to disposable test clusters and runs only after fast shutdown times out. - No process can handle `SIGKILL` or machine power loss. Host resource cleanup remains a separate backstop. - Directory cleanup on cancellation applies to the shared-server runner's own cluster. Directly terminated test workers stop their PostgreSQL process but retain disposable files for inspection. The process-exit guard cannot await file removal. - The two-second `pg_ctl -W` deadline bounds signal dispatch. It is separate from the five-second and thirty-second server-exit waits. A control-command failure fails cleanup and retains its directory; it does not report successful cleanup or force-delete live files. - Shared fixtures require a disposable PostgreSQL server and a role that can create databases. The directly invoked shared command is opt-in; the existing `pnpm test` command remains available. - These changes do not stop unrelated development servers or delete host-wide IPC resources. Clean shutdown uses the owned `pg_ctl` binary on every supported platform. Its real control path was tested on macOS. A Windows host was not available for live verification; POSIX signal tests skip on Windows. ## Model Used - OpenAI Codex, based on GPT-6. The runtime does not expose the exact serving variant or context-window size. The agent used reasoning, shell execution, code editing, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
946735d03a |
Persist task sorting in URLs and show newest routine tasks first
Save task sort choices in URLs across task views. Open routine tasks by creation time, newest first, with stable sorting before pagination. Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> |
||
|
DottaandPaperclip
|
5a07c30c1a |
fix(workspaces): reduce temporary disk use during remote capture (#18644)
## Thinking Path > - Paperclip manages AI agents and preserves their completed work. > - Native remote finalization must save workspace files before it releases the writer. > - Portable capture copied the working tree and Git history, repacked the history, and then wrote an archive. > - These temporary copies can exhaust a small sandbox after normal development work. > - A completed provider turn then fails finalization and file preservation repeats the same failure. > - This change streams checked working files and keeps only an independent Git copy in scratch. > - The save still rejects changed sources and incomplete output. ## Linked Issues or Issue Description Related: #18403 added portable remote workspace storage. #18172 handles disk pressure in Daytona directory transfer. This failure occurs earlier, while the portable archive helper creates its source archive. That earlier repair does not cover this path. **What happened?** A completed native run failed workspace finalization three times. Its file-preservation retries also failed. In a retained 10 GB Linux sandbox with about 1.5 GB free, the deployed helper failed during `git repack -a -d` with `Out of diskspace`. The working files used about 603 MB, and Git used about 444 MiB. **Expected behavior** Capture a coherent workspace when free space can hold one full archive plus an independent Git copy. Preserve working files, staged changes, unpushed commits, and existing recovery gates. **Steps to reproduce** 1. Use a disposable Linux sandbox with a Git repository and generated working files. 2. Leave enough free space for an archive and a Git copy, but not a second working tree and another repack. 3. Run the portable capture helper. The old helper fails while creating temporary data. 4. Run the fixed helper against the same source and free-space limit. Verify each archived file hash. The small regression uses real file serialization and a controlled filesystem capacity failure at archive creation. It failed with `ENOSPC` before the production edit and passes afterward. **Paperclip version or commit** The observed staging server served `42c64d4ec4b9b0898556fcdf6ba304c4285a7819`. The repair is based on `a92f616cd0`. **Deployment mode** Hosted staging. Linux x64, Node 25.9.0, Daytona sandbox. Local tests use a macOS source checkout. ## What Changed - Stream working files from checked file handles. Keep only normalized Git metadata in scratch. - Remove the second repack after a mirror clone that already uses `--dissociate` and `--no-hardlinks`. - Check each streamed file's hash, size, mode, and filesystem identity. Recheck the source inventory and Git state before returning a receipt. - Remove incomplete archives after failure or cancellation. Keep pre-existing archive files intact. - Add space-limit, concurrent-change, cancellation, and existing-file safety cases. Update execution semantics and workspace verification recipes. ## Verification - Red before the production change: the capacity regression failed with `ENOSPC`. Green afterward: 54 archive and transport tests pass. - The existing tests restore ordinary, linked, and borrowed Git repositories after their sources disappear. They retain staged, unstaged, and unpushed work. Credential and path guards also pass. - Live comparison on the same retained sandbox: the deployed helper failed after 19.4 seconds with Git disk exhaustion. The fixed helper completed in 35.3 seconds. It produced a 1,081,872,803-byte archive with 24,111 entries. An independent reader checked every archived file hash and the manifest digest. The source commit stayed unchanged. Diagnostic scratch was removed. - This live probe exercised the changed capture boundary. It did not deploy a server, upload a production snapshot, finalize the failed run, or replay its provider turn. Full task recovery still needs rollout and the normal export recovery path. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Current-head CI passes: [PR workflow](https://github.com/paperclipai/paperclip/actions/runs/41499985180), commit `d42700f7f4b0bef14a5969a44c2da0b2513e0ed2`. Greptile rates the same head 5/5 with no actionable findings or unresolved threads. - The independent Ponytail advisory also rates this exact head 5/5 with no findings. Its structured check remains queued because the automatic request was bound to the earlier head. A [fresh current-head review request](https://github.com/paperclipai/paperclip/pull/18644#issuecomment-6585296011) was accepted and queued. Its predecessor has a successful workspace export but remains active in finalization, so the new review cannot start yet. This review-service publication gate must complete before claiming every check is green. - `pnpm -r typecheck` and `pnpm build` pass. The final server-only typecheck also passes after the test additions. - Supported-runtime verification on Node 24.11.0: all 54 archive/transport tests and all 46 existing workspace lifecycle tests pass. The lifecycle suite uses a disposable real Postgres database and local filesystem transport with encrypted snapshot storage. It checks save/restore after losing the original allocation, ownership, retention, and file continuity. This is separate from the live Daytona capture probe. - The first extra lifecycle attempt could not start its local Postgres binary because the installed Darwin package lacked library aliases. Local dependency setup was repaired before the passing run. No production or repository code changed for that setup. - The full local `pnpm test:run` on the host default Node 26.4.0 stops in chat-test collection with `No test found ... in lines ...`; it has no suite verdict. A clean retry repeats the failure. The unchanged `a92f616cd0` test wrapper reproduces it with the same installed dependencies, after correcting an initial baseline setup that lacked built package entries. This is a local test-runner limitation, not a full local pass. Current-head CI remains the full-suite gate. - One focused run timed out while deeply comparing 4 MB buffers under concurrent build load. Complete SHA-256 comparisons retain the content check without that assertion cost. The capacity case then passes in under one second, and the final 54-test run passes. The disk allowance and production code did not change. ## Risks - Streaming reads the live source instead of a copied working tree. File identity, content hashes, mode checks, the final inventory read, and Git signatures must reject concurrent changes. Regression tests exercise these checks. - Capture still needs space for the full archive and normalized Git metadata together. It cannot export from a completely full disk. - Archive framing and restore validation stay compatible. The change has no database migration or new execution authority. - The incident remains unrecovered until the fixed server performs normal workspace finalization. No deployment or merge is included in this PR. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code editing, shell tools, browser diagnostics, and automated tests. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
026fd9e345 |
fix(server): keep controller identity independent of Git probes (#18649)
Read the existing boot timestamp without synchronous Git diagnostics. Keep the voice-retention precision test clock fixed. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a92f616cd0 |
fix(db): attribute bounded captures to execution owners (#18634)
## Thinking Path > - Paperclip manages agent work through shared server and database services. > - High query volume can spend main-thread time on SQL construction and result decoding. > - The existing bounded response capture counts rows and bytes, but it cannot name the operation that executed a query. > - Bundled callers can be missing, and measuring every parsed JSON value adds work to the diagnostic. > - This pull request carries a semantic execution owner into each measured query and bounds response sizing. > - Operators can rank query volume by owner before choosing a performance fix. ## Linked Issues or Issue Description Refs #17504, #17522, #17537, #17564. **What existing behavior does this improve?** The opt-in `PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW` diagnostic. **Subsystem affected** Database diagnostics and existing server execution boundaries. **Current behavior** The capture groups by table, operation, and builder caller. It cannot distinguish execution owners that reuse a prepared query. Bundled deployments need source maps for caller locations. The capture visits every returned value and serializes parsed JSON again to estimate its size. **Proposed behavior** Version 2 captures use fixed semantic owners for HTTP requests, delivery queues, execution phases, and budget work. They count completed calls, returned rows, errors, and completion wall time. They sample successful results 1, 17, 33, and so on, with fixed cell and string/byte limits. Complex values remain explicitly unmeasured. Semantic ownership works without source maps. **Reason and benefit** A short census can identify which operations issue the most queries or return the most rows. It preserves each query's execution owner through pooled connections, concurrent operations, and prepared-query reuse. It reduces diagnostic work without claiming per-query CPU attribution. **Breaking changes** Opt-in capture output moves from `entry.v1`/`summary.v1` to `entry.v2`/`summary.v2`. Sampled primitive byte lower bounds replace all-result byte estimates. Consumers must use the new fields and must not extrapolate them into a total. There is no database migration or application API change. ## What Changed - Added `withDatabaseOperation(owner, action)` to the DB package. The server owns semantic labels. Queries retain execution ownership in their completion closure. - Added fixed owners at existing HTTP, delivery, budget, and execution timing boundaries. Delivery work replaces the context of the producer that woke it. - Added complete call/row/error counters and completion wall durations within the existing bounded window. Durations include waits and are not CPU measurements. - Sampled and bounded primitive-value sizing. Complex and oversized values remain unmeasured. Known owners skip repeated stack extraction. - Preserved the deadline, group limit, paced output, query outcomes, and existing coverage exclusions. Updated the operator documentation. - Added real PostgreSQL owner-isolation and prepared-query tests, server scope tests, and plain bundled-runtime proof. - Moved projection-test and benchmark full-response sizing into an explicit test helper. Existing payload-size thresholds remain unchanged; these synthetic checks do not use the production sampler. ## Verification - Final-commit DB capture suites: 43 tests passed in 13.86 seconds against the actual PostgreSQL driver, including concurrent owners, prepared reuse, transaction failure, a dedicated client, sampling, and a query crossing the deadline. DB typecheck passed. - Server attribution and existing delivery/execution/budget suites: 53 tests passed. Final concurrent HTTP GET/JSON POST scope refinement: 5 tests passed. - Runtime bundle suite: 8 existing tests passed. The new no-source-map owner test passed after its fixture was corrected to use a driver thenable. An ordinary native Promise can bypass an overridden `.then`; the real PostgreSQL tests separately cover driver integration. - Observer-only experiment: three alternating pairs per case on Node 24.22.2, macOS arm64, 3,000 calls per run after warm-up. For an already-materialized synthetic JSON result, median capture CPU fell from 82.37 to 1.87 microseconds/call. For a 100-row primitive result, it fell from 8.10 to 4.40 microseconds/call. Every result retained identity and all calls were counted. The new diagnostic intentionally measures less byte data; these are observer costs, not database latency or a hosted application speedup. Disabled-case timings were below one microsecond/call and too small for a performance conclusion. - Final lifecycle regression: 28 unit tests and DB typecheck passed. Closing the last capture disables and replaces owner storage; overlapping captures retain it. - Final-commit full recursive typecheck passed in 38.38 seconds and full build passed in 40.81 seconds. The build stamp matches `83ed4ba9cebd68d48929c46f15ab32d5c0b9687b`. Focused server attribution tests also passed 5/5 before the test-helper-only migration. Full test acceptance will use the repository's parallel CI gates; the unsharded local command takes hours. - Verification failures retained: initial sandbox runs could not open local IPC/PostgreSQL listeners; permitted reruns exercised the real services. One old test expected JSON serialization and was updated to assert explicit unmeasured complex values. The first bundle fixture used the wrong promise boundary, as described above. A retained-callback regression also demonstrated that disabling and reusing the same AsyncLocalStorage instance could resurrect an old owner; replacing the instance after the last capture closes fixes that case. - The first CI run found older projection tests consuming the previous full-response fields. Those consumers now use the full-response test helper, preserving their byte thresholds. The two complete affected route suites plus helper tests passed 115/115 in 73.08 seconds. A temporary full-row query negative control then failed the unchanged 2,048-byte limit at 5,000,812 bytes and was restored; five other service suites passed 34/34 and the affected chat cases passed 3/3. All three previously failing CI shards passed on the corrected head. A local skill-list benchmark smoke check completed 255 captured queries with no in-flight queries or overflow; this was integration verification under concurrent test load, not a timing benchmark. - Two final-head runner jobs lost communication with GitHub before reporting a test result. One targeted retry of those jobs and their dependent aggregate passed. All 53 final-head checks passed, with 3 intentional skips; previously passing suites were preserved. - Hosted capture remains pending until the reviewed revision is deployed and a bounded window is explicitly configured. Reproduce the focused checks with Node 24 and the repository pnpm version: ```sh pnpm exec vitest run packages/db/src/response-capture.test.ts packages/db/src/response-capture-postgres.test.ts pnpm --dir server exec vitest run --config vitest.config.ts src/__tests__/database-operation-attribution.test.ts scripts/bundle-runtime.test.mjs pnpm -r typecheck pnpm build ``` ## Risks - Owner context is active while a valid future capture window is armed, up to 30 minutes before accounting starts. No-window and expired calls take the direct path. Diagnostics still add bounded work while enabled. - A nearest nested owner is reported. Broad `http.request` and unattributed groups remain visible; this does not identify every service callsite or SQL shape. - The existing census excludes tagged/raw explicit-execute/streaming paths and internal protocol commands. Its counts are not PostgreSQL transaction totals. - Completion wall time includes queue/database/network/scheduling waits. Concurrent durations overlap. Sampled bytes exclude complex values, large strings, and values beyond the sizing limits. - Activation is startup-only. Operators must choose the window after artifact preparation, allow rollout/warm-up time, and remove the expired setting afterward. ## Model Used OpenAI Codex, GPT-6 family, with code execution, repository inspection, and parallel agents. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5f0bb518b0 |
fix(runner): handle blank assigned-tool schema references (#18637)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native agents discover assigned app tools through a governed
gateway.
> - OpenCode receives search and call tools instead of the full app
catalog.
> - A blank optional schema reference selected schema lookup instead of
search.
> - Agents received an unknown-tool error and could abandon valid app
access.
> - This pull request treats blank references as search and explains
invalid schema requests.
> - Agents can find their assigned tools while the gateway keeps its
access checks.
## Linked Issues or Issue Description
**What happened?**
An agent called `paperclip_search_assigned_tools` with
`{"query":"github","schemaTool":""}`. The server tried to find a tool
with an empty name. It returned `assigned_mcp_tool_unknown` instead of
search results. The agent then reported that its GitHub tools were
missing.
**Expected behavior**
A blank optional schema reference must permit normal tool discovery. A
bad schema reference must explain how to search again.
**Steps to reproduce**
1. Assign an app tool to a native agent that uses on-demand discovery.
2. Search with `{"query":"github","schemaTool":""}`.
3. Observe the unknown-tool error instead of the assigned tool.
**Paperclip version or commit**
Reproduced from source before this fix, on master at
|
||
|
DottaandPaperclip
|
60a99307b3 |
fix(server): settle cleanup after reusable sandbox destruction (#18633)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A web-server replacement must preserve work and let completed cleanup become idle. > - Several task turns can use one remote sandbox through recorded lease handoffs. > - The last owner can destroy that sandbox while an earlier owner still waits for its own termination receipt. > - A completed agent-file cleanup can also retain a retry deadline from a competing callback. > - This pull request resolves confirmed destruction through the existing handoff contract and settles obsolete cleanup retries. > - The benefit is correct cleanup recovery without deleting history or weakening sleep checks. ## Linked Issues or Issue Description Refs: #18554. Related work: #14695 handles warm agent-file ownership; #13291 handles interrupted conversation continuation. This PR fixes cleanup recovery after confirmed final destruction. It does not change provider process reuse or start a new run. **What happened?** After ten qualified staging web-server replacements, a bounded database-idle check failed. Completed native runs still owned pending agent-file cleanup. Normal workspace archive confirmed final sandbox destruction. An earlier lease in the same recorded handoff chain remained unresolved. Two completed cleanup receipts also kept stale retry deadlines. **Expected behavior** Resolve an earlier owner's cleanup only when the complete recorded handoff chain leads to an exact provider-confirmed destruction receipt. Clear a retry when its cleanup is already complete. Keep real or uncertain work pending. **Steps to reproduce** 1. Complete a remote run, reuse its sandbox for a second run, then archive the completed workspace. 2. Recover agent-file collection for the first run after the final lease confirms destruction. 3. Separately, reconstruct a completed cleanup receipt that still has a retry deadline. 4. Observe that both records remain unfinished without this fix. **Paperclip version or commit** Observed on `1a6017d6460267ad89d7e1784e2b521c28d8e9ac`. The changed source was unchanged in master `97c4dc969e13364464765ffbde74955005bf44d3`, which is this branch's base. **Deployment mode** A managed ephemeral web server with PostgreSQL, object storage and a remote sandbox. Regression tests use local PostgreSQL and no live provider or model calls. ## What Changed - Resolve remote termination at read time through the existing strict reusable-lease handoff validator. Require one final owner and a valid destruction receipt. Preserve historical lease rows. - Use the same proof in agent-directory cleanup and remote-stop inspection. Keep stopped, ambiguous, broken and cross-company chains blocked. - Reconcile completed cleanup receipts with stale retry deadlines through the existing deadline worker. Compare the receipt before clearing a deadline. - Add recovery, refusal, restart quietness and concurrent-obligation regressions. - Document the cleanup contract and preserve deployment evidence, failed checks and remaining acceptance work in the existing plan. ## Verification - Both owning service suites passed 169 tests before the final receipt comparison was added. - The seven-case selection passed again after the release-race review fix: `pnpm exec vitest run server/src/__tests__/agent-directory-working-copies.test.ts -t 'stale retry|destroyed successor chain|stale-deadline settlement'`. - Repository-wide `pnpm -r typecheck` and `pnpm build` passed on final review fix `7935d99bed`. The full `pnpm test:run` is still running. Latest-head CI has 55 successful checks, three skips, and one neutral review result. Greptile is 5/5. The Paperclip reviewer’s race finding is fixed; its optional query-deduplication suggestion is documented and the fresh lock-boundary checks are retained. - The reviewer identified an early return after a lost receipt comparison. The strengthened race regression failed before the fix and passed afterward: release now acts on the returned authoritative row, completes the new cleanup obligation, and preserves canonical files. - The first focused attempt failed the new stale-deadline test because the public release guard skipped settlement. That guard was fixed. The original failure is retained. A sandboxed typecheck could not fetch pnpm signature metadata; the normal network-enabled command passed without changing verification policy. - Prior deployed evidence: ten Railway replacements, 12,521 reads, 1,566 verified saves, 34 matching file hashes, no HTTP failures, actual browser reconnects of 1.16–1.29 seconds, and one remote execution retained across replacement. Two harness false alarms were separately adjudicated without redeploying those steps. - That deployment evidence predates this PR. Current-image natural idle, automatic sleep/wake browser return and the isolated production canary remain unverified. Production has not changed. ## Risks - Cleanup authority must not cross a company or allocation boundary. The existing validator checks the entire recorded chain; stopped successors and unexpected owners remain unresolved. - Confirmed destruction permits no-save cleanup. It does not claim that uncollected agent files were saved. - Read-time resolution adds a scoped lease-history query only for expired reusable leases without direct termination proof. No new recurring worker or polling cadence is added. - The deadline update must not erase new work. It requires the same cleanup receipt and preserves concurrent changes. - No schema or public API changes are included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository inspection, code execution and test tools. The precise backend revision and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0710c56842 |
fix: retry saved chat input after a successful run (#18621)
## Thinking Path > - Paperclip manages AI agents and the work they receive. > - Connected conversations save incoming requests before they can run. > - A request can remain queued when the previous turn finishes successfully. > - Task Retry sent that case through the failed-run retry guard, which rejected the successful run. > - This change resumes the saved request through its existing admission owner. > - The operator can continue saved work while preserving the prior result and current permissions. ## Linked Issues or Issue Description **What happened?** Task Retry returned a conflict for a restricted connected conversation when its latest run succeeded and newer input remained saved. A GitHub review can reach this state when a new PR head arrives during the previous review. The prior review cannot publish against the new head, but its completed turn is not a failed run that can be replayed. **Expected behavior** Retry should resume the saved request with its original source identity and current authorization. It must preserve the previous successful result and must not replay that turn. **Steps to reproduce** 1. Start a review through a restricted GitHub endpoint. 2. Send a PR synchronize event while that review runs and retain its deferred input receipt. 3. Let the original run finish successfully with the task still open. 4. Click Retry. Before this change, the failed-run guard rejects the request. With this change, the saved input creates one successor through normal admission. Related work: #17725, #16965, #15033. ## What Changed - Select the newest saved input by its canonical chat action and wake receipt. - Store its receipt ID in the task retry intent and resume it through the existing chat owner. - Preserve caller permissions, source authorization, task state, agent state, cleanup waits, and duplicate-click handling. - Exclude saved-input intents from generic board-message replay. Keep board messages in separate queued receipts and allow independently saved input after successful board turns. - Add signed webhook, HTTP endpoint, accepted-result preservation, and refusal tests. Update execution semantics and capability maps. Stabilize the existing Teams lifecycle test with explicit database result ordering. ## Verification - Current-head [CI passes](https://github.com/paperclipai/paperclip/actions/runs/41482727909): 54 successful checks, one neutral review, three expected skips, and a successful Snyk status. Greptile gives commit `53f02e9105` 5/5 with no unresolved review threads. - Reproduced the original HTTP 409 before the fix. Reproduced both review findings with three failing cases before fixing them. - Passed 25 focused and related GitHub retry cases, 52 ordinary queue tests, and three native startup cases against disposable PostgreSQL. Earlier related verification also passed 83 GitHub retry cases and 109 recovery/receipt cases. - Passed `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs`, and the module boundary check. The final server typecheck passed after the review fixes. - The full local `pnpm test:run` attempt stopped on an existing Slack queue deadline assertion with a 1 ms database/host clock difference. That case passed in isolation. A second attempt was stopped when review changes superseded its source. Full local completion is not claimed; all chat, server, browser, build, and runner CI jobs passed on the final commit. - The runner CI job needed one rerun after a two-second startup timing assertion failed. That case and its two related cases passed locally; the rerun passed without code changes. - Live GitHub review publication with this patch is not verified. The signed provider fixture uses mocked GitHub responses. This branch is not deployed. ## Risks - This changes how Retry dispatches deferred external input after a successful run. It relies on the existing inbound owner to recheck source authority before execution. - Tests cover paused connections and agents, changed source bindings, completed tasks, pending cleanup, duplicate clicks, and preservation of the accepted result. - No schema migration or public API shape change is required. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code execution, and browser tools. The exact deployed model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
40b8075916 |
feat(ui): show scheduled task waits as Monitoring (#18624)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task indicators tell an operator whether work is active, waiting, or
complete.
> - An agent can yield until a saved monitor checks CI or another
external result.
> - The task currently keeps the In Progress icon between those runs.
> - Changing the stored status would affect monitor admission and
recovery.
> - This pull request adds a read-only Monitoring display state with a
blue clock.
> - Operators can see and filter scheduled waits while execution keeps
its existing rules.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Task status indicators, task collection filters, board columns, and
search facets.
**Current behavior**
A task that waits for a scheduled monitor shows the same In Progress
icon as active work.
**Proposed behavior**
Show Monitoring with a static blue clock when an agent-owned task has a
saved ordinary monitor and no active run, execution lock, review
participant, or active recovery action. Return to the normal task
indicator when the check starts. Preserve review, blocked, terminal, and
quota-recovery states.
**Reason and benefit**
Operators can distinguish scheduled waits from current agent work
without reading the task history.
**Breaking changes**
None to stored task statuses or execution rules. Task reads gain
optional displayStatus. Collections and search gain a separate read
filter. Existing status filters keep their lifecycle meaning.
Related historical work: #4396 covered a monitoring label exemption.
This change uses the saved monitor and a read projection.
## What Changed
- Add one SQL projection for task detail, compact lists, summaries,
relations, search results, and display-status counts.
- Show the blue clock in shared task icons and chips, task detail and
breadcrumbs, properties, references, list rows, recent navigation, and
agent task cards.
- Add Monitoring to task filtering, grouping, sorting, board columns,
and search facets. Prevent a drag from creating this derived state.
- Keep stored lifecycle statuses, monitor scheduling, checkout,
recovery, permissions, and agent status filters unchanged.
- Add database and UI regression coverage, full-page Storybook examples,
and product and access-map documentation.
## Verification
- Passed `pnpm -r typecheck`.
- Passed `pnpm build`.
- Passed `pnpm check:token-gates` and the map/catalog structural checks.
- Ran the new database suite on a disposable PostgreSQL 17 database. It
covers read surfaces, filter pagination/counts, queued/running checks,
rescheduling, completion, retained monitors, review, and recovery holds.
- Ran existing search, list, board, sidebar, task status, and filter
tests. Corrected board queries use displayStatus and eight columns.
- The full local `pnpm test:run` stopped in a chat integration shard: 12
failures (four timeouts and eight FIFO publication assertions). That run
spanned a rebase. A focused rerun on the current source passed all 21
selected cases without assertion or timeout changes. The original cause
is not established; this does not claim the full local suite passed.
Final-head required CI passed (ci / verify and ci / e2e, run
41488267209).
- Passed 82 tests for task API summaries, recovery precedence, search,
and optimistic updates; 54 tests for status cards, database reads, and
board moves; and 27 tests for blocker relations and recent-task cache
updates. One intermediate database setup hit its 20-second hook timeout;
the serial rerun passed.
- Greptile is 5/5 on
|
||
|
DottaandPaperclip
|
b9e94b85a6 |
feat(observability): trace Daytona lease acquisition and resume (#18626)
## Thinking Path > - Paperclip manages work performed by AI agents. > - The server prepares a remote execution environment before the agent can respond. > - OpenTelemetry traces show the time spent in each preparation phase. > - A retained Daytona sandbox can take minutes to resume from an archive. > - The provider worker previously exposed no child timings for this operation. > - This pull request adds bounded lease phase spans under the existing worker trace. > - Operators can separate provider startup from lookup, teardown waits, and workspace verification. ## Linked Issues or Issue Description **What existing behavior does this improve?** Daytona environment acquisition and resume tracing. A long worker invocation currently hides the operation that consumed its time. Related tracing foundation: #10999. No duplicate open PR was found. **Subsystem affected** The Daytona sandbox provider and the server's existing plugin span validation boundary. **Proposed behavior** Show lookup, start or recovery, teardown waits, workspace verification, and fresh acquisition setup as timed child spans. Group verification steps beneath one parent. Record the bounded provider state before start, including archived. **Reason and benefit** An operator can identify which step delayed the first agent response. SDK startup includes its internal readiness wait; this change does not claim to measure provider queue time separately. **Breaking changes** The opt-in trace schema gains fixed span names and one closed state attribute. Execution, authorization, resource ownership, deadlines, and retry policies keep their existing contracts. ## What Changed - Add lease phase spans through the existing invocation tracer. - Add the fixed span names and state values to host validation. Define the state key in the shared startup attribute contract. - Preserve the exact work result or error if tracing fails. Never repeat work through the tracing wrapper. - Test worker receipts for archived and warm resume, recovery, concurrency, failure, disabled tracing, and cleanup. - Document the waterfall and its attribution limits. ## Verification - Daytona plugin suite: 525 passed, 7 skipped. The expanded tracing suite separately passes all 12 cases. - Host span validation and instrumentation tests: 46 passed, 3 skipped. - Standalone Daytona TypeScript check passes. - Module boundary and whitespace checks pass. - Full recursive typecheck and full build pass. - The full local Vitest attempt at |
||
|
DottaandPaperclip
|
97c4dc969e |
test(evals): retire duplicate chat TestOps plans (#18625)
## Thinking Path > - Paperclip manages AI agents and preserves their work history. > - Product E2E cases use TestOps plans to select real AWS evaluations. > - The full chat catalog now includes the former smoke and completion-pilot cases. > - The two old bootstrap workflows can recreate their smaller plans after deletion. > - This pull request removes those workflows and stops their plan-creation functions before API access. > - The full-catalog workflow keeps its shared clients, evidence rules, case identities, and measured results. ## Linked Issues or Issue Description Refs #18542, #18516. The user requested removal of the duplicate smoke and completion-pilot plans and investigation of their recreation paths. Both old workflows accepted manual dispatch only. Their bootstrap operations created plans by name when the plans were absent. They had no schedules in their workflow definitions. The old workflow records were disabled before plan removal. An existing pilot execution was allowed to finish successfully. This change does not cancel executions or delete cases, results, launches, or private evidence. ## What Changed - Remove `chat-feature-testops.yml` and `chat-completion-testops.yml`. - Make the former singleton plan creator reject every call before API reads or mutation markers. - Reject pilot bootstrap before API access. Keep the full-catalog branch and its integration validation. - Remove tests that require the retired workflow files. Keep shared helper and full-catalog controls. Run the trusted-checkout boundary test against the active catalog workflow. - Replace old setup instructions with retirement guidance and the active full-catalog route. The master sync preserves the incoming typed launch-search fix from #18620, including both regression tests. The only conflict was the retired pilot documentation. Its search-format note remains as a retained-helper contract. ## Verification - Two new retirement controls fail against the parent code because it reaches an API or mutation marker. Both pass after the change. - Six owning Node suites after the master sync: 146 passed (144 before the sync). Shared contract: 22 passed. Catalog and retained export controls: 12 passed. Focused workflow and release controls: 7 passed. - Two modified API modules pass Node syntax checks. `git diff --check` passes. - The pure workflow-security file has the same four failures and 13 passes on both the untouched parent and this branch. Two assertions expect eight trusted bootstrap steps where master now has nine. Two lists omit the already-shipped catalog workflow. This change does not repair or relax those unrelated assertions. - A broader release-test invocation had 73 passes and 20 sandbox failures from blocked mock HTTP listeners or npm permissions. Its focused affected workflow control passes. A standard Vitest setup attempt stopped on a missing `acpx/runtime` bundle dependency before tests. No local app, database, provider, or native runtime was qualified. - Current-head PR CI passes: 48 successful jobs and one skipped job, including both required checks. Greptile is 5/5. Independent source review passes. No new model run is needed for this retirement. ## Risks - Existing integrations that point at either retired workflow will stop dispatching. This is the requested behavior; use the full catalog instead. - Historical evidence readers still use shared helpers. Those helpers remain, including the transport and official-context parser used by the active catalog. - Workflow removal is a repository change. TestOps plan deletion and disabling obsolete executors are separate service actions. Neither action should delete shared cases or measurements. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with source editing, shell tools, and independent source review. The context-window size is not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3f7811eef0 |
fix(runner): keep requests responsive during retired backup cleanup (#18619)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote native runs keep durable harness backups for continuation and
recovery.
> - Publishing a new backup retires earlier backup directories in the
server process.
> - The synchronous permission walk and recursive removal block
unrelated requests.
> - This pull request makes that cleanup asynchronous and keeps
publication waiting for completion.
> - The benefit is continued request progress while large retired
backups are removed.
## Linked Issues or Issue Description
**What happened?**
Publishing a backup with many small files blocks the Node event loop
during directory permission repair and removal. A local comparison with
20,000 retired files reproduced 1.37–1.95 second event-loop stalls. This
is a pause candidate. A short hosted profile has not established it as
the dominant source of sustained CPU use.
**Expected behavior**
Unrelated requests can progress during cleanup. Publication still waits
for complete removal, preserves rollback on a failed lease stamp, and
leaves live files and symlink targets unchanged.
**Steps to reproduce**
Run the benchmark in `doc/performance/native-backup-publication.md`. It
compiles the actual publication service from a pinned baseline and the
working tree, prepares immutable retired backups, and sends foreground
HTTP probes from a separate process.
**Paperclip version or commit**
Measured baseline: `774ddc72713d6aa7cb516691c71f1fc23bcfa5ff`. The
measured treatment source is committed in this branch. Current head:
`0a69cf2d43c2f9fcf032d96c1dceb56163bf21ba`. The runtime patch is
unchanged from the measured source; later commits only strengthen the
test and benchmark evidence paths.
**Deployment mode**
Local compiled service on macOS arm64 with Node v24.22.2. No hosted
rollout has occurred.
Related work: #18380 changed retention scheduling. #13676 included the
existing immutable-backup cleanup behavior. Neither change makes
publication cleanup asynchronous. The GitHub duplicate search also found
#11929, which concerns database dumps rather than native harness
backups. No matching open event-loop cleanup issue was found.
## What Changed
- Use asynchronous filesystem operations for retired-backup cleanup.
Visit directories serially and skip unnecessary metadata calls for file
entries.
- Await cleanup before publication proceeds, during rollback, after
successful rotation, and in executor finalization.
- Exercise immutable current and previous backups in the publication
test. Preserve the existing symlink-isolation and rollback tests. Add a
real separate-process HTTP probe that must complete while publication is
still active.
- Add a repeatable service benchmark and document its fixture, metrics,
and acceptance limits.
## Verification
Draft status: focused correctness tests, the local comparison, and
repository typecheck pass. The complete local test gate is incomplete.
The build passed. Complete CI, hosted acceptance, and final-head review
remain pending.
- Final focused Vitest run: 2 files, 4 tests passed in 649 ms with the
normal server configuration. The regression warms a separate Node
client, waits for its HTTP request during the stamp callback, queues the
response for the next event-loop turn, and then allows final
retired-tree removal to start. The async treatment serves that response
before publication completes. The original synchronous cleanup fails the
same final regression in a separate source fixture. That expected
control failure is retained.
- Syntax and `git diff --check` passed.
- `pnpm -r typecheck` passed on head `332c57b88c` in 152.18 seconds,
including the native Runner build. Validation uses Node v24.22.2 and the
repository-pinned pnpm v9.15.4 with isolated dependencies.
- `pnpm test:run` was intentionally stopped after 1,390.24 seconds (exit
143). All six chat shards passed, covering all 1,817 parameterized cases
across their exact partitions. The general server phase was advancing
but incomplete. The repository timing manifest estimates another roughly
two hours for the serial remainder, so validation moved to final-head
parallel CI. This is not a full local test pass. The command began on
`332c57b88c`; later changes only refine the regression and benchmark,
and the final regression has its own focused pass.
- Benchmark failure/metadata checks pass in a tiny isolated Git fixture:
an injected worker exit 17 is reported in 0.44 seconds without hanging,
and the result includes a staged source change when the unstaged diff is
empty.
- `pnpm build` passed in 41.80 seconds on `9f7e4e669b`. The final commit
only releases a pending test probe on failure; runtime and build sources
are unchanged. An injected client exit verifies that final test failure
path exits promptly in 0.97 seconds instead of hanging.
- Three alternating baseline/treatment pairs used fresh fixtures with
100 directories × 100 files in each of two retired backups. Each
publication removed 20,000 files. A separate process sent HTTP probes at
fixed 5 ms intervals, including overdue arrivals.
- Every run verified the replacement manifest, complete cleanup, and
unchanged symlink-target contents and mode. All 2,395 HTTP requests
completed without errors. The largest measured generator delay was 8.8
ms.
| Metric | Baseline median (range), ms | Treatment median (range), ms |
| --- | ---: | ---: |
| Foreground request p95 | 1,357.7 (1,287.4–2,014.0) | 1.8 (1.7–25.9) |
| Maximum event-loop delay | 1,439.7 (1,371.5–1,954.5) | 19.9
(17.4–40.7) |
| Publication completion | 1,429.1 (1,360.2–1,952.9) | 1,439.5
(897.9–1,649.6) |
| Process CPU during publication | 1,194.1 (1,041.8–1,225.5) | 2,423.1
(1,995.7–2,514.7) |
The treatment served 180–329 requests while publication was active. The
baseline served zero during that interval. Process CPU includes
foreground work that the baseline cannot serve until cleanup completes.
The result supports responsiveness, with a CPU tradeoff. It does not
establish lower total CPU or a hosted latency gain.
Reproduce with Node v24.22.2:
```sh
pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-harness-backup-cleanup.test.ts src/services/native-runtime/native-harness-backup-publication.test.ts
node server/scripts/benchmark-native-backup-publication.mjs --baseline-ref
|
||
|
DottaandPaperclip
|
0748b90062 |
fix(server): stop goal-less native runs before board task closure (#18616)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task owns its active run and its tool access. > - A board user can close a task while that run is still working. > - The task update route stopped native runs only when they had an active session goal. > - Providers without goals kept running after the route cleared task ownership. > - This pull request stops those native runs before a board closure releases ownership. > - The benefit is an orderly stop instead of tool denials and failed finalization. ## Linked Issues or Issue Description **What happened?** A board user marked a task Done while its native OpenCode run was active. The route cleared its execution lock without stopping the provider. Later tool calls failed with `paperclip_runner_tool_binding_not_authorized`. The final wake fence then failed with `native_current_wake_comments_binding_changed`, leaving workspace finalization in retry. Cancelled also released ownership before its later Stop call. **Expected behavior** A board closure must confirm native Stop before it releases task ownership. A failed Stop must leave the task open. The provider does not need to support session goals. **Steps to reproduce** 1. Start a native task with a provider that has no active session goal. 2. Mark the task Done while its run is active. 3. Observe that the old route updates the task without cancelling the run. ## What Changed - Reuse the existing audited cancellation path before a board user marks an active native task Done or Cancelled. - Keep the existing 409 response when Stop is unconfirmed. - Record the board user in the Stop metadata. The existing acknowledged-Stop policy then leaves queued input and recovery quiet until a later explicit wake. - Inspect additional runs only when a task has an execution owner; tasks without a run lock keep their existing completion path. - Keep native semantic completion, legacy completion, and goal control behavior. - Add route regressions for stop ordering, failed Stop, and legacy completion. - Update the execution contract and task maps. ## Verification - Red before the fix: three regressions failed. Done had no Stop; Cancelled stopped after the update; an unconfirmed Stop allowed Done. - A second red proved that a Stop receipt without the route's operator metadata was not recognized by the real recovery policy. Both cases pass after the metadata repair. - Final local result on macOS arm64 / Node.js 26.4.0: **348 tests pass across eight suites**. These cover the route, native self-completion authority, cancellation, acknowledged Stop, recovery queues, dependency wakeups, and completion telemetry. - CI found seven sibling route failures from an unnecessary active-run lookup on tasks without execution ownership. Those cases failed locally before the correction and pass afterward. - Earlier concurrent local runs hit a module-load hook timeout and one watchdog socket hang-up. The isolated watchdog case and the complete final eight-suite run pass with a 90-second startup budget. No assertion was removed. - `node scripts/check-maps.mjs` and `git diff --check`: pass. - `pnpm build` and `pnpm -r typecheck`: pass. Server build and typecheck also pass after the Stop metadata change; the final server build includes the lookup correction. - `pnpm test:run`: attempted; the unchanged chat shard selector failed before executing its selected tests (`No test found ... in lines ...`). This is separate from the PostgreSQL startup failure. CI supplies full-suite verification. - PostgreSQL integration and live-provider acceptance did not run locally. Embedded PostgreSQL failed before app startup with `could not create semaphores: No space left on device`. No usable isolated local database runtime was available. CI supplies the PostgreSQL regression environment. The full live OpenRouter workflow remains unverified. - The first final-head CI pass hit a 15-second timeout in the unchanged `attention-retention-work.test.ts` blocker-history fixture. Cleanup then deadlocked and later inserts reused IDs. This service test does not call the changed route. The original log is [retained here](https://github.com/paperclipai/paperclip/actions/runs/41476191754/job/125384289220). The one allowed rerun also failed on the same commit: the `running` case exceeded 15 seconds again ([rerun log](https://github.com/paperclipai/paperclip/actions/runs/41476191754/job/125393915025)). The rerun otherwise passed 2,346 tests. No retention source or assertion was changed. This repeatable failure is the remaining CI gate; the PR is **not ready to merge**. The next step is to isolate that retention fixture/query in a usable PostgreSQL environment and repair its cause before rerunning CI. - Greptile reviewed final commit `b3cb41af2d911e9ced219e846cada3ecec835dbf` at **5/5**, with no unresolved review threads. Greptile is green. CI is blocked by the repeated retention test timeout described above. - CI native cancellation evidence: all 16 tests in `server/src/__tests__/native-cancellation-request.integration.test.ts` pass ([job log](https://github.com/paperclipai/paperclip/actions/runs/41476191754/job/125384289222)). - CI PostgreSQL wake-queue evidence: all 96 tests in `server/src/modules/wake-queue/adapters/postgres.test.ts` pass ([job log](https://github.com/paperclipai/paperclip/actions/runs/41476191754/job/125384289470)). ## Risks Board closure now waits for native Stop even when the provider has no goal. A failed or uncertain Stop keeps the task open. The change uses the existing cancellation and workspace-preservation path. It adds no retry loop, credential access, schema change, or publication authority. Direct service updates and agent semantic completion retain their existing paths. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, shell execution, and browser inspection. The exact serving snapshot ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
412834a860 |
test: overlap long recovery and maintenance waits (#18544)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - CI verifies recovery, execution authority, and background work before changes merge. > - Diff selection saves work, but a selected file still runs its cases in sequence. > - The three longest files in a recent full master run each took over eight minutes. > - Many of these cases must observe real lease expiry or idle periods. > - This pull request gives independent cases separate fixtures so those periods can overlap. > - The benefit is a shorter test path with the same assertions and observation windows. ## Linked Issues or Issue Description Refs #18464. The latest measured successful master [Cloud readiness run](https://github.com/paperclipai/paperclip/actions/runs/41300933157) spent 581 seconds in issue recovery, 569 seconds in native restart recovery, and 512 seconds in maintenance server routes. File-level selection cannot divide those long serial files. Shortening their lease or idle windows would weaken the regressions they protect. ## What Changed - Split issue recovery into four files with separate database and worker fixtures. Keep all 18 cases. - Split native restart coverage into recovery, lease renewal, preparation, checkpoint cost, and disk-loss files. Keep all original 79 cases, including the existing container-qualified cases. The master update also retains its newly added recovery cases. - Close each native case's database clients after pending failure reports drain. The shortened file must not rely on idle timeouts to stay below PostgreSQL's connection cap. - Run the native Runner lane with two isolated fork workers. Register every new file in the lane and verify that the partition still includes every suite. - Give each maintenance case an owned database. Run at most two cases together with test-local assertion contexts. Keep both real 90-second quiet windows and all 17 cases. - Preserve serial execution when any explicit shared `PAPERCLIP_*_TEST_DATABASE_URL` is present. An admin URL that creates a separate database per fixture can run concurrently. - Refresh shard weights from the protected master run. Mark the initial split-file weights as conservative estimates until master measures them. - Anchor the attention-retention fake clock to PostgreSQL time. Its fixed clock started trailing real database timestamps after 12:00 UTC on November 1, causing an archive-deadline test to time out. Keep its deadline assertions and timeout unchanged. - Document the fixture and concurrency rules. Leave runner authorization and workflows unchanged. ## Verification - Final head `c9a4ff11b3ba81277341294c7721b34fb8705ede`: [full PR CI passed](https://github.com/paperclipai/paperclip/actions/runs/41477848686), including typecheck, build, all test shards, E2E, and the final verification gate. The rollup has 55 successful checks, one neutral review check, four expected skips, and no pending or failed checks. Greptile reviewed this exact head at 5/5 with no actionable findings. - November 2 master update: merge commit `c9a4ff11b3ba81277341294c7721b34fb8705ede` resolves the native recovery import conflict while retaining the new busy-row and remote continuity cases from master. All 38 test-runner checks and the three new busy-row cases passed locally. - Retention verification after the master update: the first local full run passed 52 cases and timed out at 15 seconds in the paged-backlog case. That case passed in isolation. The unchanged master suite passed all 53 cases (82.95 seconds), and a full PR rerun also passed all 53 (76.65 seconds). The initial timeout cause remains unproven; no timeout or assertion was weakened. - Previously verified head `fd288a8e8c88df54dcdca7b7e42366b019182782`: [full PR CI passed](https://github.com/paperclipai/paperclip/actions/runs/41312625439). There are 55 successful checks, one neutral review check, four expected skips, and no pending or failed checks. Greptile reviewed this head at 5/5 with no actionable findings. - Issue recovery: 18/18 passed before and after the split on the same local machine and PostgreSQL fixture path. Wall time changed from **630.6 seconds to 319.7 seconds** (49% less). All case titles match. - Maintenance also passed all 17 cases on AWS: structured file cost dropped from **511.8 to 282.6 seconds** across the linked master run and the first PR run. These are different CI runs, so the matched local comparison below is the stronger isolated measurement. - Maintenance: 17/17 passed before and after with the restored pinned dependencies. Wall time changed from **488.1 seconds to 317.2 seconds** (35% less). All 120 assertion lines remain. Both real 90-second windows also passed a final targeted run. - Native restart: 71 passed and eight existing container-only skips before and after the split. Wall time changed from **632.1 seconds to 346.8 seconds** (45% less). All 79 cases and 254 assertion lines remain. - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - Extra TypeScript checking of the moved recovery and maintenance tests passed. The standard server config excludes `src/__tests__`. A separate check of attention-retention fixtures reports existing type errors; the unchanged baseline has identical diagnostics, with none introduced by the clock fix. - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs scripts/ci/test-selection*.test.mjs`: 61 passed. - `pnpm test:run`: stopped in the second chat partition: 335 passed and four failed (two GitHub retry-state assertions and two Teams timeouts), after 339 passed in the first partition. These chat tests are unchanged. All four pass in a focused run (19.1 seconds). The full-suite ordering/load cause remains unproven. All three CI chat jobs passed. The previously failing cases passed there in 1.3, 1.3, 2.4, and 2.2 seconds; see [the chat job log](https://github.com/paperclipai/paperclip/actions/runs/41309691650/job/124857580714). The attention-retention timeout repeated in CI and in an isolated local run. Its fake clock was fixed at November 1 noon, which had fallen behind real PostgreSQL timestamps. Anchoring it to the database clock passes all 53 cases; the failed case now takes 425 ms. - The native CI lane passed all 162 runnable cases with eight existing skips after client cleanup: **650.7 seconds on the earlier master run to 391.0 seconds on this PR**. - CI found native connection exhaustion after the long waits moved out of the main file. The same failure reproduced locally at a 100-connection cap. Per-case client cleanup fixes the shortened file at that cap, and all five native files now pass together at that cap (71 passed, eight existing skips). - Early attempts exposed a missed fixture reference and a missed reporting setup hook during extraction; both were restored. The first CI policy run also required moving the new native fixture under test helpers; that boundary check now passes locally. A separate auxiliary command picked up the host pnpm version and removed dependencies. The original maintenance baseline reproduced the same missing-package failures, so that failed attempt is excluded from timing comparisons. A later unchanged-baseline run passed all 17 cases. The pinned install was restored before the passing maintenance run. The audit also covered heartbeat process recovery, deferred receipts, chat, workspace recovery, attention, and connection management. Their shared fake clocks/provider mocks or existing partitioning need different changes. The 16,400-row attention case and the large checkpoint stream remain intact because their size protects real limits. ## Risks - Two isolated processes use more memory and database connections than one. Concurrency is capped at two. Shared database overrides force serial execution. - Moving setup into helpers can change test lifecycle order. The split suites keep the same case assertions and use independent process state; focused runs and the full CI suite check the move. - Split-file weights are estimates based on case timings from the linked run plus repeated setup cost. The next protected-master refresh replaces them. Weights never permit tests to be skipped. - Local timing varies with host load. CI measurements after merge are needed to establish the overall PR wall-time improvement. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, tool use, and code execution. The exact backend revision and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (changed suites and focused reruns; the full local-run caveat is recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
baa4a2e64e |
fix(runner): restore Codex auth before source recovery (#18612)
Restore disposable Codex authentication and safe provider configuration before cold source recovery. Preserve source receipts and conversation state until reconciliation, and rebuild assigned tools from the admitted launch binding. Validate remote launch destinations, clear stale authentication when no selected credential exists, and atomically publish SSH file transfers without following retained links. Red/green recovery regressions, a real two-turn Codex/MCP probe, and all current PR checks pass. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
aab87ce707 |
Prove workspace delivery before inspecting cleanup candidates (#18493)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Completed tasks leave workspaces that need safe cleanup. > - Providers without ancestry inspection currently need a recorded merged PR before cleanup can start. > - A squash merge changes commit identity, so direct ancestry cannot prove that delivery. > - This change proves delivery from the finalized HEAD and captured destination before storage inspection. > - Fresh storage checks and a second delivery check still protect removal. > - This reuses the existing worker and receipts. It adds no table or timer. ## Linked Issues or Issue Description Refs #18410. Related: #13400 addresses workspace identity during inspection; this change adds destination-aware delivery discovery. ## What Changed - Resolve direct ancestry or discover an associated merged GitHub PR without a recorded work product. Require exact HEAD, destination repository and branch, and a reachable integration commit. - Use the latest settled finalization receipt as a candidate. Reject missing, failed, pending, conflicting, or changed evidence. - Inspect storage only after positive remote proof for providers that disable ancestry inspection. Recheck HEAD, branch, source identity and delivery before removal intent. - Add an optional provider capability for a specific PR recovery hint and an optional prepared destination receipt. Older providers keep their existing protocol. - Wake cleanup when finalization writes settle or gain a workspace binding. Keep remote-only retries for concrete unresolved delivery. Park missing or unsupported candidate identity. - Reuse the source project's explicit GitHub consent for delivery checks through the existing authorization service. Revalidate the bound repository and consent before removal; missing or revoked consent retains storage without switching identities. Public sources remain anonymous. - Update the contracts, documentation, access map, and targeted regressions. ## Verification - Initial-head GitHub CI passed at `5acd4c818be6f1764f8319f42891f9956c357c26`, including full build and typecheck. Standard Greptile is 5/5 with no open threads. The first Apex review scored 3/5 and found startup latency from remote proof lookups and removal retry identity changes across provider capability changes. Both corrections are pushed in `9d1a38df3d`, with the qualified destination-ref length fix. Red regressions reproduced each issue. Latest targeted checks passed: cleanup/worker 26, provider integration 41, SDK contract 8. Head `9d1a38df3d` passed standard and Apex reviews at 5/5 with no open threads, plus all CI including full build/typecheck. Canary Dry Run initially failed on an npm 404 fetching `rollup-4.67.3.tgz`; the exact tarball subsequently returned HTTP 200, and one targeted retry passed without source changes. Aggregate verification also passed. After #18403 merged, master introduced conflicts in shared workspace cleanup and metadata handling. Resolved them in `7700832536`, preserving remote snapshot cleanup and Git delivery-proof cleanup. Targeted merge verification: 93 tests passed in the first run (59 opt-in skips), then all 16 remote lifecycle tests passed with explicit database opt-in; map checks passed. The isolated database is stopped. The merge head received standard review 5/5. Its CI found two failures in unchanged test cleanup paths: native runtime temporary-directory ENOTEMPTY and agent-chat cancellation lock timeout. Those failures remain recorded and require successful final-head verification. The subsequent project-consent change has targeted coverage: authorization36, resolver33 and cleanup19 tests passed, including revoked consent, anonymous public access and source-binding races. The positive consent test failed before the fix. Current head `c772679e05` has green GitHub CI, including full build/typecheck and both previously failing test jobs, plus standard Greptile 5/5 with no open threads. Neither cleanup failure required changes to its test path. Apex also completed at 5/5 on this exact head (check `124651548532`, summary updated 2026-10-31 20:45 UTC), with no open review threads. The PR remains draft. Earlier failures remain recorded above. - Final focused run on the rebased branch: **182 tests passed across six files**, covering the resolver, delivery cleanup, provider bridge, finalization notifications, existing workspace service and SDK contracts. The disposable test database was stopped afterward. - Targeted resolver tests: 33 passed. These cover ancestry, discovered squash PRs, wrong HEAD/base/repository, removed integration commits, bounded responses, cancellation and credential isolation. - Targeted cleanup integration tests: 17 passed. Provider bridge integration: 36 passed. SDK contract tests: 7 passed. Finalization notification tests: 9 passed. - A negative control that removed the final delivery check failed the race regression as expected. The check was restored. - Read-only live GitHub probe: the new resolver discovered PR #18410 from HEAD `3f5ade4cd42b707040e9a82fdaed343e3d736a09`, without PR hints. It proved integration commit `6484f96c28bfb11a793ce4be6cc5ac15643a5a02` reachable from observed master `5bc029bf6c1db6647e96acea323f3723fd1d3eba`. GitHub also advertised the exact original HEAD at `refs/pull/18410/head`. - `node scripts/check-maps.mjs` passed. Structural checks do not prove deployment behavior. - Local tests are targeted by user request. Full build, typecheck and test suite run in GitHub CI. A standalone server typecheck could not complete because local Runner build declarations were absent; no changed-file diagnostics were reported. - Initial local execution hit a stalled pnpm wrapper, sandbox socket denial and stale generated SDK declarations. Direct installed binaries, the authorized isolated test database and rebuilt SDK resolved those setup failures. No deployment was performed. ## Risks - Destructive cleanup still requires fresh storage safety and lifecycle checks. Remote proof alone never authorizes removal. A persisted removal intent preserves existing recovery semantics. - Legacy workspaces without an unambiguous destination or completion HEAD remain retained. This does not guess the original default branch. - Private repositories use the explicit standing GitHub consent on the bound source project, with current membership, grant, repository access and trust checks. No separate company token is required when that consent is valid. Missing or revoked consent and unavailable or rate-limited access retain storage; the worker does not borrow a task or agent identity. - A known undelivered commit keeps the existing five-minute retry because external GitHub changes have no guaranteed ingress to this lane. This can keep cleanup scheduled. Missing evidence parks instead. - Providers need the new recovery capability to clean a squash merge whose original HEAD is reachable only through its PR ref. Providers without it remain conservative. - The resolver currently supports GitHub.com. Existing local and other-provider ancestry paths stay in place. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools and parallel code agents. The exact runtime model ID and context-window size were not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
71a20453e1 |
fix(workspaces): honor inherited remote workspace bindings (#18611)
## Thinking Path > - Paperclip manages AI agents and their work. > - Task creation can bind a child to its parent's execution workspace. > - Native sandbox runs reserve that workspace before the provider starts. > - Remote reservation rejected inherited isolated workspaces and operator branches because the original source task differed. > - This check stopped valid child tasks before their existing access and writer checks could run. > - This pull request uses the persisted binding and existing source-access checks to admit that continuation. > - The benefit is that authorized child tasks can continue saved work while the single-writer lock stays effective. ## Linked Issues or Issue Description **What happened?** A child inherited an active isolated remote workspace through normal task creation. Its next run failed during setup with `Workspace ownership changed.` No provider work started. The reservation guard treated the original `sourceIssueId` as exclusive ownership, although task creation had persisted a valid child binding. **Expected behavior** A bound child or follow-up can reserve the workspace after the prior writer saves. Current task/run access, source visibility, trust, company scope, closed-workspace state, and writer checks must still apply. The original source task remains provenance. **Steps to reproduce** 1. Create a task with an active saved remote isolated workspace or operator branch. 2. Create a child without an explicit workspace override. 3. Confirm that the child inherited the execution workspace ID. 4. Start its native sandbox run. The old reservation guard rejects it before provider execution. **Paperclip version or commit** Reproduced on source based on `06bc8db75c71142ed108e0f331394c5187015d6a`. The same guard is present at `1a6017d6460267ad89d7e1784e2b521c28d8e9ac`. Related: #17156 proposes separate workspaces for child tasks. This repair preserves the current inheritance contract and enforces the remote writer lock. #16911 addresses child bindings across repositories. ## What Changed - Remove the original-task equality restriction for active remote workspaces. Keep the existing persisted-binding and retained-source authorization path. Explicit task-root selection follows the existing V1 filesystem contract; task-assignment grants do not define filesystem permissions. - Keep closed workspace rejection and unfinished-writer checks. - Add database-backed regressions for inheritance, replay, competing children, an unbound task, private source data, company isolation, and closed records. - Extend real save/restore tests for inherited isolated workspaces and operator branches. Keep shared-workspace discovery coverage. - Update the development guide and workspace maps. ## Verification - Red: before the repair, the inherited isolated/operator cases fail with `Workspace ownership changed.` The shared case passes. The unfixed suite has five failures and four passes. - Green: 11 reservation cases pass with real PostgreSQL 18 in an isolated Docker container. - An earlier run of the four affected workspace/preparation/trust suites passed 191 tests. The review follow-up passes all 15 selected reservation/save/restore cases. Inherited cases remove the original files, use a distinct allocation and target, restore the saved file, append child work, and independently read that work from the child's new snapshot. Shared-source discovery stays covered. The filter excludes 42 other integration cases. - `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs`, and `git diff --check` pass. A direct server typecheck also passes after the final test edit. - `pnpm test:run` stopped in its first chat shard: 338 passed, one Discord receiver-lease setup failure, 1,478 outside the shard. The failing test passes in isolation on both the unchanged baseline and the fix. Its suite-order failure is not explained by that retry and remains recorded. The unchanged baseline also fails three other Discord publication cases in that chat shard, while its next chat shard passes. This does not reproduce or explain the original lease failure; both results remain recorded. CI is complete on `e697f6e1b8`: 54 successful check runs, one successful legacy status, three skipped checks, and one neutral check. No check is pending or failed. Greptile gives this commit 5/5 with no remaining findings. The security reviewer accepted the V1-contract explanation; both review threads are resolved. - Live sandbox acceptance remains unverified. The available test credentials could not resolve the required sandbox snapshot. The provider rejected creation before workspace execution, and a scoped follow-up found no allocation. No live model turn, deployment, or original-task recovery is claimed. - Local environment: macOS arm64, Node 26.4.0, pnpm 9.15.4. Native PostgreSQL startup hit the host semaphore limit; tests use disposable Docker databases. No shared database was reset. ## Risks - A bound child can now write after the prior writer releases the workspace. Incorrect binding or access checks could expose source data. Regressions cover unbound, private-source, and foreign-company rejection. - Concurrent children must not become two writers. A real transaction race grants exactly one reservation; saved-file tests keep the second writer blocked until save completes. - No schema or migration changes. This PR does not change task-creation inheritance, requested models, credentials, trust, approval gates, or recovery budgets. ## Model Used - OpenAI Codex, based on GPT-6. The exact serving model ID and context window are not exposed in this session. Used reasoning, tool use, code execution, and browser diagnostics. No sub-agents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — the targeted workspace suites pass; the full local chat-suite failure and baseline comparison are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
774ddc7271 |
test(evals): add the full chat catalog to TestOps (#18542)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat feature evaluations check real user requests through the product. > - The YAML catalog has 244 scenarios, but the shipped TestOps completion route selects only three. > - The preserved catalog already supplies the execution mappings and independent oracles. > - A larger TestOps plan needs exact identities, bounded selection and source-bound private evidence. > - This pull request adds that plan route without merging the catalog app into master or replacing its grades. > - Users can run the full catalog or a selected subset and see the actual outcome and prerequisites. ## Linked Issues or Issue Description Refs: #18516 and #18515. This extends their completion pilot and reuses its trusted reporting path. **What existing behavior does this improve?** The existing three-case TestOps completion workflow. **Current behavior** The remaining catalog scenarios cannot be selected through that TestOps job. The one-case backlog plan has a separate strict singleton contract. **Proposed behavior** A separate candidate-scoped plan maps all 244 scenarios. Existing case aliases, jobs, plans and result history stay intact. Registration is distinct from a measured pass. ## What Changed - Add a source-derived 244-row manifest. It binds the 242 domain descriptors and two legacy tasks to the unchanged YAML queries, contexts, outcomes and original bounds. The JSON snapshot avoids importing the unmerged domain tree into master. - Reconcile exact case identities and a dedicated full-catalog plan. Reuse legacy aliases, including the existing backlog case, without renaming them. Preserve history and stop on conflicting aliases or uncertain writes. - Extend the shared completion adapter with a separate trusted full-catalog workflow context. Return a compact observed binding that fits one 48 KB protected variable. Keep the three-case pilot contract intact. - Add a fixed-source manual workflow. Small selections run serially. Larger selections allow eight independent workers and one serial external-input worker. Each worker and report reader keeps a singleton approval and exact-key AWS session. The eight-cell private authority limit is unchanged. - Validate and merge metadata-only singleton projections before one protected Allure upload and complete paginated readback. Preserve failed originals and cleanup failures beside passed results. Keep screenshots and raw evidence private. - Discard a changing full-catalog page-count snapshot and retry within the same observer bounds. Malformed page fields still fail at once. A pass still needs one consistent exact selection and all final evidence checks. - Keep workflow source and app source separate. The full lane builds candidate `3b66c3c144360ddf32714a0742fd774b8ceafb16`. Its only changes from the existing `de0941835cc8f0d01c982574ee258e110babdbeb` pin are finite TestOps host admission and its offline test. This is not current-master app qualification. - Give full selections above three cases a bounded metadata-read allowance above the existing 120-second deadline ceiling. Readback has two upper bounds: at most twelve polls with five-second pauses, and a size-derived deadline of at most 374 seconds, whichever exhausts first. There is no guaranteed minimum wait or processing SLA. Small selections and pilot deadlines stay unchanged. Final identity, grade and evidence predicates stay intact. - Document the real readiness limits and setup path. Five coverage gaps, six evidence exceptions, eighteen additional-credential declarations and thirty-seven external-input declarations overlap. They are not new pass claims. ## Verification - Related Node controls passed: pilot API and dispatch, 244-row manifest, idempotent reconciliation, full selection, three-page result readback, source guards and partition/output-size bounds. The latest ten adapter controls include the actual registration receipt's variable-size limit and the real bootstrap facade's closed failure codes and privacy checks. - A deterministic 244-result replay with one second per HTTP response failed at the old deadline and now verifies after 127 virtual seconds. A three-second response replay still rejects at 375 virtual seconds. The full-lane allowance accounts for paginated reads and four-way result batches; it is not a guaranteed network SLA. All 55 related API/adapter controls pass. They also reproduce the changing-page-count failure before the fix, then verify the consistent snapshot. Permanently changing counts fail after twelve polls. A malformed numeric field still fails before a retry. - Owning export controls: 26 passed. They include all 244 synthetic carriers, mixed passed/failed originals, missing and foreign projections, privacy filtering and exact source binding. - Candidate host owner: 31 offline controls passed. The new finite-catalog case failed before admission changed. - Preserve two actual local regressions: canonical scenario provenance failed before its one-line fix; the full registration binding exceeded the variable limit before compact output. Both now pass. - Initial local harness setup attempts failed before test execution. The final narrow owner configuration avoids provider-bundle setup. - Full repository tests, build and typecheck were not run locally. Normal required PR CI provides the remote checks. - Owning TypeScript no-emit, actionlint, paid-workflow policy with unsafe controls, and diff checks passed. - Local tests used Node 26.4.0. These are synthetic source controls. No local app, database, native build or provider qualification ran. - CI [41307571776](https://github.com/paperclipai/paperclip/actions/runs/41307571776) on `23e518be` recorded one 15-second timeout in `attention-retention-work.test.ts:153`. All other ordinary leaves passed. The test used a fixed JavaScript date but its retention deadline comes from PostgreSQL `updatedAt`. The exact failing await and original SQL timestamp were not captured. The corrected pre-sync head’s server shard (2/12) passed on AWS in CI 41309030928. Current master independently fixed the same clock assumption in #18543; conflict resolution adopts that whole upstream file. No local PostgreSQL run or timeout increase was used. - Current-head normal CI [41463567187](https://github.com/paperclipai/paperclip/actions/runs/41463567187) passes all required checks on `fc6c2309cfc7d1e5a83f0a597c7fa1d2beb987e1`, with Greptile 5/5 and all review threads resolved. The final diff has 23 catalog files. Current master’s whole retention test is adopted; the redundant earlier test change is absent. The valid changing-page snapshot regression passes with all 55 related API/adapter controls. No new TestOps import or live evaluation has run for this change. After delivery, use metadata-only bootstrap and a three-case normal TestOps selection to verify the new route. Do not claim a new 244-case measurement. ## Risks - The app target remains the preserved catalog candidate. Each result records its target and workflow revisions. No old campaign ordinal, acceptance gate or core fix is changed. - Some mapped cases need private account configuration or have known source/evidence gaps. Their original checks and evidence rules can still fail. A registration does not establish readiness or success. - Whole-plan execution can take hours and spend provider and AWS money. Original per-case limits and zero retries remain. Unknown setup usage is not free. This PR does not run all 244 cases. - The raw private `SOURCE_REF` field retains the prior integration-lineage label. It is not a verified checked-out branch. The actual candidate checkout and all authority checks use immutable `3b66c3c144360ddf32714a0742fd774b8ceafb16`; public projections set `source.ref` to null. - TestOps has no atomic bulk upsert. Reconciliation retains partial progress and performs no automatic resend after an uncertain response. - The trusted inventory keeps a 12 MiB limit per public carrier file. The larger parser ceiling does not promise admission above that limit. The existing 244-case fixture produces a 1,359,465-byte normalized carrier; this is not a guarantee for future payload sizes. - Raw evidence remains under the existing private retention owner. Public GitHub artifacts and TestOps attachments contain only allowlisted metadata and availability receipts. - The new workflow needs the exact master entry in the existing runner group's selected-workflow list before a live run. No repository or runner-group visibility is broadened by this source change. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) assisted with implementation and independent source review, using code and tools. The host does not report the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run the relevant local controls and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
06bc8db75c |
fix(runner): install verified API artifacts in execution workspaces (#18494)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents use authorized API tools to read saved results. > - Copying a large response as model-generated text can lose bytes. > - The execution workspace needs an exact copy of the saved artifact. > - This change copies the authorized bytes without shell API credentials. > - The runtime verifies size and SHA-256 before it reports success. > - A live evaluation checks capture origin, installation and delivered bytes. ## Linked Issues or Issue Description **What happened?** A native agent lost one byte while copying a 24576-byte API text page. Existing capture and paging controls passed. The runtime had no verified saved-artifact-to-workspace copy tool. A separate historical screening refusal does not establish capture corruption. **Expected behavior** An authorized native run can install the complete saved artifact. The stored and installed bytes must match in size and SHA-256. A conflicting file must remain unchanged. A lost publication receipt must not cause an unsafe repeated write. **Steps to reproduce** Run `server/src/services/native-runtime/artifact-transfer.integration.test.ts` on baseline `c46b8f35add9255fc71be294d27f5a7865bcb47f`. Its capability assertion fails because `download_artifact` is absent. Apply this PR and run the same regression with the filesystem and API controls. It verifies 388131 synthetic bytes through the authority and installation boundary. Refs #18429. Related #17393, #17086 and #17285 cover paging, transfer foundations and remote output reads. This PR adds the inbound installation contract. ## What Changed - Add `download_artifact` for supported standard-mode native runs, with a 10 MiB limit. - Reuse current company, task, run, asset-content and responsible-user access checks. Check authority again before publication. - Bind installation to the current execution lease. Stage through confined Linux descriptors and publish with an atomic create-only link. Keep API credentials out of execution commands. - Verify bytes at storage and execution boundaries. Reject unsafe paths, symlinks, different existing files, truncation and corruption. Reconcile an unknown publication through verification only; an identical explicit retry remains safe. - Add API documentation, access-map entries and content-free audit records. - Preserve only the pinned Daytona SDK numeric HTTP status when failed creation cleanup is unconfirmed. Keep provider exception messages and headers private. - Keep only the finished runner binary and portable provider pack in their builder layers. Remove Rust targets/downloads and the Node install workspace/store before export. Report tree sizes. - Add one explicit Codex/Daytona Product E2E cell. Its independent oracle checks the server capture audit, redacted tool receipts, stored source, delivered attachment and browser link after reload. Uploaded substitutes and one-byte loss fail. ## Verification - Merge update at `6766b46553cab5b9787ade87a002eccaf50e68fc`: merge current master and preserve both explicit suites in the catalog. Focused catalog, image and oracle checks: 122 pass. Filesystem and authority integration: 20 pass. Maps, module boundaries and whitespace checks pass. New-head CI and Greptile are running. Full local build and typecheck remain unavailable because cargo is absent. - Behavioral red/green: the absent-tool regression failed before implementation and passed afterward. The initial missing-package attempt is recorded as setup failure. - Linux filesystem and real PostgreSQL/authority integration: 19 filesystem tests and 1 authority test pass. Existing API authority, capture and paging controls: 70 pass, 2 skipped. Provider transport in these integration controls is simulated. - Qualification calibration: 118 focused tests pass. The real persisted-event redaction case and a byte-identical uploaded substitute each failed before their oracle repair. The unchanged controls then passed. Harness typecheck passes. - Full recursive typecheck and build pass locally on the integrated production source at `086f7a2901`. Later commits change only qualification tests and documentation. - Structural map/module checks and whitespace checks pass. - Full local `pnpm test:run` was attempted by the implementation worker and stopped under its 8 GiB memory limit, with no full-suite verdict. Current-head CI must supply the full test result. - The complete local Product E2E support suite has 13 failures. The unchanged pre-qualification baseline reproduces all 13: host PostgreSQL semaphore capacity, stale timeout/profile/workflow inventory expectations, and expired fixed-date allowance fixtures. This is not a full local suite pass. - Earlier CI at `086f7a2901` failed a browser test during cleanup cancellation with a database lock timeout. It remains recorded. All browser shards passed on both later fully checked heads. - Two qualification campaigns were canceled during setup before provider execution while correcting the oracle. They provide no live product verdict. - Previous-head CI at `28fa001457f59df1c300c016dc569049e72d0061`: 53 checks pass, 3 intentional selection skips; no failed or pending checks. - Real image/export regression: campaign 41272351780 compiled the Linux runner and passed root/unprivileged CLI and transport checks, then failed BuildKit export with `no space left on device` (exit 102, filesystem 99.99%). No provider cell ran. Removing the Rust build trees alone was insufficient: campaign 41273604815 also failed during provider-pack export with exit 102. Commit `5bbb549c54` also consolidates dependency installation and pack construction so the disposable workspace/store can be removed in the same layer. All 122 image/oracle/catalog checks pass. Image and remote-pack publication then passed in campaign 41275196741 after cleanup changes plus a scoped 60-to-120 GiB expansion of only its delete-on-termination builder disk. The VM terminated and the expanded volume was deleted. This does not establish that cleanup alone makes the default 60 GiB builder sufficient. - The Daytona admission probe then failed before agent creation, with unconfirmed provider cleanup. The queued continuity case was canceled because it shares this prerequisite. No artifact-transfer or continuity success is claimed. - Status-diagnostic regression: the original generic wrapper failed the HTTP-status assertion. After the repair, the full plugin suite passes 501 tests, with 7 skips; its standalone TypeScript check passes. Controls reject invalid status values, avoid accessors, preserve the original create status and exclude credential canaries. - Previous-head Greptile: 5/5; its summary names `28fa001457f59df1c300c016dc569049e72d0061`. All review threads are resolved. - Live artifact transfer and three-turn warm continuity: NOT VERIFIED. The diagnostic case in https://github.com/paperclipai/paperclip/actions/runs/41277598289 confirmed Daytona HTTP 401 before agent creation. The configured `DAYTONA_API_KEY` in the existing `runner-e2e-paid` environment needs verification or refresh. The queued continuity control was not run after its shared admission prerequisite failed. No further provider attempt is planned until the credential changes. Unconfirmed provider cleanup remains recorded. ## Risks - Source-provenance changes can repeat the provider dependency install. The combined build layer trades install-cache granularity for lower retained disk use. - Linux descriptor semantics and Node are required on the execution host. Unsupported providers and restricted runtime profiles fail closed. - Storage reads buffer at most 10 MiB. A provider outage can leave a reserved staging file. Partial bytes never become the destination. - A lost publication receipt can leave a complete file with an unknown result. Recovery verifies the existing file without installation. Conflicts are never overwritten. - Artifact contents remain untrusted. The copy does not bypass screening or authorize downstream execution. - Live qualification uses the governed Codex API profile. It does not establish every model, subscription profile, platform or historical failure cause. ## Model Used OpenAI GPT-6 in Codex assisted with diagnosis, code, tests and qualification. The desktop runtime does not expose a more specific model deployment ID or context-window size. The model used reasoning, source inspection, tools and code execution. The native implementation agent is configured with OpenAI `gpt-6.1-sol`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
67948173f4 |
fix(sentry): report saved-message task attempt failures (#18541)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tasks retain user messages when an execution attempt cannot finish. > - The task interruption projection drives the saved-message failure notice. > - That state can occur without a new terminal run transition, which the existing Sentry reporter requires. > - This pull request reports that failed task state through the existing Sentry capture path. > - Operators can find the failure through the same tag and saved run links. ## Linked Issues or Issue Description Refs #13446 and #16300. The related PR search found no open implementation of this change. **What happened?** A task can show “Couldn’t finish this attempt. Your message is saved.” without a matching run-failure Sentry event. A stale saved-message wait can produce this state after the previous run succeeded. **Expected behavior** When run-failure reporting is enabled, this failed task state also reports with `event_kind=run_failure`. **Steps to reproduce** 1. Enable the backend Sentry DSN and `PAPERCLIP_SENTRY_RUN_FAILURES=unexpected` or `all`. 2. Leave a saved-message wait on an unfinished task after its last run has ended. 3. Read the task after the wait grace period expires. 4. The task shows the failure notice. Before this change, that read does not report it to Sentry. **Paperclip version or commit** Base commit: `e122478368`. ## What Changed - Report the system-owned failed presentation from the existing batch task reader. The pure projection remains unchanged. - Reuse `captureRunFailure()` with `event_kind=run_failure`, `failure_source=task_interruption`, and `error_code=task_attempt_failed`. - Preserve the actual saved run status and company, agent, task and run references. Load the run agent's adapter through a company-scoped join in the existing run query, including after reassignment. Add no database query or background timer. - Send fixed error text. Do not read chat bodies, prompts or raw run diagnostics for this event. - Suppress repeated reports for the same company/task/run/intent with a bounded process-local cache. Failed or disabled captures do not consume the report. - Document the enabled modes, event fields, duplicate behavior and cache limits. ## Verification - Regression before implementation: the two new Sentry tests failed because the capture entry point was absent; the existing 38 Sentry tests passed. - Focused verification: 332 tests passed across `run-failure-sentry.test.ts`, `task-interruption-query.test.ts`, `task-interruption-presentation.test.ts`, and `run-failure-report.test.ts`. The reassignment regression first reproduced the missing adapter, then passed after the query correction. - Command: `PAPERCLIP_TEST_POSTGRES_ADMIN_URL=<local-test-admin-url> pnpm exec vitest run server/src/__tests__/run-failure-sentry.test.ts server/src/__tests__/task-interruption-query.test.ts server/src/services/task-interruption-presentation.test.ts server/src/services/__tests__/run-failure-report.test.ts`. - PostgreSQL tests used isolated databases on a local PostgreSQL 17 server, with automatic cleanup. Initial embedded starts failed because installation scripts had not restored library links; after repair, the host semaphore limit prevented new clusters. The existing external-database helper supplied the working test path. - `pnpm -r typecheck` passed. - `pnpm build` passed. - `git diff --check` passed. - `pnpm test:run` was attempted. It stopped in the unchanged chat integration shard collection step with “No test found” for line filters. The complete local suite has not passed. - Sentry capture assertions use the SDK test double. Live Sentry ingestion and a deployed browser journey were not tested. ## Risks - An enabled instance can send more Sentry events when task reads discover retained failures. It can emit a task-state event alongside the separate terminal run event. The source tag distinguishes them. - Deduplication is process-local and retains 1,000 reports. Restarts, multiple replicas or cache eviction can report a retained failure again. - No schema, permission, execution or UI changes. The existing opt-in reporting gate stays in place. ## Model Used OpenAI GPT-6 through Codex, with tool use, code editing and test execution. The exact runtime model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1a6017d646 |
fix(server): resume suspended sessions after controller replacement (#18554)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A web server replacement must preserve a completed conversation as well as active execution. > - A warm remote Runner can suspend before the replacement admits another turn. > - The replacement restores its controller checkpoint from PostgreSQL, but lacks the Runner suspension receipt. > - It therefore tries to adopt a live process and rejects the safely suspended source. > - This change restores the verified receipt and preserves retained launch files while the old authority is reconciled. > - Later turns can keep the same provider conversation after the old web server filesystem is deleted. ## Linked Issues or Issue Description Refs #18552, #18543, #18366. **What happened?** The real rolling-deployment journey passed both HTTP handoffs, with 765 reads and 78 verified saves. The first three agent turns kept their native and provider sessions. After the second replacement, the fourth continuation failed with `remote_warm_session_not_idle`. The remote Runner was suspended with an empty outbox and no pending terminal. A focused regression reproduced that exact failure. After restoring its suspension receipt, the same test exposed a stale launch-file read during source recovery. **Expected behavior** A replacement can resume the same provider conversation from a settled suspended Runner. It must validate the physical sandbox and all persisted identity fields. A live source must still pass the existing adoption checks. **Steps to reproduce** Enable the Linux Runner fixture for `native-runner-restart-recovery.integration.test.ts`. Run the `diskless-warm-suspended` cases. The fixture completes and gracefully suspends each source, deletes the controller projections, and admits a later turn from another process. It does not write a synthetic suspension receipt. ## What Changed - Reuse the existing per-turn suspended-receipt reader before warm admission chooses live adoption or suspended recovery. - Keep the receipt read retryable when a failed remote read leaves only the controller projection. - Leave every non-suspended warm state to the existing live-adoption checks. A preliminary backpressure observation cannot reject admission before that owner inspects the current Runner state. - Preserve current sandbox binding, settled-outbox, exact-identity, local-evidence and admission checks. - Pass the structured launch specification to remote asset preparation. Source reconciliation keeps the old authority's retained launch files; a current-authority launch still stages its current files. - Add two Linux Runner/PostgreSQL regressions for two controller-cache deletions and three turns on one provider thread, including an interrupted receipt read. - Update the deployment qualification record, including the failed live evidence and the successful staging database-idle observation. ## Verification - Before the fix, the focused regression fails with the exact live `remote_warm_session_not_idle` error. - Both new regressions pass with real PostgreSQL and Linux Runner. The provider is deterministic; this is not a live model qualification. - The first receipt-only fix failed on the deleted controller skills cache. The completed fix passes that same scenario without recreating fixture launch assets. - All 1,067 surrounding launch supervision, warm recovery, launch-asset and controller recovery tests pass. The exact `cc23019691` server build passes. All ten Linux continuity cases pass, including live adoption, graceful suspension, per-turn resume, changed configuration and failed-read retry. Each case checks provider-thread identity and exact turn counts. The live Codex/Daytona journey passes on the same clean source and exact build: 518 reads, 53 saves and 53 verified saves with zero probe failures; both predecessor homes deleted; the active execution completes once; all four turns preserve the provider conversation and native receipts; saved files and the final command output are independently checked. No automatic retries. Child credential and object-storage bucket cleanup pass. Browser reconnection remains a separate hosted acceptance check. - The review refinement has a failing-before regression: only the preliminary read reports transient backpressure; the subsequent adoption reads the real Runner state. It and both suspension cases pass on the final source (three cases), with server typecheck. The existing idle check remains unchanged. An initial retry hit a fixture port left by the intentional pre-fix failure; a fresh container resolved that infrastructure error. - Independent cleanup verification found two archived setup sandboxes left by the live harness. Both were deleted and confirmed absent. The failed automatic-cleanup check remains recorded. - No claim of the ten-deploy Railway campaign or production canary yet. ## Risks - A cold warm-session admission can now make one bounded remote state read when local lifecycle evidence is absent. Its failure stops admission and remains retryable. - This path cannot replace contradictory local evidence, transfer authority, or treat a suspended process as a live one. - Source recovery must keep the previous authority's launch files until reconciliation completes. The launch identity selects that behavior; ordinary launches still prepare current assets. - No schema migration or public API change. ## Model Used OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution and integration-test tools. The exact runtime model variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge CI qualification note: first final-head run `41322376054` passed the native/runtime and remaining suites but failed the unrelated `automatic_message_on_reactivation` browser fixture at its pre-reactivation deferred-queue count (0 rather than 1). The unchanged test passed locally against isolated PostgreSQL (1 test, 43.6 s); inspection found an asynchronous company-pause/fixture-seeding timing window. Original CI logs and local evidence are retained. The single failed-job retry passed; all final-head CI gates are green. Both reviews are 5/5 and no review threads remain unresolved. The first local embedded-Postgres launch failed before tests during cluster initialization; the isolated external-Postgres test passed, and its disposable database was removed after correcting the private launcher cleanup call. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c0550e7010 |
fix(server): preserve sessions and defer busy overlap recovery (#18552)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A web server replacement must preserve remote execution and later task continuation. > - The active-run handoff now survives deletion of the departing server home. > - The next admission still compared disposable instruction and skill paths as configuration. > - A native receipt could also settle without the old adapter returning its terminal metadata. > - A replacement can also collide with a run row locked by the serving controller. > - This change preserves durable session identity and terminal evidence, and defers locked recovery work so overlap can reach readiness. ## Linked Issues or Issue Description Refs #18543, #18366, #9977, #18118. **What happened?** A real Codex/Daytona journey replaced both control-plane servers and deleted their homes. Reads, saves, saved files and active execution passed. A later useful answer used a new provider session. All four run records lacked the native terminal contract. The whole qualification therefore failed. A later attempt also exposed startup failure: the replacement waited for a row lock held by the serving controller and aborted on its one-second lock timeout. **Expected behavior** An unchanged managed instruction entry and immutable skill version retain the session across server cache relocation. Actual content and configuration changes still reset it. A committed native receipt retains its terminal contract even when the originating adapter never returns. A busy recovery row defers through the existing scheduler while the replacement reaches readiness. **Steps to reproduce** Run the `rolling-web-deployment.runner-codex.daytona.recover-controller` Product E2E cell with separate controller homes and shared PostgreSQL/S3. The focused regressions reproduce the failures without a model call. ## What Changed - Read the managed instruction entry digest from the company-scoped durable file head. Do not restore or read the cache for this check. - Omit disposable instruction paths only when that durable authority exists. Fingerprint selected immutable skill versions in both runtime skill inputs. - Preserve the existing local and external instruction path checks, and unversioned skill identity. - Publish the accepted native terminal contract in the same transaction as the canonical result receipt. - Skip locked recovery rows and schedule the same run for retry. Avoid an audit write that would block on the same row. No ownership is transferred while busy. - Keep replacement readiness independent of an in-progress skill selection; migration and cache eviction still require exact pins. Missing durable skill versions still refuse startup. - Project only the instruction digest in SQL instead of loading the entire agent-file manifest per admission. - Add real PostgreSQL lock contention tests for the issue, run and coordinator, including the heartbeat startup caller and later claim. - Limit native test teardown to its owned database on a shared test host. The old behavior also fails on clean master. - Add cache-loss, scope, changed-content, changed-version and finalization regressions. Update the deployment qualification record. ## Verification - Both focused regressions failed before the source changes. - All 249 tests pass across `heartbeat-workspace-session`, `agent-file-durability`, and `paperclip-control-plane-port`. - Server typecheck and exact-revision build pass. - Assembled admission ran 59 passing cases but failed during native-suite teardown. The teardown failure reproduced on clean master. All 16 native resumption cases and teardown pass after the database-scope fix. - All three row-lock regressions fail with the pre-fix lock timeout and pass with deferred recovery. The full recovery suite passes 74 cases, including the startup caller; eight Docker-specific cases were not enabled. - The real attempt at `d2165dec5d` failed when B aborted on the run lock. Its credential and bucket were removed; all three provider sandboxes were independently confirmed absent. The next real attempt at `aae8a53d01` failed a different readiness gate while a run was preparing skill pins. That race now has a failing-before regression; all 33 readiness, durable-file and managed-skill tests pass after the fix. Its owned resources are cleaned. The `a15a12651c` live attempt passed both web replacements: 765 reads, 78 saves and 78 verified saves with zero HTTP probe failures. The first three turns preserved the provider session and native terminal contract. The fourth continuation failed because a safely suspended remote Runner was treated as live/idle; that distinct recovery gap is being handled in a follow-up. All three owned sandboxes were independently confirmed absent. No claim of full Railway qualification yet. ## Risks - The new fingerprint shape may reset a session once when upgrading from older fingerprint metadata. It is stable across subsequent cache replacements. - A versioned skill is identified by its immutable selected version; materialization and admission still enforce availability. Unversioned skills retain their existing path boundary. - A native terminal describes the accepted execution result. Workspace and issue settlement remain separate authorities. - This does not yet prove the ten-deploy staging campaign, natural database autosuspension or the production canary. ## Model Used OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution and live integration-test tools. The exact runtime model variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7751fc3107 |
fix(server): complete volume-free controller admission and recovery (#18543)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its volume-free web server must admit remote warm execution without
depending on a local disk.
> - Fresh run rows retain the initial legacy label during workspace
preparation.
> - The native runtime selector commits the execution mode later.
> - The earlier guard rejected valid remote runs before that selection.
> - This PR checks storage during preparation and the resolved lifecycle
during runtime selection.
> - Agent-file recovery also has to reconstruct disposable cache paths
from the retained remote directory.
> - The deployment test removes the complete predecessor home, including
immutable caches, to enforce that contract.
## Linked Issues or Issue Description
Refs #18518, #18529, #18533.
**What happened?**
A real Codex/Daytona deployment test failed during setup because the
volume-free workspace guard checked the initial run label. After fixing
admission, controller retirement first failed on an immutable skill
cache. Removing that cache exposed a production recovery failure: B
rejected the saved agent-directory path from A, even though the remote
execution identity was unchanged. The next live attempt recovered that
directory and transferred the same runner, but adoption then tried to
recreate provider launch assets from the deleted controller home.
**Expected behavior**
Allow remote workspace preparation before native selection. Reject
actual legacy and per-turn execution at the runtime boundary. Delete the
entire stopped predecessor home, including immutable caches, without
following links to shared dependencies. B must reconstruct its local
agent-file projection from the exact registered remote lease while
preserving the save baseline and unsaved remote edits. Reattaching an
already-running provider must not prepare a new launch or require the
old launch files. The full backend must recover the admitted
instructions from the matching durable run checkpoint.
**Steps to reproduce**
Run the explicit
`rolling-web-deployment.runner-codex.daytona.recover-controller` Product
E2E cell with a complete server/UI build, disposable PostgreSQL/S3, and
an owned Daytona fixture. The new boundary regressions also reproduce
the premature guard and missing runtime rejection without a model call.
**Paperclip version or commit**
The failures were observed on
`247552fe0719dcc92f7b42c50b658be6f624986d`. This branch is based on
`e122478368`.
## What Changed
- Validate Runner and remote workspace authority before runtime
selection without trusting the initial run label.
- Reject resolved legacy execution and local-source fallback, preserving
the native warm-policy check before admission.
- Rebind remote agent-file caches on ephemeral controllers and retrieve
their validated contents without uploading over the retained execution.
Preserve pending saves when the remote collector is unavailable.
- Restore over-quota remote directories without applying save quotas to
run admission; ordinary saves still enforce quotas.
- Skip provider launch asset materialization when adopting an existing
Runner. A real-process regression deletes the old skill/instruction
sources before reattachment.
- Recover admitted system instructions through Runner-owned checkpoint
validation for the same run, session and context digest. Fresh launches
still compose current instructions.
- Make the attention-retention test advance past its database-assigned
deadline, fixing a clean-master clock mismatch.
- Serialize remote collection and cleanup across controllers with an
operation-scoped Postgres advisory lock; fence stale cache-path updates.
- Keep synthetic deployment probe tasks in a separate disposable company
so they cannot block the business completion oracle.
- Share the launcher's directory cleanup with rolling retirement. Reject
a symlink root and skip linked descendants.
- Test admission with real PostgreSQL transactions and cleanup with real
read-only directories and external symlinks.
- Update developer documentation and the canonical qualification record.
Broader deployment acceptance remains in progress.
## Verification
- `pnpm -r typecheck` and `pnpm build` passed.
- All 156 workspace preparation, runtime selection and ephemeral
contract tests passed. Two new regressions failed before the source fix.
- Seven focused Product E2E cleanup/replacement tests and `pnpm
test:e2e:runner:typecheck` passed.
- `pnpm test:run` stopped at an unrelated 15-second chat integration
timeout after 338 passes. That exact test passed in isolation without
increasing its timeout. The later full-suite shards did not run.
- `pnpm test:e2e:runner:unit` reported 5,898 passes and 13 failures.
Clean master reproduced 11 failures in unchanged onboarding
expectations, operating receipts, Pi catalog assertions, workflow
bootstrap counts and embedded PostgreSQL startup. The other two workflow
assertions were fixed by the intervening master commit. These failures
are retained, not counted as passes.
- All 122 workspace-preparation tests pass after the reviewed
host-fallback fix. The first broader attempt exposed a test environment
leak; restoring the ephemeral flag between cases resolved it.
- Agent-directory and instruction regressions passed 179 tests. Five
focused cache-loss cases pass: different home, same path on an empty
filesystem, foreign lease, concurrent controller, and missing collector.
- The live fixture at `a18f6b85a3` gracefully stopped A and deleted its
entire home, then failed the 420-second remote reattachment deadline on
the saved local agent-directory path. That failure is retained; exact
owned sandbox cleanup was independently confirmed. The next live attempt
at `e14ca5f3e9` restored the cache and transferred the same runner, but
failed while rematerializing launch assets from A. Both failures are
retained and their owned sandboxes confirmed absent. No active-overlap,
ten-deploy, or production-canary acceptance is claimed.
- All 71 related Runner adoption/recovery cases pass, including the new
removed-context regression (failed before the source fix).
- The revised full agent-directory suite passed 110 cases and failed
three cleanup timing cases. All 20 cleanup/cache-loss cases then passed
in isolation with unchanged timeouts, including over-quota recovery.
- Server and Runner TypeScript checks pass after the latest corrections;
the live retry remains pending.
- The live test at `67714af9ff` adopted the same remote execution after
A was deleted, then failed while the full backend read the retired
instruction bundle. All owned sandboxes were independently confirmed
absent. Its regression now passes with actual backend recovery and no
new thread or turn. All 59 backend factory tests and focused checkpoint
identity tests pass. Runner and server TypeScript checks pass; the exact
committed server build passes. A fresh paid-model retry followed.
- The attention-retention CI timeout reproduced on clean master. The
deadline correction passes all 53 tests in that file with unchanged
timeouts.
- The real `cd690de4d3` run transferred the same runner/session after
deleting A, completed, and completed the next revision. During handoff,
372 reads and 38 saves had zero failures; all 38 saves were verified.
The full attempt was interrupted because the probe's open task
contaminated the business company oracle. This remains incomplete
evidence. The scope regression fails before the fixture fix, then
passes; all nine deployment support tests and E2E typecheck pass. All
three owned sandboxes were confirmed absent and temporary
credentials/storage were cleaned up. A clean corrected attempt is
running.
## Risks
- The runtime guard must still reject legacy and per-turn execution
before dispatch. Tests cover these refusals and a valid resolved warm
policy.
- Directory permission changes apply only during cleanup of stopped,
test-owned homes. Traversal does not follow symlinks. This changes no
production cache permissions.
- Cache reconstruction must not change remote execution authority or
rebase the original save baseline. Tests verify foreign-lease rejection,
preserved receipts, and absence of local-to-remote upload.
- Postgres locking is scoped to active remote file work and adds no idle
polling; local/durable-server locking remains unchanged.
- No schema or public API changes. External conversation exemptions
remain unchanged.
## Model Used
OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution
and live integration-test tools. The exact runtime model variant and
context-window size are not exposed in this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Latest real qualification at
|
||
|
DottaandPaperclip
|
247552fe07 |
test(server): avoid deep traversal in durable file assertions (#18533)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Cloud web servers restore agent files from shared storage after replacement. > - The durability test checks restoration, corruption, cancellation and retry. > - Its generic equality assertions traverse every byte in many large buffers. > - The post-merge check for #18531 exceeded the 15-second test timeout. > - This change uses native buffer equality and keeps every byte check. ## Linked Issues or Issue Description Refs #18531. The post-merge Cloud-source verification run timed out in the new cold-folder test. Other shards passed. The unchanged test passed locally. This change removes avoidable assertion cost; it does not increase test or production timeouts. ## What Changed - Use `Buffer.equals` for the cold-folder content assertions. - Keep the concurrency, cancellation, cleanup, old-projection and retry assertions. - Record packaged-image proof, the hosted deployment state and the original CI failure in the existing plan. ## Verification - Eight durability tests passed against a real local PostgreSQL fixture. Object storage uses controlled asynchronous streams. - The same test file took 10.03 seconds before and 5.90 seconds after this change. The changed cold-folder case took 561 ms after the change. - The first local attempt could not start embedded PostgreSQL and ran no tests. The existing external test database enabled the comparison. - Original failed CI: https://github.com/paperclipai/paperclip/actions/runs/41296253723 . That failure remains in the qualification record. - Server typecheck and build passed. All latest-head PR checks passed, including the full test, typecheck, build and browser gates. One cancelled runner job was retried after its log showed an infrastructure shutdown; the retry passed. This PR changes no production code or database schema. ## Risks - A failed content check now reports a boolean mismatch instead of a large byte diff. Equality still covers buffer length and every byte. - Local timing is evidence for the fixture, not a general CI performance guarantee. - Hosted active-agent and rollout acceptance remain incomplete as recorded in the plan. ## Model Used OpenAI Codex, based on GPT-6. The exact runtime snapshot and context-window size are not exposed. Used code editing, shell execution and test analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1d348bbef0 |
fix(server): restore durable agent folders with bounded concurrency (#18531)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A replacement web server restores agent files from shared storage. > - The old restore reads each object in sequence. > - Modest network latency makes a 1,024-file folder exceed its deadline. > - This pull request uses at most eight concurrent restore streams. > - Every file must pass its size and hash checks before the folder is published. ## Linked Issues or Issue Description Refs #18518 and #18526. Local deployment qualification found that a 64 MiB folder could not be restored after the old server filesystem was removed. ## What Changed - Restore at most eight files concurrently within the existing file authority service. - Cancel sibling transfers on failure. Await all workers before removing the temporary folder. - Keep the existing deadline, file-transfer database pool, authorization and publication rules. - Add regression coverage for bounded concurrency, corruption, settled cancellation, retained previous bytes and retry. - Document the behavior and update the deployment plan with local qualification evidence. ## Verification - Real PostgreSQL and S3-compatible storage: 1,024 files of 64 KiB each, plus instructions, with 25 ms added to each object GET. Delete the entire predecessor home and disable local adoption before reading through the successor. - Before: the read failed after 30.065 seconds. After: the same case passed in 4.180 seconds and fetched 1,025 objects. All owned test resources were cleaned. - The durable-file suite passed eight tests against PostgreSQL. Server TypeScript compilation passed. All 56 latest-head CI checks are complete and green or intentionally skipped, including full typecheck, build, unit and browser suites. Greptile is 5/5 with no review findings or open threads. - The plan separately records a released-container overlap proof: 130 reads, 26 saves, zero failures, two deleted predecessor containers and a retained authenticated session. That image predates this restore change. ## Risks - A cold folder can use eight simultaneous streams instead of one. Two file-transfer database connections bound the simultaneous restore callers. Streams verify bytes without loading entire objects into memory. - The 30-second deadline is unchanged. The measured size and latency do not qualify every folder at the maximum storage limits. - No schema, API or UI changes. Paid-model and hosted deployment acceptance remain separate work. ## Model Used OpenAI Codex, GPT-6. Tool use, code execution and repository review. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d941ed3879 |
feat(server): verify existing-volume ephemeral enrollment (#18529)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud web servers can now keep durable state in PostgreSQL and S3.
> - Existing servers still have files and a decision signing key on
their volume.
> - The volume must stay attached until every data owner verifies its
migration.
> - This pull request adds a dedicated maintenance process for that
transition.
> - Cloud can preserve the original key and remove the volume only after
a matching receipt.
## Linked Issues or Issue Description
Refs #18518 and #18526. Existing-volume enrollment is a required step in
the zero-downtime web deployment work. This is the Core maintenance
side; the Cloud transition is a separate private change.
## What Changed
- Package a standalone maintenance entry that starts no normal app
routes or execution workers.
- Coordinate the existing agent-file, run-evidence and skill migration
owners. Follow every page and verify shared bytes before final
eligibility.
- Require stopped writers. Refuse active execution, missing local bytes,
local workspaces and external instructions.
- Read the existing decision key without generating a replacement.
Preserve normal key creation for ordinary server startup.
- Require an attempt token for status and export. Export the original
key only after successful migration. Keep status and errors free of
secret material.
- Document the Cloud ownership boundary, deadline, retry and retention
requirements.
## Verification
- Full workspace typecheck and build passed before the review fixes. The
final server TypeScript compilation passed after correcting Drizzle's
optional predicate type.
- Production runtime bundling passed: 2,702 inputs, 122 chunks,
including the maintenance executable.
- The focused suite passed 21 cases using PostgreSQL and real
S3-compatible storage. It covers multi-page migration of 26 agents,
exact reads after deleting the predecessor home, corrupt-object
rejection, active/missing-data refusal, and the bundled plain-Node
process with authenticated export and clean SIGTERM exit.
- Regression cases refuse unsaved terminal working copies, require
explicit stopped/cleaned receipts for unavailable copies, clean 101
expired traces through the owning service, and enforce the deadline
after migration succeeds.
- Initial integration attempts exposed fixture cleanup ordering
problems; corrected runs pass. CI also caught a Drizzle predicate type
error, now fixed and compiled locally.
- All repository CI gates passed on
|
||
|
DottaandPaperclip
|
73cd8eb9a5 |
fix(server): deliver answers promptly during web handoff (#18530)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its web servers can overlap while Railway replaces one deployment.
> - An answer can reach the replacement while the original server owns
the native execution.
> - A competing delivery claim can make the owner miss the first
notification.
> - The normal retry interval then delays the answer by 30 seconds.
> - This pull request gives pending answers a one-second retry in
ephemeral mode and verifies delivery between separate processes.
## Linked Issues or Issue Description
Refs #18518 and #18526. This closes a measured native-question delivery
delay during control-plane web deployment overlap.
## What Changed
- Register question delivery through its owning service. Use a
one-second retry for ephemeral servers and retain the configured cadence
for ordinary servers.
- Keep the existing durable claims, response identity, authorization and
continuation behavior.
- Add a two-process PostgreSQL regression for a claim held by the
non-owning server. Verify one command, settlement, idle behavior and
reactivation by another committed answer.
- Keep the assembled startup fixture on the real registration function.
Update the deployment plan with current evidence and remaining
acceptance work.
## Verification
- The same local two-process race took 30.157 seconds before the fix and
1.197 seconds after it. The command sink is simulated; PostgreSQL,
notification transport, interaction response, delivery claim and worker
are real.
- The cross-process regression and assembled startup suite passed 122
tests. No further sweeps or pending inspections occur after settlement.
A second answer wakes the owner through PostgreSQL without a test-owned
sweep.
- Server TypeScript compilation passed. All PR CI gates passed on
|
||
|
DottaandPaperclip
|
16787e4167 |
Qualify rolling web deployments and preserve Stop through handoff (#18526)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server owns request handling and remote execution coordination. > - Deployment must start a ready replacement while the old server still serves. > - An outage test does not prove that two live servers can overlap safely. > - This pull request adds a separate Product E2E cell for that boundary. > - The existing run, file, recovery, and retained-session checks still apply. ## Linked Issues or Issue Description Refs #18518. The runtime prerequisite is merged; this PR qualifies its ephemeral server and deployment contracts. ## What Changed - Add the explicit `rolling-web-deployment` Codex Daytona workflow. - Start independent A/B/C homes and compiled builds behind a test ingress. - Wait for readiness before routing, then drain and delete the old home. - Observe active authority before handoff and use the existing read-only recovery oracle. - Require ordinary reads and saves to succeed throughout replacement. Read back every accepted save exactly once. - Preserve the five-minute outage suites and their existing validators. - Add a model-free supervisor smoke and an unready replacement control. - Require graceful exit before deleting an old home; forced cleanup cannot satisfy the oracle. - Preserve a durable Stop through lease release/expiry until the recovered execution owner acknowledges it. Permit only fenced reattachment of the original execution for Stop cleanup; submit no new turn. - Exclude events saved by A during shutdown from evidence of B making progress. ## Verification - E2E TypeScript check passed. - Affected harness checks: 143 passed. Catalog, fixture and report checks: 120 passed. - Actual supervisor smoke passed two replacements: 436 reads, 45 saves, zero request failures, and confirmed owned resource cleanup. - B became ready in 18.7 seconds; C became ready in 109.2 seconds. A prior C timeout remains failed evidence. - Fresh browser proof: 11 checks passed, 246 reads, 50 acknowledged saves, zero failures, 1,058ms reconnect, both old homes deleted and cleanup confirmed. - Stop lease-gap regression: three PostgreSQL delivery tests passed with a simulated provider edge. Current server TypeScript build/vendoring passed. - Full harness: 5,814 passed, 17 failed, four skipped. Two shutdown races passed the focused file; two report timeouts passed an isolated 42-test rerun. Onboarding/workflow-security/Pi/date-expired fixtures are unchanged on inspected master; the host embedded PostgreSQL fixture remains a setup limitation. This is not a green full local suite. - CI at `172509b8be` passed all gates. Greptile is 5/5 with every review thread resolved. The earlier access-service race failure passed its isolated regression and the subsequent full CI. - Final review corrections: six focused executor cases, three PostgreSQL command-delivery cases, 19 harness cases and server/E2E typechecks passed. - Three live attempts failed before agent creation: two harness setup gaps were fixed, then provider authentication failed. The saved OpenAI key returns 401; a refresh is pending. The Daytona probe reported cleanup uncertain, and no sandbox with that attempt label was found through the authenticated staging manager. No live model execution is claimed. - Live Runner overlap, released-container qualification and hosted deployment proof remain pending. Merging this opt-in qualification lane does not claim those results. ## Risks - The source lane shares workspace dependencies and does not establish container packaging correctness. - The live workflow needs an explicitly owned S3 bucket, PostgreSQL, pinned remote image and provider credentials. - Qualification cleanup stops all owned processes after failure. A failed replacement must not be reported as a pass. ## Model Used OpenAI Codex, GPT-6. Tool use, code execution and repository review. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2b2b5a43b7 |
fix(daytona): restore the recorded directory for file recovery (#18527)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A stopped sandbox can hold workspace files that still need a durable copy. > - File recovery must reopen the exact allocation and directory recorded by the run. > - Workspace preparation can place that directory below the allocation root. > - Daytona resume returned the allocation root even when recovery needed the recorded child directory. > - This pull request lets the file-recovery owner request that exact directory. > - Ordinary resume keeps its allocation-root behavior. The recovery source checks stay strict. ## Linked Issues or Issue Description **What happened?** A stopped native run had a prepared remote workspace. The recovery service stopped and retained its allocation. The next file-recovery step failed with `source_binding_invalid` and `reopened_cwd_mismatch`. Daytona resumed the correct allocation and matched its sentinel, but returned the allocation root instead of the workspace directory recorded on the lease. **Expected behavior** The file-recovery owner must reopen its exact recorded directory without creating an allocation, an empty directory, or an agent turn. Ordinary execution resume must keep returning the allocation root for normal workspace preparation. **Steps to reproduce** 1. Prepare a task workspace below a Daytona allocation root. 2. Stop and retain the allocation with unsaved workspace files. 3. Request source-bound file recovery. 4. Before this change, the provider returns the allocation root and the strict cwd check rejects it. **Paperclip version or commit** Reproduced from `050365e09d88411960fc3bacc1c08c46c0d9b6fa`. **Deployment mode** Docker with the bundled Daytona provider. Local tests use synthetic provider boundaries and disposable PostgreSQL. Refs: #18517, #15749. Related retained-export work: #17916. That work does not change the Daytona resume handler. ## What Changed - Add an optional driver-only `requestedCwd` resume field to the SDK contract and existing host forwarders. - Set the field only after the owned file-recovery path verifies the recorded source. - Require Daytona to match the requested directory to the recorded lease directory. Require a normalized absolute path without NUL bytes. Keep the allocation and sentinel checks. - Keep ordinary resume unchanged when the field is absent. - Add provider, host, and real workspace-preparation regressions. Document the separate directory contracts. ## Verification - The original provider regression failed for reusable and ephemeral leases. The scoped regression also failed to read a real retained file from the wrong root. Ordinary resume controls passed before the fix. - Current provider handler: **345/345 passed**. This includes exact-directory, malformed, missing, foreign, sentinel, reusable, ephemeral, ordinary-root, and real-file cases. - Removing only the owned producer's cwd request makes its real PostgreSQL regression fail with the original `reopened_cwd_mismatch`; the source file was restored exactly afterward. - Current host selection: **10/10 passed** against disposable PostgreSQL 17. It covers the inspected owner, sandbox forwarding, generic plugin forwarding, normal resume without the field, same and later task workspace preparation, and ownership rejection. The same-workspace control retains saved bytes and does not append another workspace path. - Current SDK and Daytona builds, server build, module boundaries, maps, and diff checks passed. The server used its normal unchanged-Runner cache. Its canonical input digest matched the previously verified build, and the canonical assessor accepted the copied original stamp and outputs. The changed SDK and server compiled from current source. - Local package-wide validation exposed two unchanged 30-second timeouts in the 145,000-member tar-listing cases. A single-file control reproduced both timeouts. That test mocks this provider module. The failures remain recorded; no timeout changed. The earlier package run had 497 passes, two failures, and seven skips. It is historical evidence from the abandoned broader resume experiment, not final-source package acceptance. - Initial host attempts timed out during database setup before any assertion. The final host selection used the existing supported pre-migrated disposable template. No test timeout changed. - Greptile reviewed exact head `8d0e577d1234ad394887255f13707c010dee6d24`: **5/5, no actionable issues, zero review threads**. [Current review](https://github.com/paperclipai/paperclip/pull/18527#issuecomment-6555253315). - The official [Ponytail review](https://github.com/paperclipai/paperclip/pull/18527#issuecomment-6555237850) also assessed exact head `8d0e577d1234ad394887255f13707c010dee6d24`: **5/5, no findings**. Its check is terminal neutral; its completed source assessment is not a test-execution claim. - [Primary CI](https://github.com/paperclipai/paperclip/actions/runs/41286802674) initially lost its spot runner in the UI/CLI workspace lane. The log records an explicit shutdown signal and no test assertion result. UI, CLI, lane scripts and dependencies are unchanged; the adjacent identical-source baseline lane passed. The single failed-lane retry passed at the same source. The initial failure is retained. All exact-head CI gates are now green. - No live repair or image deployment is claimed by this PR. Commands: ```sh node packages/plugins/sandbox-providers/daytona/node_modules/vitest/vitest.mjs run --config packages/plugins/sandbox-providers/daytona/vitest.config.ts packages/plugins/sandbox-providers/daytona/src/plugin.test.ts node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/services/workspace-recovery.test.ts server/src/__tests__/environment-runtime.test.ts -t 'recovers a protected retained source|owned export directory|prepares the (same|later) task workspace|does not reopen a retained source|resumes only the exact stopped sandbox lifecycle|explicit export directory' pnpm -C packages/plugins/sandbox-providers/daytona build pnpm --filter @paperclipai/server build node scripts/check-module-boundaries.mjs node scripts/check-maps.mjs git diff --check ``` ## Risks - A provider that does not support the optional field can still return a different directory. Recovery keeps failing closed on that mismatch. - Ordinary resumes omit the field. The tests cover the same workspace and a later task so that normal preparation does not gain an extra path prefix. - The field grants no source ownership, stop proof, file-persistence proof, or execution authority. Existing allocation, company, task, run, claim, and cwd checks remain in force. - No schema migration, retry-policy change, model replay, or historical-row repair is included. Existing failed attempts remain historical facts. Deployed file recovery still needs its normal admission and independent persistence and physical-stop receipts. ## Model Used - OpenAI GPT-6 through Codex. Used reasoning, source inspection, tool use, and code execution. The exact serving model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
006d5691a6 |
feat(server): support volume-free web server handoff (#18518)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. **Current behavior** The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. **Proposed behavior** Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. **Reason and benefit** This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
050365e09d |
Preserve native workspace files after lifecycle Stop (#18517)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent termination must stop execution and preserve unfinished workspace files. > - A native run can acknowledge a lifecycle Stop, then finish without an accepted result. > - The file preservation worker required a board actor even when the exact native Stop receipt was valid. > - This left a prepared remote workspace allocated without a file recovery action. > - This pull request admits the exact audited receipt for file preservation only. > - The existing recovery service can then save the files and stop the same allocation without another agent turn. ## Linked Issues or Issue Description **What happened?** A cancelled native run can have a terminal, unowned coordinator and a prepared remote workspace. Agent lifecycle cancellation records a valid run-only Stop receipt without a board user. The preservation worker rejects that receipt because it uses the board Stop predicate. The source lease remains active. A resolved execution watchdog does not supply the missing file recovery action. **Steps to reproduce** Seed a cancelled native run with a terminal-failure coordinator, no result or owner, an acknowledged and audited run-only Stop, and a prepared remote workspace. Omit board attribution. Start the registered cancellation preservation worker. Before this change, it records no workspace recovery marker. **Expected behavior** Record file recovery for the exact bound allocation. Keep the cancelled result, accounting, task state and watchdog history. Use the normal recovery service to stop, save and clean up that allocation without another provider turn. **Paperclip version or commit** Reproduced on source `1e8e32290c657227857aeda2620a73db52bab4f2` with real local PostgreSQL 17. The observed deployment uses a sandbox provider and remote workspace storage. Related current workspace work: #18514. Public PR and issue searches found no duplicate repair. ## What Changed - Expose the existing exact run-only Stop receipt check. Reject empty audit identifiers. - Use that receipt only in cancelled workspace preservation. Keep board Stop and continuation checks unchanged. - Defer preservation while the run still holds a live controller lease. - Schedule that wait at the actual controller lease expiry. A lease renewal observed under the preservation lock also sets the next deadline. - Add a PostgreSQL regression for the remote-workspace-only lifecycle record, resolved watchdog history, normal recovery admission, ownership controls, rollback and idle behavior. - Document the separate file recovery obligation and its physical cleanup limits. ## Verification - Before the fix: the registered-worker PostgreSQL regression fails because the file recovery marker is absent. - After the original repair at `6fd1042efd`: 20/20 focused tests and all five sibling suites pass, 320/320 tests. The exact-head server build and typecheck pass. - The repository reviewer then found that a live controller caused a fixed 30-second retry. Both the existing-lease and real row-lock renewal cases fail against that source. The new `adb1d4eccdd48be4c86906d5bd111a6e39f20191` schedules the actual recorded deadline before locking and after the locked reread. The lock still enforces the ownership guard. - At `adb1d4eccd`: all 22 focused PostgreSQL controls pass. The directly affected retention and board receipt sibling suites also pass, 124/124 tests across three files. They prove no intermediate reads or transactions before expiry, preservation after expiry, and no extra execution. Server typecheck and the exact-head server build pass. - Map checks, module boundary checks and `git diff --check` pass. - Run `pnpm exec vitest run server/src/services/native-runtime/native-cancelled-workspace-preservation.test.ts server/src/services/native-runtime/native-failed-workspace-recovery.test.ts server/src/services/acknowledged-native-stop.test.ts` with the supported external test-database variable set to a disposable local PostgreSQL database. The earlier broader run also covered run control and workspace recovery. - Exact-head [canonical CI](https://github.com/paperclipai/paperclip/actions/runs/41274963476) completed successfully. All 59 current-head checks are terminal with zero failures; required `ci / verify` and `ci / e2e` pass. Both [Greptile](https://github.com/paperclipai/paperclip/pull/18517#issuecomment-6553528545) and the [repository assessment](https://github.com/paperclipai/paperclip/pull/18517#issuecomment-6553542139) reviewed exact `adb1d4eccd` at 5/5, with zero findings or unresolved threads. The repository assessment's check concludes `NEUTRAL`; the validated assessment itself is 5/5. Earlier reviews, including the valid 4/5 deadline finding at `6fd1042efd`, remain in review history. The [earlier CI run](https://github.com/paperclipai/paperclip/actions/runs/41273965963) is also retained. No old result substitutes for a current-head gate. - Real sandbox recovery and physical cleanup remain unverified. Earlier failed QA outcomes remain failed. This PR changes no test grade or historical row. - Repo-wide local tests and typecheck were not run. The focused server checks cover this change; canonical CI supplies the remaining repository gates. ## Risks - The newly admitted receipt permits only file preservation. It cannot authorize a continuation, accept a native result or certify provider termination. - Recovery still requires exact company, task, run, lease, allocation and workspace bindings. Accepted results and live owners remain with their existing owners. - Provider or storage failure can leave recovery pending. The source must remain retained until saved-file and physical termination receipts exist. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI GPT-6 through Codex. Used reasoning, source inspection, tool use and code execution. The exact serving model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1e8e32290c |
Give metadata project tasks private remote workspaces (#18514)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task can belong to a project without using that project's files. > - The specification gives a task without a selected or configured source its own private folder. > - The remote source planner instead rejects such a task when its project has no configured workspace. > - This stops an ordinary delegated task before native execution starts. > - This pull request selects an empty, task-owned remote source for that narrow default case. > - Explicit source choices, existing workspace authority, and repository requirements retain their current checks. ## Linked Issues or Issue Description Refs #18504 and #18351. The earlier reference-project fix covered additional project mentions. This fix covers the task's own organizational project. **What happened?** A task in a project with no configured workspace fails with `remote_workspace_source_unavailable`. Its project has no workspace policy, its four workspace selection and binding fields are null, and it requests no repository, branch, or local folder. The failure occurs before native runtime selection. A derived project folder exists in the project facade even when no source is configured; that derived path does not grant file access. **Expected behavior** Use the task's private remote workspace. Keep the project on the task for organization. Do not adopt the project folder, agent home, or another repository. Preserve explicit workspace and source requirements. **Steps to reproduce** 1. Create a project without configured workspaces or a workspace policy. 2. Assign a normal task in that project to a native sandbox agent. Leave all workspace selections and bindings unset. 3. Start the task with remote workspace storage enabled. 4. On the prior source, observe the source-unavailable error before allocation. On this source, observe a company-owned remote workspace whose `sourceIssueId` is the task and whose filesystem project and source IDs are null. **Paperclip version or commit** Reproduced against the deployed source planner at `756599e4ea069c9338c320fe8846523d3dfb92a0`. This branch starts at `987e0932bf66e703750cd04698b3c7e8993580a7`; the relevant planner and preparation inputs at that parent match the deployed source. **Deployment mode** Native sandbox execution with remote workspace storage. The regression tests use real PostgreSQL with the current migrations. ## What Changed - Allow an implicit empty source only after a company-scoped query confirms the project has no workspace rows and no explicit workspace policy. - Keep existing bindings, pending selections, explicit source choices, Git requirements, and local folder choices outside this default. - Add real PostgreSQL source controls and an assembled heartbeat preparation control through workspace reservation and the environment acquisition boundary. - Clarify the existing task-owned folder contract in the implementation specification. ## Verification - Causal RED: the new source control and the assembled allocation control both fail against the unchanged deployed planner with the observed source-unavailable error. - GREEN: 19 focused PostgreSQL controls pass across the source planner and assembled heartbeat preparation suites. The allocation control checks the persisted company/task owner, null filesystem project IDs, private remote root, original organizational project, and unchanged project and host sentinel files. - Negative controls retain configured repository sources and reject a configured source without a URL, an explicit missing source, an unknown selection, a required project policy, a required branch, an explicit local folder, a foreign project, and unsupported read-only project sources. - Current-source transitive TypeScript checks and `node scripts/check-maps.mjs` pass. The map check is structural evidence only. - Fresh exact-head Greptile is 5/5 with zero unresolved threads at `680461e180232d0f4a057528617e39b64b4f0e3a`. [Review](https://github.com/paperclipai/paperclip/pull/18514#issuecomment-6552946468). The official [Paperclip assessment](https://github.com/paperclipai/paperclip/pull/18514#issuecomment-6552931726) is 4/5 and remains recorded unchanged. A separate non-author source review found no actionable issue after the full producer/consumer trace, with manual confidence 4/5. Neither static review ran the tests or proved deployed native execution. - Full repository tests, build, and type checks passed in [canonical CI](https://github.com/paperclipai/paperclip/actions/runs/41269733360). All 59 checks are terminal: 55 success, three skipped, and the official Paperclip check neutral; no failed or pending checks. The successful build checked out merge preview `03dce9f9bc708e778dd214ca85dfa1ba58e0a806`, with parents current main `9dbbfa4fb298448db83ace946b338bff321d8b2a` and this exact head. Its four helper blobs match this head; the four added mainline document-harness paths remain unchanged. Local tests use source aliases with the current DB, shared, and Runner source, rather than borrowed generated declarations. - Actual QA staging qualification is pending deployment. The previous failed journey and cleanup evidence remain failed; these local checks do not relabel them. ### Review follow-up - Fresh workspace retry: the normal producer sets `taskDirectory:true` for this empty-source case. [Heartbeat](https://github.com/paperclipai/paperclip/blob/680461e180232d0f4a057528617e39b64b4f0e3a/server/src/services/heartbeat.ts#L2288-L2296) separates the saved filesystem project from the organizational project. [Environment selection](https://github.com/paperclipai/paperclip/blob/680461e180232d0f4a057528617e39b64b4f0e3a/server/src/services/heartbeat/environment-selection.ts#L506) selects the fresh task directory. Preparation passes that flag, and the planner's existing task-directory return runs before the new binding guard. The proposed fresh failure omits that producer flag. The existing fresh-intent and binding checks stay strict. This is source evidence, not an extra runtime test claim. - Empty policy object: the [public project policy schema](https://github.com/paperclipai/paperclip/blob/680461e180232d0f4a057528617e39b64b4f0e3a/packages/shared/src/validators/project.ts#L24-L43) requires `enabled:boolean`. The project PATCH schema makes outer fields optional and does not make nested policy fields optional. A clear uses `null`; `{}` is rejected. A defensive parser's treatment of unknown stored data does not establish a valid `{}` producer. Keep the narrow fallback closed for non-null policies. - Boolean grouping: JavaScript evaluates the existing predicate as `sourceId || (projectId && !implicitTaskDirectory)`. The explicit-source and metadata-default controls cover those branches. The optional parentheses suggestion does not identify a behavior change; this reviewed source remains frozen. - The official 4/5 assessment and its static verification limits remain visible. The independent manual review withdrew its initial fresh-retry concern after the full producer trace. Actual staging qualification still requires deployment and a new real journey result. ## Risks - The new query runs only for a source-less default admission. It reads one authorized project and uses `NOT EXISTS` for that project's source rows. It adds no recurring poll, write, provider call, or local folder access. - A project with an explicit workspace policy remains subject to the prior source requirement. This conservative rule does not reinterpret policy fields. - The allocation control uses the real planner and PostgreSQL reservation. It stops at environment acquisition, so it proves workspace ownership rather than provider filesystem execution. Live staging qualification is still required. - No API, database schema, permission, model, budget, run limit, or QA journey oracle changes are included. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. This session does not expose a more precise backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
34f8d25089 |
fix(github): expose review finding categories (#18429)
## Thinking Path > - Paperclip manages AI agents and their work. > - GitHub review bots submit structured assessments through task-bound tools. > - Each repository can set its own finding categories. > - The schema accepts a string, but the server rejects values outside the saved review policy. > - Review instructions did not list the accepted values. The error did not list them either. > - This change gives agents the values before submission and keeps server validation. ## Linked Issues or Issue Description **What happened?** Review submissions failed with an unknown finding category. The agent corrected the assessment before publication. The sampled trace has two rejections followed by a processed review. Historical tool inputs are null. Authorized diagnostic inspection found no saved raw provider trace for the representative run. The exact rejected values remain unknown. **Expected behavior** Agents can find the accepted categories before submission. Unknown values still fail validation. **Steps to reproduce** Set a custom finding category. Start a review. Submit a different category. Before this change, the instructions and error do not list the accepted value. ## What Changed - List configured categories in manual and automatic review instructions. - Return categories from the saved assessment policy in `begin_review`. - Describe the dynamic constraint in the shared discovery schema. Keep custom categories valid. - List accepted values in category errors. Keep authorization and publication checks. - Add contract and database integration tests. Update documentation and access maps. ### Hypothesis and thesis for a fix The missing category guidance can cause avoidable submission retries. A model can also ignore the policy returned by metadata reads; the historical inputs do not establish which occurred. The repair belongs to the GitHub review contract. Use the saved policy categories at review start, explain the constraint, and retain server rejection. A fixed global enum would reject valid custom categories. ## Verification - Baseline `f1d7808`: two new contract assertions failed for missing category guidance and error detail. The same assertions pass after the change. - `pnpm exec vitest run server/src/services/chat-github-review-policy.test.ts server/src/services/chat-github-tools.test.ts`: 41 passed. - `pnpm exec vitest run server/src/__tests__/chat-channels.integration.test.ts -t 'returns snapshot finding categories'`: 1 passed. Real PostgreSQL and service code; simulated GitHub HTTP. Checks policy changes, rejection before publication, accepted custom category persistence, and processed publication. - `node scripts/check-maps.mjs`: passed. Structural evidence only. - Shared TypeScript check passed. Server TypeScript checking was killed by the 8 GiB memory limit, including a separate retry after the other checks stopped. - Full typecheck and build could not complete: `cargo` is absent. Full tests were stopped after concurrent checks exhausted memory. CI must supply the full checks. - Follow-up compiler repair: the unchanged baseline reproduces TS2352 in the category discovery test. Runtime matchers replace the unsafe schema cast. They retain both guidance and custom-category assertions. The repair is commit `0dcb94c676`, with current master merged at `14d7052854`. The final head is `c46b8f35add9255fc71be294d27f5a7865bcb47f`; the remaining code-comment review finding was removed and resolved. - Final-head CI: all gates pass, with no failed or pending checks. Greptile gives this head 5/5 with no outstanding findings; the only inline thread is resolved. The workspace test shard passed on one bounded rerun after an earlier CLI child-server readiness timeout; no code change was made for that rerun. - The review host passed full recursive typecheck and build after the compiler repair and master integration. Fresh database rechecks hit the host PostgreSQL semaphore limit. The full local test command hit a pre-existing chat line-selection failure before assertions. These local limits do not replace the passing final-head CI evidence. - No live model or GitHub publication at the changed revision. No deployment or merge. ## Risks - Instructions add policy text. An agent can still ignore it. - Existing bot catalogs get the schema description when tools are synchronized. Trusted review instructions and `begin_review` also expose categories. - The service test uses simulated GitHub HTTP. It does not prove that deployed agents avoid retries. ## Model Used The original worker used OpenAI `gpt-6.1-sol` through Codex with reasoning, tool use, and code execution. A later Codex review repaired the test compiler error. That review runtime exposes GPT-6 but no exact snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Dotta <bippadotta@protonmail.com> Co-authored-by: cryppadotta <34892728+cryppadotta@users.noreply.github.com> |
||
|
DottaandPaperclip
|
e09f40888a |
fix(recovery): preserve native task access during resolution (#18489)
## Thinking Path > - Paperclip controls agent execution and task ownership. > - A native run needs its task binding until it reports its result. > - Direct task updates already reject self-status changes that remove this binding. > - Recovery resolution can also change the source task status. > - That route did not use the same protection. > - This change rejects the unsafe status change before the task or recovery receipt changes. > - The run can then use semantic completion while normal recovery permissions remain effective. ## Linked Issues or Issue Description Refs #18411. **What happened?** A running native owner can resolve an older recovery action with `sourceIssueStatus: "in_review"` or `"done"`. The request succeeds and changes its own task status. That can release the execution binding before the run reports. Later task tools fail with `paperclip_runner_tool_binding_not_authorized`. **Expected behavior** Return a conflict before the task status or action receipt changes. Tell the run to report through `paperclip_finish` or `paperclip_block`. Preserve unchanged-status resolution, Board recovery, legacy-agent behavior, and the existing conversation and handoff rules. **Steps to reproduce** 1. Give a running native owner the task execution lock and an older recovery action. 2. Call `POST /api/issues/{id}/recovery-actions/resolve` as that run. Use `outcome: "restored"` and `sourceIssueStatus: "in_review"`. 3. Before this fix, the route returns 200 and changes the status. After this fix, it returns 409 with `native_run_status_reported_by_finish` and makes no task or action mutation. ## What Changed - Reuse the existing native self-status guard inside recovery resolution's source-task transaction. - Let the guard receive the requested status from either route contract. - Add HTTP route regressions for rejected review/completion transitions and continued task editing after rejection. - Cover unchanged-status native resolution, Board recovery and legacy-agent resolution. - Update execution semantics and both recovery maps. ## Verification - Baseline `91fdf02`: the two new rejection cases fail because the real Express route returns 200. Three safety cases pass. - Fixed source: all 198 tests in `server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts` pass. - `node scripts/check-maps.mjs` passes. This is structural evidence. - Greptile and Paperclip Review both give the integrated head 5/5 with no actionable findings. - Full recursive typecheck, full build, and a repeat of all 198 route tests pass at integrated head `0fa104dfb240500acd01303cd446363ea39c6c75`. All current-head CI gates pass (55 successful checks/statuses; no failed or pending checks). - The full local `pnpm test:run` command fails in chat-test line selection with `No test found`, before assertions. This also occurs with the unchanged master chat test file. It is not a passing full local suite. - The route tests use the real route, validation and guard. They mock service and database boundaries. The observed hosted failure establishes the original mechanism. Fixed-source live provider acceptance remains unverified. No deployment or merge is part of this PR. ## Risks - A native owner now receives a conflict for a previously accepted recovery status change. It must use the existing semantic result path. - This prevents a new binding loss. It does not prove that an old remote executor stopped or recover missing provider state. - Board, legacy, conversation and handoff behavior retains existing authority. The regression includes representative allowed paths. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific snapshot ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (198 focused route cases; the full local command has the recorded selection failure) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
756599e4ea |
fix(workspaces): keep project references without source folders (#18504)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can receive project references in normal task and chat messages. > - Remote workspace preparation checks those references before a native run starts. > - A project with no workspace source has no folder to attach. > - The current check rejects that project as an unsupported read-only folder. > - This pull request keeps the project reference and reports the missing source without attaching files. > - Projects with configured sources keep the current remote access restrictions. ## Linked Issues or Issue Description Refs #18403 and #18351. This is a product prerequisite for the hiring journey. It changes no journey code. **What happened?** A native chat run failed before execution when its message referenced an authorized project with zero configured workspaces. The error was `workspace_validation_failed`: “This remote runtime cannot safely provide read-only project folders.” The project had no repository or checkout. The failed QA run and original grade remain preserved. **Expected behavior** Keep the task workspace when a referenced project has no configured source. Report that no project folder was attached. Continue to reject configured cross-project sources that require an unsupported read-only mount. **Steps to reproduce** 1. Use a native remote task folder with multi-project references enabled. 2. Mention a same-company project that has no configured workspaces in a normal comment. 3. Prepare the run workspace. **Paperclip version or commit** Reproduced on `b889ac39b7fbed558c5f4cfbbc2b73c4ca916810` with the real workspace resolver and PostgreSQL migrations. The staging failure was recorded in workflow [41253428446](https://github.com/paperclipai/paperclip-cloud/actions/runs/41253428446). ## What Changed - Distinguish authorized project references with a known empty workspace list from configured sources. - Return a resolution warning and failure receipt for an absent source. Do not attach a folder or create a workspace. - Preserve project authorization, company filtering, the disabled remote-source gate, and the read-only restriction for configured sources. - Add real PostgreSQL regression cases and document the reference contract. ## Verification - Causal RED on unchanged `b889`: the metadata-only project case failed with the same read-only error. The real repository negative already passed. - GREEN: eight PostgreSQL cases pass. They cover an absent source with the remote-source switch both on and off, a saved existing workspace, a repository, a configured source without a repository URL, the disabled remote gate, mixed references, and a foreign company. - The existing assembled workspace test also passes: a real read-only project source is rejected before reservation or environment acquisition. Total: 9 selected tests pass; 119 unrelated tests were not selected. - Tests use PostgreSQL 17 and the current source migrations. The local Vitest configuration selects current DB, shared, and Runner sources because the borrowed dependency output predates the remote workspace feature. - Current-source TypeScript checking of both changed TypeScript files and their dependency graph passes. It uses the repository's real Express actor declaration. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Canonical reproduction: `pnpm exec vitest run --project server server/src/services/heartbeat/remote-workspace-source.test.ts` with the existing embedded PostgreSQL support or `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` set to a disposable test cluster. - Historical `0f2b` passed all 59 CI checks. A separate non-author manual source audit reported no actionable findings at confidence 4/5; it did not rerun tests or qualify live QA. The official 4/5 review requested a flag-scope test and saved-workspace control. This test-and-documentation update adds both, with production source bytes unchanged. - Current-head canonical [CI 41258109282](https://github.com/paperclipai/paperclip/actions/runs/41258109282) passed: all 59 checks are terminal (55 success, 3 skipped, 1 neutral), with no failures or pending checks. Build and typecheck passed. The eight-case regression file passed in the canonical Linux server 1/12 lane. Its checkout `6f69ccca70d10e78885b133fb54a80940bf175a5` has parents `b889` and this exact head; its tree equals this head. - Fresh Greptile and official Ponytail assessments each give 5/5 for exact `27660027d12ff31427d611019e90dabfb6d8503a`, with zero unresolved review threads. The separate manual audit remains 4/5. - Live QA remains pending on a reviewed app build. This regression proof does not claim that the hiring journey passed. ## Risks A project source could be added after project hydration. This change uses the same run-preparation source snapshot as the existing authorization and source check. It only permits a positively known empty workspace list. Configured sources, unknown shapes, foreign projects, and existing execution authority keep their current checks. No schema, credentials, provider, model, budget, or filesystem permissions change. ## Model Used OpenAI Codex, GPT-6 family. The session does not expose a more specific backend model ID or fixed context window. Used reasoning, repository tools, TypeScript and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
66e8fab944 |
fix: complete native chat responses and improve API discovery (#18364)
## Thinking Path > - Paperclip must perform work and report its outcome accurately. > - A chat response and its delegated or operator-owned work are separate stages. > - Native chat contracts instead required literal completion of the user's whole request. > - Agents kept searching or waiting even when a truthful handoff should finish the response. > - OpenCode also has separate acknowledgement backlog and empty-response failure paths. > - This PR aligns chat completion with its current stage and makes OpenCode completion truthful. ## Linked Issues or Issue Description Refs #18098 for retained OpenCode evidence and #18377 for separate eval setup, oracle and reporting repairs. **What happened?** Real pinned OpenCode 1.18.34 reproduced an MCP timeout behind 800 text fragments. Pilot [41222196525](https://github.com/paperclipai/paperclip/actions/runs/41222196525) separately exposed a fabricated successful reply after a reasoning-only response. Later restart attempts timed out without an answer. Their native criterion literally required restarting the instance, although the chat instructions and authority required an operator handoff. General authority guidance alone did not resolve it. **Expected behavior:** A standing chat completes its current authorized response. It performs and verifies permitted actions, verifies delegated handoffs, or explains the specific unavailable action and operator path. The worker's task remains separate. Ordinary execution tasks must still complete their requested work. Empty provider responses must not become invented success. **Version and reproduction:** Actual pinned OpenCode and native Rust Runner, with retained full-stack AWS attempts. Product baseline master `51ea624344`. This PR adds no historical catalog to master. ## What Changed - Build standing-chat completion contracts from the server-owned conversation identity. Scope each current request to the response stage; preserve request text, criterion IDs, order, human-response scope and source revisions. Require verified authorized actions and truthful handoffs. Ordinary task contracts stay strict. Existing admitted inputs retain their saved contract during recovery; new contract content uses the existing revision/hash mechanism. - Guide prospective defaults to their owning resource and preserve existing work unless backfill is requested. Guide unavailable board/operator operations to the supported handoff. These are general chat instructions with no eval identities or expected answers. - Batch adjacent assistant text for at most 50 ms and 4,000 UTF-8 bytes. Preserve text, metadata and order; flush before non-text frames and shutdown. - Give an empty OpenCode response one correction within its existing turn, model, reasoning, context and deadline. Accept actual completed answer parts with incomplete headers. Preserve actual active tool work, bound idle evidence grace and fail truthfully if correction stays empty. - Reject empty OpenCode terminals in Rust without actual prose or an accepted semantic result. Never invent successful replies or evidence. Refresh generated project-discovery contracts. - Preserve accurate public API-discovery descriptions for existing project defaults and managed instance restart operations, including their field scope and authority limits. ## Verification **Qualified: [pilot 41256592937](https://github.com/paperclipai/paperclip/actions/runs/41256592937) passes 3/3, with all 35 independent checks passing.** | Case | Checks | Full case duration | Evidence and cleanup | | --- | --- | --- | --- | | Instance restart | 9/9 | 140.1 s | complete | | Project defaults | 11/11 | 204.2 s | complete | | Project work | 15/15 | 204.0 s | complete | All three are original-attempt results from source `c51e079f7f2364305b131ffd549e263ff78c5335`, containing the exact product files from `13c2b61f31d873a475ea0c3c026b19926ff42301`. Actual saved conversation contracts were inspected, as were ordinary native worker contracts where present. Each protected archive includes the actual API state and final PNG; cleanup completes. Original queries, `openrouter/deepseek/deepseek-v4.1-flash-1017`, finite case clocks, concurrency one and zero eval retries are preserved. Earlier passes do not transfer into this qualification. Both current heads pass all checks: product **62** (56 success, one neutral, five skipped), catalog **56** (54 success, one neutral, one skipped). Both have exact-head Greptile **5/5**, zero unresolved review threads and clean mergeability. The three trusted private-evidence protocol files remain byte-identical to the dispatch workflow. Raw evidence stays protected. - **2,044 API discovery tests pass.** Red controls showed that “restart instance” selected unrelated drain/settings operations and “save project defaults” selected repository replacement. Accurate public summaries now rank the existing operations correctly; route registration preserves descriptions. No endpoint, permission or field behavior changes. - **112 real-database, authenticated HTTP, native Runner and OpenAPI consumer tests pass locally**, using disposable task-owned Docker PostgreSQL. It was removed afterward; shared IPC was untouched. These cover persisted contract selection, admitted native input and the model envelope together. The isolated CI suites also pass. - **77 focused contract/input/context tests pass.** Red controls catch the wrong conversation predicate; canonical identity requires both agent and user IDs, including the actual heartbeat projection. Ordinary tasks, agent-only issues and planner tasks remain strict. Coalesced requests preserve criterion identities, source hashes and human-response scope. - **144 OpenCode driver tests pass**; native core **441 pass, one ignored**; packaging **56 pass, six skipped**. Server typecheck and build pass. - Real pinned OpenCode/native Runner: a delayed 25-second acknowledgement completes within the original 30-second MCP deadline. The text-fragment control changes 817 notifications / 54.1 seconds / timeout to 45 / 19.0 seconds / accepted finish. Separate fast, six-minute and real authenticated API probes complete 32 tool calls plus finish. These do not reproduce every remaining historical MCP timeout, and no common cause is claimed for all such failures. Retained failures remain attributed to their original source. Earlier runs exposed fabricated empty-response success, prospective-default backfill, a literal whole-task chat criterion, fixture readiness and oracle/reporting defects. The immediately preceding [41250502608](https://github.com/paperclipai/paperclip/actions/runs/41250502608), product `c0bf249807` / source `7b1de24ce1`, is **1/3**: work 15/15; restart 4/9 with the repaired contract actually admitted but no answer; defaults timed out before settlement, then exceeded the unchanged 256 MiB original-evidence allowance. Defaults domain behavior on that attempt remains inconclusive. #18377 retains the full attempt ledger, including missing archives and pre-paid cancellations. The fresh 3/3 establishes these selected outcomes; it does not establish full-catalog reliability. ## Risks The shared contract change applies to standing native chats across providers. It changes model instructions and completion scope, while permission, approval and review gates remain enforced by their existing owners. Incorrect model claims remain possible; the product evals independently require actual mutations, native work and delivery. The correction adds at most one provider request within the existing deadline; text batching can delay adjacent text by 50 ms. Three qualified attempts do not establish full-catalog reliability. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, tool use, code execution and live eval tooling. Exact hosted model revision and context-window size are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge - [x] Final same-source three-cell pilot passes 3/3 with required evidence and cleanup --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b889ac39b7 |
fix: serialize native completion replies and bounded retries (#18502)
## Thinking Path > - Paperclip manages agents and their work. > - A delegated task reports its result to the source conversation through a durable completion outbox. > - Native execution can succeed before the presentation worker saves the accepted final reply. > - The outbox treated that short interval as a missing answer and cleared the target run. > - The presentation worker then rejected the valid answer because its delivery no longer belonged to that run. > - This change retains the existing target for one fixed presentation grace while accepted presentation is pending. > - The existing presentation owner can save the reply without another model run or more budget. ## Linked Issues or Issue Description **What happened?** A completion-delivery retry can run immediately after a native callback succeeds. It clears the callback's delivery target before the presentation worker saves the accepted answer. The reply then fails with `chat_completion_superseded`. A daily run limit correctly blocks the unnecessary replacement run, leaving the conversation active without its saved answer. **Expected behavior** The outbox must retain its current target and attempt while that exact native run has accepted presentation work. The existing presentation worker must save the answer once, acknowledge the delivery, and settle the conversation. Existing actor, session, result, digest, budget and retry checks must remain enforced. **Steps to reproduce** 1. Accept a native DONE result and current-turn final message for a completion callback. 2. Run its due completion-outbox handler after terminal projection but before reply presentation. 3. Run the existing presentation worker. Before this fix it rejects the reply because the outbox has cleared the target. **Paperclip version or commit** Reproduced on `b7f1c3769c00a107ffe4128b48b0ef8cc452cf3f` through the real PostgreSQL outbox and presentation services. The reported failure remains preserved. A new QA run is required after deployment; local tests do not qualify the deployed fix. Related: #17696 schedules accepted-response recovery. #18118 preserves canonical completion receipts. #18364 handles provider output and response-stage contracts. Those changes do not protect the completion-outbox target during this interval. ## What Changed - Defer a new execution attempt when the existing selector finds pending presentation for the same company, conversation, run and agent. - Retain the queued delivery target and attempt with an atomic compare-and-set. Await the existing scoped worker notification before that write in the same transaction. - Serialize retry with reply insertion under the existing child-task-first lock. Recheck locked task Done state and the delivery's stable completion-event version, status, target, attempt and exact lease. A delivered, retargeted or newer-claim winner prevents another wake. Admit the next attempt only after the winning transaction commits and releases its locks. - Limit presentation grace to five existing one-minute delivery leases, measured from the original run `finishedAt`. Clamp both the delivery lease and the native retry deadline to that immutable limit. Repeated sweeps cannot extend it. After expiry, resume the existing retry, exhaustion and daily-limit checks. - Keep publication authority in the existing materializer. The candidate query only defers a retry. Read the freshly reloaded delivery status, target and saved receipt after acquiring its child-task lock. - Fence both hold and retry writes with the exact claimed event version, attempt and lease. A stalled old hold cannot overwrite a newer lease. - Keep completion-event identity separate from mutable task ownership versions. Ordinary Done reassignment preserves its existing completion event; a real reopen retains normal supersession. - Add real PostgreSQL coverage for the observed ordering, outer commit and rollback, one saved reply, conversation settlement, idle SQL, retry boundaries and superseded deliveries. - Document the completion-outbox and presentation contract. ## Verification - Before the fix, the real PostgreSQL regression fails with `chat_completion_superseded` in `existingChatCompletionReply` when the outbox handles the terminal run first. After the fix, the same test passes without a new model wake. - Focused command: `vitest run --project @paperclipai/server server/src/services/native-runtime/agent-chat-response-recovery.test.ts server/src/__tests__/chat-completion-delivery.test.ts` — **95 pass** on `45f53a8c32fcca0bbdacf717286d37e30c1ba308`, using a disposable database on an owned local PostgreSQL 17 container. - Controls preserve retries for failed, cancelled, missing-final, missing-accepted-result, old-generation, foreign-company and foreign-agent targets. Reset or superseded task versions retain supersession. Digest and external-context declines never authorize a reply, never extend the fixed window, and leave it at exact expiry. An external native deadline cannot extend the limit. A valid reply at the last millisecond saves once without another model run. - The first reviewed head `b6f9c484b9567bc60af6d684c04e00058c1048ed` received Greptile 5/5 but an independent 3/5 review correctly found an unbounded hold. Three new real PostgreSQL controls failed on that source. This head adds the fixed bound and makes them pass. The earlier review, controls and CI remain retained; its review score does not qualify the new head. - The bounded predecessor `7300d2a978a5c4b3c13d85899bed3e98ae3c49fa` received an independent 4/5 review that correctly found an expiry-versus-in-flight-reply race. A real PostgreSQL advisory gate reproduced it after reply authority checked the delivery but before insertion. The retry lock and exact claim CAS now make that control pass. Both orderings, outer rollback, delivered/retargeted/superseded winners and competing lease claims pass. The next-wake test can acquire both parent and child locks, proving dispatch occurs after release. The valid 4/5 review and failing control remain retained. - At `39dd86e8f51d937cf382b2757736e0477201d21d`, both reviews found valid remaining defects: the reply helper still used its pre-lock delivery snapshot, and ordinary Done reassignment could stall a delivery by changing the task version. Four real PostgreSQL controls fail on that source and pass here: concurrent retry-first, a newly saved receipt after waiting, and reassignment within grace or at expiry. A real reopen through the issue service still supersedes the old event. - A later coordinator review found the hold CAS still lacked the attempt and lease fences already present on retry. A real PostgreSQL producer barrier reproduced an old hold shortening a newer claim from 120 seconds to 60 seconds. The same exact claim fences now prevent that overwrite. This failed control is retained; no old run was blindly retried. - The real worker saves one exact run-owned reply and acknowledges the delivery only after outer commit. An outer rollback publishes no reply. After settlement, advancing the scheduler clock by 60 seconds causes no further SQL. - Module boundaries, map structure, `git diff --check` and the canonical server build (including TypeScript and Runner vendor preparation) pass. The first local build stopped in the host package-manager setup; the installed pinned pnpm completed the build. Full repository typecheck, test and build gates pass in current-head CI; they have not all been run locally. - Final local checks also caught a test cleanup type error (TS2677) and one 94/95 readiness failure: the worker saves its comment before post-commit conversation settlement finishes. The test-only follow-up removes an invalid cleanup type predicate and waits on the existing bounded worker-quiet check before reading final state. All assertions and the same five-second bound remain. Source `45f53a8c32fcca0bbdacf717286d37e30c1ba308` passes all 95 cases and the canonical server build. The earlier failures are retained. - The final policy cleanup removes only 15 newly added narrative comment lines (seven service, eight test), as required by CONTRIBUTING.md. Both TypeScript AST signatures and `removeComments=true` emitted JavaScript are identical to tested source `45f53a8c32`. This was checked with the TypeScript 6.0.3 API and the installed native TypeScript 7.0.2 compiler. Assertions, timeouts and behavior are unchanged; the 95 tests were not repeated for identical compiled code. - The superseded `45f53a8c32` CI run recorded two 15-second timeouts in attention-retention cases. That failure remains retained. The test and its ten inspected owner or support files are identical on the tested master revisions. The canonical base run passes all 53 attention cases, and the comparison PR passes the same server lane, including those two cases in 9.3 and 9.8 seconds. This does not establish the timeout cause. Current-head CI is a new run after the required comment cleanup, not a retry of the failed source. - The optional Paperclip review bot ended neutral without a completed assessment because its execution paused. That result is not an approval. A separate [independent read-only audit](https://github.com/paperclipai/paperclip/pull/18502#issuecomment-6550216227) reports 5/5 confidence on exact head `9cfd3f74c69cc7a2da7b4ee5d1fca80f15d4af0c`, with no findings. Its manual assessment does not replace or relabel the incomplete bot result. - Current head `9cfd3f74c69cc7a2da7b4ee5d1fca80f15d4af0c` has a fresh Greptile 5/5 review with no open threads. [CI run 41249007430](https://github.com/paperclipai/paperclip/actions/runs/41249007430) passes all primary jobs and both required gates (`ci / verify` and `ci / e2e`). All 59 PR contexts are terminal, with no failed contexts. The actual CI checkout is `18db9e2750d6a799f26b5b8ba31bcccbea254227`, which combines master `681f7b77262a9df6c1b4aad55291bf960061e491` with this exact head. Fresh QA verification on the deployed application remains required before claiming the live failure is fixed. ## Risks A candidate can retain a delivery whose stricter presentation checks still prevent publication, but only within the fixed five-minute window. The existing worker keeps publication authority. If a valid reply remains unavailable at expiry, the normal callback retry may require another admitted model run; the original result and completed child work remain preserved. Existing retry, exhaustion, model-run and spending limits still apply. Failed or cancelled execution and genuinely missing accepted output retain the existing retry path. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, tool use and code execution. The runtime does not expose the exact hosted revision or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
681f7b7726 |
perf(server): retire aggregate maintenance polling (#18425)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
9ca9af544a |
Drive remaining live run and resource recovery from durable work (#18419)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
eb12cfc96b |
perf(github): activate connection work from events and deadlines (#18395)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
5e4cc01073 |
feat(workspaces): keep remote files off the webserver disk (#18403)
Keep native sandbox workspace files independent of the webserver disk. Use encrypted saved versions and Postgres ownership so a replacement server can recover files and continue work. Enable the path by default for new eligible workspaces when durable storage is configured. Preserve explicit opt-outs and existing host workspaces. Include safe snapshot repair, Files access, import, migration and retention. Bound saved-Files resources, preserve imported Git recovery, reject unfinished Git operations, publish immutable concurrent captures, and verify permanent object-version deletion. Validated 471 targeted local controls, build/typechecks, all 40 CI partitions, fresh Greptile 5/5 and security review. The PR records original failed attempts and scoped retries. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
75fa37f30f |
fix(server): expire native backups from retention deadlines (#18380)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
453f5fbb78 |
perf(external-objects): schedule scoped refresh obligations (#18384)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
c32ecf6657 |
fix(server): schedule only outstanding merge confirmations (#18382)
Reviewed and verified at
|
||
|
Nicky Leach
|
6484f96c28 | fix(workspaces): inspect storage only when needed (#18410) | ||
|
DottaandPaperclip
|
9a3f395737 |
fix(server): settle routine history without idle scans (#18399)
Reviewed and verified at
|