mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-06 23:20:53 +01:00
4e29f001396c9fc76075850165ccfa3a6743e3ef
233
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
0ab3ce7a03 |
fix(workspaces): preserve legacy recovery across storage migration (#18841)
## Thinking Path > - Paperclip manages durable tasks and agent work through the control plane. > - An ephemeral web server needs existing workspace files in shared storage. > - Failed historical runs can still own recoverable sandbox files or unfinished coordinator state. > - Moving local files must preserve those obligations without replaying the old agent turn. > - Migration journals the exact source and fences old copyback; the recovery owner saves verified sandbox files separately. > - New user input can continue only after matching preservation receipts and independent execution-stop checks. > - Accepted results retain their finalization owner and must have its durable workspace receipt before storage moves. ## Linked Issues or Issue Description Builds on merged #18826 and #18835. **What happened?** Existing workspaces with failed native copyback could not migrate to remote storage. Simply changing their storage authority would let later recovery target a retired local directory, lose child-task attachments, or leave a recovered task unable to accept a new turn. **Expected behavior** Preserve local files, exact sandbox provenance, task bindings, checkpoints, and unresolved history. Recover surviving sandbox files into immutable task-scoped attachments after migration, and let a new authorized user turn use those files without replaying the old execution. **Steps to reproduce** Create a terminal native run with a prepared copyback descriptor and a retained sandbox. Migrate its local workspace, remove the old host filesystem, recover the sandbox, and submit a new user turn. Also exercise a child task reusing its ancestor's workspace and an expired observed coordinator. ## What Changed - Journal exact legacy run, task, descriptor, and unfinished coordinator bindings during offline migration. Descriptor-free runs retain the complete declared lease/allocation/directory identities; publication locks and revalidates them. - Publish a run-scoped fence that prevents automatic restart or copyback into the retired local authority. - Release only the exact journaled expired coordinator lease under the offline deployment fence and locked revalidation, recording the prior owner in recovery history. Preserve phase, checkpoint, result, and run status. - Save recovered sandbox files through the existing immutable snapshot owner, retaining allocation and provider-stop checks. - Attach recovered files using the current task's identity when a child reuses an ancestor workspace. - Accept matching migration and snapshot receipts for workspace-repair admission and explicit new-user continuation. Changed coordinator bindings and invalid receipts remain blocked. - Require the finalizer's successful workspace_finalize operation before moving a workspace whose accepted result has finalized copyback; the existing owner can then finish assessment/status effects. - Document unresolved workspace-policy, task-ownership, and missing-provenance cases as separate owner actions that block detachment. ## Verification - 569 tests across migration, retained recovery, and explicit continuation passed on |
||
|
DottaandPaperclip
|
451d06e2fc |
fix: resume interrupted Runner preparation and confirm failure report delivery (#18853)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns run recovery and task continuation after a restart. > - A shutdown can interrupt Runner preparation before any provider work starts. > - Workspace recovery can save the files but leave that task behind a legacy execution hold. > - Retry then accepts a request without starting a successor. Startup also leaves the task blocked. > - This pull request verifies the interrupted preparation, preserves its files, and admits one bounded continuation. > - It also keeps a dropped Sentry capture eligible for a later report. ## Linked Issues or Issue Description **What happened?** A hosted task remained blocked after a graceful server shutdown interrupted Runner preparation. Its remote workspace was recovered. Retry returned an accepted request, but no successor started. Startup did not resume it. The task also had no matching Sentry event. The recovery cause is verified. The exact cause of the missing historical Sentry event is unknown. A separate regression with the real Sentry SDK proves that processing backpressure can drop a capture while the task cache marks it as reported forever. **Expected behavior** Retry must continue eligible interrupted work from its saved workspace. Startup must schedule one bounded retry when durable evidence proves that no provider started. A later task read must be able to retry an unconfirmed Sentry report. **Steps to reproduce** 1. Interrupt Runner preparation during a server shutdown before native runtime selection. 2. Retain and recover its remote workspace under the legacy recovery hold. 3. Press Retry, or restart after file recovery has completed. 4. Observe that the task remains blocked without a successor. For the reporting case, hold the first Sentry event in its asynchronous enrichment step. Fill the SDK processing buffer. Capture a second task failure, release the first event, and read the second failed task again later. **Paperclip version or commit** Reproduced against source commit `4d31588dea`. The hosted incident served `6ef37398c8d799c5142a3a6590ecc70ad352b663`. **Deployment mode** Hosted Linux server with remote workspaces and external PostgreSQL. Local regressions use the source checkout on macOS arm64, Node 26.4.0, PostgreSQL 17.10, and the audited optional Sentry SDK 10.71.0. Related public changes: #18541 added task-state failure reports. #18678 retained remote exports before Retry. #18803 fenced remote workspace writers before continuation. The small backend-port fixture correction follows #18739. It is included here because the same existing setup defect blocked CI. ## What Changed - Verify pre-native shutdown interruption against the immutable adapter claim, controller ownership, run events, native records, stopped leases, and exact workspace recovery receipts. - Retire only the server's unchanged bootstrap failure block. Admit explicit Retry through the existing continuation path. - Discover completed workspace recovery at startup and schedule one retry under the existing failure budget. Let saved input own its continuation. - Apply the existing safe replacement dispatch checks to preparation retries. A later human block, competing execution, dependency, pause, approval, or budget gate must still prevent dispatch. - Keep execution evidence available through the bounded run projection. - Confirm Sentry task-report delivery from a successful SDK HTTP response. Keep unconfirmed captures pending for 60 seconds before a later task read can retry them. Both caches remain bounded and start no timer. - Document recovery and report delivery semantics. - Align existing handoff/budget assertions with the guarded preparation retry reason. Keep the owned-runtime fixture outside reserved proxy ports, with bounded allocation attempts. ## Verification - Red before the recovery fix: all three new Retry/startup cases failed because no successor was created. Green after the fix: all three passed with real saved workspace archives and PostgreSQL. - The pre-merge revision passes all 384 tests in the six affected recovery, run-control, run-state, retry, and dispatch suites. - The late-block regression failed before its repair. Its four focused recovery/dispatch cases then passed, including final dispatch cancellation. - Review found a second stop check that still rejected verified local and lease-free preparation. Both new cases failed before that repair. All 17 interrupted-preparation regressions passed on the pre-merge revision. - The Sentry regression failed with the real SDK before the fix. All 91 affected reporting tests then passed. The transport stays in memory. No test sends events to Sentry. - Full repository typecheck passed with `node scripts/typecheck-low-memory.mjs`. The full `pnpm build` and module boundary check passed. - CI found untyped delivery-hook parameters when the optional Sentry SDK was absent. Explicit callback types fix that build. The SDK-absent server build and all 91 SDK-present reporting tests pass. - Server typecheck passes with the SDK installed and the real Sentry SDK contract CI job passes. Greptile reviewed the pre-merge commit `103e175c44` at 5/5 with no open findings. - The local full-suite attempt initially overlapped the rebase and stopped at test-line selection. Its unchanged-head rerun passed the first 309-test chat shard. I stopped the old-head run before applying review fixes. Current-head CI owns the remaining full-suite verification. - CI exposed two old retry-reason expectations. The handoff and retry-budget checks pass with the new guarded reason. A separate existing process-adoption fixture chose port 42953 in the reserved 42000–42999 proxy range. The fixture now rejects reserved ports, following #18739. Its real process/adoption/termination test passes locally. - The final Runner package job [initially failed](https://github.com/paperclipai/paperclip/actions/runs/41718507919/job/126205912033) one unchanged Codex credential case after it could not acquire its port quorum, then timed out. The complete unchanged 42-case credential suite passes locally. One unchanged CI retry passed; the original competing port owner was not identified. No Runner implementation or credential test was changed. - Required CI passes on `103e175c4490e1bdf804c64a8635b0e4568545ae`, including `ci / verify` and `ci / e2e`. That revision has Greptile 5/5 with no open findings. The optional Paperclip Review check on that earlier head remained queued, waiting for its assigned agent. - Merged current `master` without replacing published commits. The retry gate retains both shutdown interruption checks and the new workspace preparation timeout policy. All 311 tests in the five combined recovery, retry scheduling, completion, and native startup suites passed after the conflict resolution. Server typecheck passed. Final head `005d380b00d45a2393c4a2a190eab27611168d3c` includes the next conflict-free master update. All 17 interrupted-preparation regressions passed again on that final head. Greptile reviewed that exact head at 5/5 with no new findings. Required CI passes on that exact head, including `ci / verify`, `ci / e2e`, typecheck, build, the full test shards, and the real Sentry SDK isolation check. The optional Paperclip Review remains queued, waiting for its assigned agent; it is not a required branch check. - On the final head, six test jobs were cancelled after their spot runners received shutdown signals at the same time. Another server shard passed 2,419 tests, but its unchanged 103-record cleanup backlog case exceeded the 15-second limit. The unchanged full cleanup suite then passed all 29 tests locally. One unchanged retry of only the seven unsuccessful jobs passed. The [original cancellation and timeout logs](https://github.com/paperclipai/paperclip/actions/runs/41724898038/attempts/1) remain in the workflow history. No cleanup implementation or test was changed. - The signed-in hosted UI reproduced the failed Retry before the fix. The fixed path is covered through the real HTTP Retry route and assembled heartbeat/recovery services. This PR has not been deployed to the hosted incident. A live repaired Retry remains unverified until rollout. ## Risks - Automatic continuation must not repeat provider work. It requires positive preparation evidence and rejects retained provider, process-launch, coordinator, result, live-controller, unsafe lease, or mismatched workspace evidence. - Repeated recovery calls must create one successor. Tests cover concurrent requests, repeated startup visits, pending Retry, retry exhaustion, and changed task ownership. - Sentry delivery state stays local to the process. A restart or cache eviction can report a failure again. An unconfirmed event needs a later task read to retry. - No schema or API contract changes are required. ## Model Used OpenAI GPT-6 through Codex. This session does not expose a more specific deployment ID or context window size. Reasoning, code editing, test execution, and browser diagnostics assisted the change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f187de99aa |
Preserve domain deadlines across owned idle sleep (#18656)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
0a1e4c3d0f |
fix(workspaces): retry pristine remote setup timeouts (#18852)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A remote task needs a workspace before its model can start. > - Fresh workspace preparation can reach its transport deadline. > - Failure handling records that timeout as a generic setup failure and stops the task. > - An unpublished workspace has no saved task work to repeat. > - This change routes that narrow failure through the existing bounded retry scheduler. > - Saved work, retained executions, and normal admission gates keep their existing protection. ## Linked Issues or Issue Description Refs #18827, which preserves remote timeout diagnostics. This change builds on that merged repair. Related #16727 concerns controller-managed clones and optional partial clones; this change keeps remote clone contents and deadlines unchanged. **What happened?** A fresh remote workspace reached its 300-second command limit before model execution. The task then stopped with `setup_failed` and no scheduled continuation. A separate full clone completed in 137 seconds on a disposable Daytona sandbox. The original delay remains unexplained. This change repairs the missing recovery path; it does not claim to remove the underlying delay. **Expected behavior** A fresh, unpublished workspace preparation timeout can use the existing two automatic retries. Retry scheduling remains durable and company scoped. Stop, pause, dependencies, budgets, task ownership, saved work, and retained native execution must still prevent unsafe work. **Steps to reproduce** 1. Start preparation for a fresh remote workspace with no saved version or published allocation. 2. Make its initial directory/clone command return a transport timeout. 3. Observe that the base code records a generic setup failure without bounded recovery. 4. With this change, the owner validates current workspace authority and preserves a specific bootstrap failure for the normal retry scheduler. **Paperclip version or commit** Base: `e0276e0166`, including #18827. **Deployment mode** Native Paperclip Runner with sandbox workspace storage. Verification uses real PostgreSQL, filesystem commands, and a separate live Daytona transport probe. ## What Changed - Recognize only the initial preparation timeout while the same company, writer, and epoch still own a pristine workspace. - Keep restore, additional repository, provisioning, save, changed ownership, dirty workspace, and retained native execution failures outside this recovery policy. - Persist the typed failure before using the existing retry scheduler, admission gates, and attempt budget. - After successful terminal lease cleanup, reclaim an unpublished bootstrap-only reservation even when a provider returns no stop receipt. Preserve receipt requirements for saved/published work, unknown execution, and native identities; do not synthesize termination evidence. - Prevent generic recovery from granting another budget after exhaustion. Keep a specific recovery notice. - Add regressions for persistence, concurrent scheduling, restart, exhaustion, Stop, pause, dependencies, fresh allocation, lost replies, and saved output after cold restoration. - Document the recovery contract. ## Verification - Red: the real-PostgreSQL workspace regression first failed because the timeout had no recovery classification. The completion regression then failed because it became `setup_failed`. A retained-native negative failed before the admission guard. The review follow-up reproduced a failed successor reservation after production lease release returned void. - Green: all 127 transport/archive/real-PostgreSQL workspace cases pass after the review repair. All 12 receipt/void handoff cases pass against the final query. The 141 complete completion/retry/recovery cases and the fresh-allocation selection case pass. Safety cases cover concurrent scheduling across restart, the two-attempt cap, Stop/pause/dependency gates, native identities, saved/published files, active leases, failed cleanup, and cold-restored output. - `node scripts/typecheck-low-memory.mjs` and `pnpm build` pass at `ba96cff771707415e4a5e893c8b3a97499ead25b`. `git diff --check` and the module-boundary check pass. The full local test wrapper completed all 1,835 chat cases across six partitions, then was interrupted during its serial general-server phase after the complete current-head CI suite passed. Its interrupted run is not a full local-suite pass; its log is retained. The owned process tree and database children exited. - The [complete GitHub Actions run](https://github.com/paperclipai/paperclip/actions/runs/41718108375) is successful at this exact head, including all server/workspace/chat partitions, all nine serialized groups, all eight browser shards, build, typecheck, native Runner, package, Rust, and release-policy checks. The first Rust worker shut down during cache restore before tests; one retry passed without source changes. Expected Storybook skips remain skips. - Earlier failures remain accounted for: the first typecheck found two incomplete process-result fixtures, now corrected. An intermediate completion run exposed a test double that returned no promise and a teardown timeout; the complete 141-case suites subsequently passed. The first-head CI had four worker shutdowns and one worker communication loss. A three-second monitor settlement wait failed in CI and locally, then all 21 cases passed unchanged at base and head. An earlier CI retry found a real-process history adoption assertion failure; that exact case passed locally at base and head. Both server groups pass in final-head CI. These timing causes remain unknown. [Earlier CI evidence](https://github.com/paperclipai/paperclip/actions/runs/41713712522). - Live supporting evidence: unchanged production preparation code completed a full repository clone in 137 seconds using Daytona SDK 0.203.0, Node 24.22.2, Git 2.39.5, and Linux x86_64, with 1 CPU, 1 GiB memory, and 1 GiB disk. The approved diagnostic account could not access the original fleet snapshot, so the probe used public `node:24-bookworm`. Both probe allocations were deleted and absence verified. Full task acceptance at the fixed revision remains unverified: the original fleet snapshot and an explicitly configured disposable Codex subscription fixture were unavailable. No CLI login or hosted subscription was copied. - Greptile is 5/5 at the final head with no actionable findings. The Paperclip review finding is repaired, replied to, and resolved. The separate **Paperclip Review** check remains queued. Its task failed workspace export after publishing the first assessment; the staging service was then observed showing a workspace recovery page after an update failure. Subsequent browser rechecks timed out, including a fresh tab, so the normal recovery control could not be used. Reconcile that existing export after service access returns, then let the saved current-head review request run. That external review is the remaining check; this PR is not yet fully green. - The original hosted task was last verified blocked. This PR has not been deployed and the task has not been retried by this work. Reviewers can run `pnpm test:run`, `node scripts/typecheck-low-memory.mjs`, and `pnpm build`. The remote-workspace integration suite additionally needs an owned disposable PostgreSQL database named `remote_workspace_*test` through `PAPERCLIP_REMOTE_WORKSPACE_TEST_DATABASE_URL`; otherwise that suite is skipped. ## Risks Each retry keeps the existing preparation deadline, so persistent slow clones can consume the bounded budget before intervention. A retry receives fresh allocation admission; it does not replay the command against the old allocation. The workspace owner checks persisted authority before classifying the failure, and normal admission still rechecks that authority. Historical generic setup failures do not gain automatic replay. No database migration, provider-pack update, model fallback, credential change, or deployment is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser tools. The host does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1f55073f96 |
fix(cursor-cloud): support API-only controllers on managed stacks (#18310)
## Thinking Path > - Paperclip helps people manage AI agents at work. > - Cursor Cloud hosts execution behind its API. > - Product setup still expected a Paperclip Computer. > - Managed-only tenants therefore rejected valid setup. > - This admits the validated API-owned controller. > - Authority, Stop proof, budgets and cleanup remain enforced. ## Linked Issues or Issue Description Refs #18068 and #18069; both dependencies have merged. **What happened?** Managed-only setup still requested a Computer. Live tests exposed missing Cursor Stop proof and duplicate replay. **Expected behavior** Use Cursor’s API controller. Verify provider/controller termination and replay each log identity once. **Steps to reproduce** Enable native Runner/managed-only. Configure Cursor model/repository/key without a Computer, then Test, finish setup, run and Stop. ## What Changed - Derive API-only placement from validated native Cursor configuration; reject placement/command/workspace/home overrides and keep Kubernetes-only execution closed. - Skip Computers/checkout/workspace realization. Keep scoped tools, completion, private staging, budgets, leases and recovery. Persist placement; reject contradictions. - Align Cloud cards, Test, Save and Finish setup; hold unready drafts. Legacy keeps its Computer, credential and policy checks. - Verify Stop from exact API ownership, audited intent, provider-terminal proof, no Computer/process ownership and controller retirement. Recheck immediate/deferred retirement; wake existing recovery on durable evidence. - Clear warnings when the actual stopped target later has authoritative confirmation. Keep missing/other-run negatives and the existing predicate. - Publish safe status refresh after full-proof ACK. Retain bounded native/physical cancellation history; add no poller or weaker proof path. ## Verification Core merged as `3910352c923a`; Product merged as `27d35314be8e`. Public branch `5f86eb66b081` has tree `67344151d303`. Its exact-head Greptile summary is 5/5; the fresh audit found zero unresolved threads and complete pagination. The controlled workspace retry and required summary passed. All 59 current-head contexts are terminal and clean. Prior green-36 evidence keeps its identity. [Current CI](https://github.com/paperclipai/paperclip/actions/runs/41680511731) first failed workspaces-a 1/2: the CLI company-import/export beforeAll 30-second runtime-info/child-PID/HTTP readiness wait expired. All 431 UI files/4,714 tests and 285 other CLI tests passed (one skip). The cause remains UNKNOWN. Child output records startup recovery complete, but buffered CI prefixes do not establish its relation to the waiter deadline; runtime-file/PID/last-health state was not captured. Both unchanged local probes were RED. The second observed dependency preflight still pending at the deadline; observer overhead was not quantified. Neither proves the CI cause. After all functional jobs finished, only the queued metadata summary was cancelled; the other 48 records stayed unchanged. Attempt 1 retains 46 successes, the original failure, one skip and the cancelled summary. One same-head debug retry was admitted at 18:42 UTC on November 3. Its workspace job passed at 19:23 UTC on the same checkout: UI 431 files/4,714 tests and CLI 33 files/286 tests, including company import/export. Test caching was disabled. The new `ci / verify` summary passed; workflow attempt 2 completed SUCCESS. Required `ci / e2e` also passed. The final check at 19:53 UTC on November 3 found all 59 current-head contexts terminal and clean: zero pending and zero failing. The 47 inherited completed/skipped rows are not new executions. No code, assertion or timeout changed; success does not explain the original failure or erase the local REDs. CI checkout is `1ea9d8608ced`/tree `78cb1ec13d59` (parents `2e37c73c05fe`, 5f86); both workflows use 2e37. Its 20 extra paths add low-memory build/Rust task-memory. Scripts cohere; later matrix changes only labels, with no newer debug-cache steps. Static audit found no blocker. Supported API arguments omit the memory flag: no default monitor/instruction changes. Flagged external launchers can change Cursor instructions/digests and remain unqualified. The merge added 41 master-identical paths, retained the green-36 inbox hunk and found UI fixtures matching upstream. Startup/backup, recovery projection, sleep diagnostics and binary backup change production. Static audit found eight Stop/ACK owner blobs and 37 sleep-work SQL templates unchanged, with sleep fences/single-flight accounting intact. Exact precommit tree: 416/416 across eight server files, binary backup 1/1, server types PASS; repeated scopes are not summed. The deferred-retirement P1 reproduced three real-DB/owned-process failures; missing proof stayed unverified. Recovery passed 47 process cases/33 proof negatives. Warning repair binds actual targets to same-run proof. Validation passed 729 scoped UI cases, a separate 359-case thread run, 111 backend cases (84 CAS, 6 assembled, 21 activity/privacy), types/tokens. Proof validators/predicate are unchanged; no new poller or increased timeout. The d8d3 attempt stays FAILED: 45 passes, two workspaces-a failures, derived verify failure and one skip. Both shards shared three failures: two successful mocks omitted the account guard’s required userId. The two-line fixture repair changed no guard, assertion or timeout. Three failures reproduced; 54 access/auth tests then passed, including account-change/denial negatives, plus UI types/tokens. The 69285 attempt stays FAILED: 49 jobs, 46 passed, serialized shard 9 failed, its verify aggregate failed and one selection job skipped. The shard passed 575/576 cases. Its aborted-snapshot case expected nine computes but saw eight: a final HTTP 200 coalesced while seven ordinary reads had not proved the aborted compute settled. Earlier CI and the unmodified local baseline passed with a miss; no local RED is claimed. The repair awaits the snapshot, allows exactly one extra GET if coalesced, then requires a miss and retains nine computes/all HTTP429 assertions. No production code, hook, loop, sleep or timeout changed. Six inbox and seven cache/admission cases passed locally, plus server types; 30 unrelated cases were filtered. Green 36 passed 576/576 without taking the extra-GET branch; local runs also missed that schedule. Current CI event merge 1ea9 now passed 576/576 and naturally exercised the former failure: coalesced (count 2, 1ms), then exactly one fresh miss (count 1, 16ms), with strict nine-compute/HTTP429 assertions intact. No instrumentation or retry was added. Earlier compatibility passed scoped remote/inbox, routine, API-placement and UI tests, plus types/migrations/snapshots/maps/tokens. The 89-path adoption restored fleet routing without workflow overrides or added skips. Each result retains its source identity. Live qualification remains actual `9af1a5de42a2`: unchanged Artifact grader PASSED 17/17 and active-Stop grader PASSED 21/21. All three provider test resources were deleted canonically with GET404; post-delete download returned the exact 27 bytes. Histories stayed stable (Artifact 121 and Stop 67 canonical identities), with 21 company runs, zero active work and no accounting/cancellation/reservation debt or integrity findings. Saved settings reopened; no-prompt Test/Linux admission passed with `claude-sonnet-4-5`. No new Save is claimed. The first Stop sampling case stays FAILED on a 10.373-second gap. Unchanged retry kept all 24 samples: 35.076 seconds quiet, maximum gap 4.584 seconds, plus a separate CANCELLED/artifact-absent read after 180 seconds. Initial grading rejected locator spelling before functional checks; metadata changed only after same-file/inode/byte proof, with raw evidence/oracles unchanged. Original Question/Plan/reconnect and historical deleted-artifact results keep their identities. Git trees verified 11 history-only conflicts and six earlier mainline paths. Hosted 9af to the public BRANCH differs in 222 paths; 13,500 other mode/blob leaves match. Runner runtime implementations and six observer/driver pins match 9af on that branch only; inherited `drivers/acpx/runtime-host.test.ts` differs. No all-Runner/src, CI/latest-master runtime or image equality is claimed. Later static audits through `8ab89f41e499` found no concrete API-only blocker. That base adds completed-history sleep proofs and an MCP cleanup migration; protected Stop/ACK owners are unchanged. These later changes are outside this CI and have no live qualification here. Host duplicate-card and later recovery/UI repairs retain separate proof, not deployed 9af identity. Direct-key coverage requires absent aiConnection; API-only placement skips inherited remote-workspace disk-full paths, and config-read gates match their parent. Other retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; title/startup/custody/Stop/deletion unknown causes and bounded recoveries; superseded warning/50 duplicate identities; pre-repair ACK/Core CI failures. Fixture RED/GREEN passed 7/7 browser and 121/121 shutdown tests with a failing broken-unsubscribe control; stable WSP wrapper passed 128 after physical teardown failure. No full local-suite pass is inferred. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and behavior checked live; unpriced provider dollar cost stays UNKNOWN. The placement exception applies only to validated native Cursor API execution. Other local work remains denied under managed-only policy. Missing/stale Stop or cleanup proof stays pending; damaged archives remain failed. Explicit Legacy keeps its existing policy owners. ## Model Used OpenAI Codex/GPT-6 family: tools, execution, browser checks, delegated review. API ID/context window unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; retained failures are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e0c6992426 |
fix(heartbeat): prevent saved input starvation during queue reloads (#18819)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Accepted messages and task Retry requests use durable wake receipts. > - The recovery worker loads receipts in pages of 100 and checks batches of 25. > - Company activity can reload those pages before the pending batch drains. > - Reloads reset ready receipts' dispatch times and can repeatedly favor receipts already checked. > - This pull request preserves the priority of unchecked ready receipts so saved input can reach normal admission. ## Linked Issues or Issue Description **What happened?** Saved input can remain deferred after the preceding run finishes. Repeated company eligibility changes reset the dispatch times of pending receipts during paginated reconstruction. A PostgreSQL regression checks only 75 of 178 receipts on the original code despite twelve eligibility changes. **Expected behavior** Each ready receipt must reach its admission check. Existing wait deadlines, retry backoffs, authorization, and execution gates must still apply. **Steps to reproduce** 1. Seed more than one page of deferred receipts. 2. Keep those receipts pending after their checks. 3. Send company eligibility notifications between scheduler passes. 4. Check that every receipt is reconsidered, including receipts later in the backlog. **Paperclip version or commit** Reproduced on `367f29478d`. **Deployment mode** Built from source with PostgreSQL. The defect occurs before provider execution. Related: #10195 reports a similar starvation symptom in a different promotion loop. #18621 added saved-input Retry after a successful run. This change fixes fairness in the deferred-receipt worker. ## What Changed - Add an optional entry merge callback to scoped deadline reconstruction. - Use it in the deferred-receipt worker to retain unread ready receipts' priority while refreshing their state. - Place receipts whose wait has expired into the ready queue during reconstruction. - Keep real wait, claim, and dispatch deadlines authoritative. - Add a paginated starvation regression, real process dispatch coverage, and GitHub lock-collision recovery coverage. - Document the recovery fairness rule. ## Verification - The PostgreSQL starvation regression checks only 75 of 178 receipts before the fix. It checks all 178 afterward, both with ordinary receipts and with expired execution waits. - Seven focused scheduler cases pass, including moved interrupt grace, moved native error deadline, fresh claim/wait deadlines, and unchanged-gate grace. - GitHub endpoint lock-collision recovery, retained board authority, HTTP Retry, and five current admission-gate cases pass. - Real process acceptance passes on the final code: task Retry crosses the repeatedly invalidated backlog, reconstructs its persisted queued run after a worker restart, saves `receipt-proof` in run output, and has one successor for the Retry receipt. The test holds executor capacity while advancing queue time, then starts fresh workers at PostgreSQL time before releasing a real process lease. The defect is in admission scheduling before provider execution; this acceptance uses PostgreSQL and a local process adapter. - The full workspace build and typecheck (`node scripts/typecheck-low-memory.mjs`) pass locally. The affected worker suites pass 52 of 53 cases together; the existing native multi-company batch case times out in that combined run and passes in isolation without code changes. All new regression and acceptance cases pass. - Local `pnpm test:run` stops in its first chat shard on three 15-second timeouts (Teams long native HTTP picture delivery, direct final-response materialization, and contended inbound notices). All three pass in isolation without code changes. Later full-suite groups did not run locally after that shard failure. - [The full GitHub Actions matrix passes](https://github.com/paperclipai/paperclip/actions/runs/41697489374) on `ac91df2cc6db6c77529d923177b375b8c92efd45`, including all general-test, serialized-server, browser, build, and typecheck jobs. Server shard 8 initially failed an upstream process-adoption test and passed its one rerun without code changes. - [Greptile is 5/5](https://github.com/paperclipai/paperclip/pull/18819#issuecomment-6612091535) on the same current head, with no actionable findings or unresolved review threads. The separate Paperclip Review check is neutral; its previous request was bound to an older commit. Focused command: ```sh pnpm exec vitest run server/src/__tests__/deferred-execution-receipt-work.test.ts server/src/__tests__/scoped-deadline-work.test.ts ``` ## Risks - Ready receipt order changes during repeated reconstruction. Admission still reads current durable state. - A ready receipt must not inherit an obsolete priority after it acquires a future deadline. The merge applies only when both pending and loaded entries are immediate. Deadline safety tests cover that boundary. - The merge callback is optional. Other scoped workers retain their current reconstruction behavior. ## Model Used OpenAI `gpt-6.1-sol` in Codex, max reasoning effort, with tool use and local code execution. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused regressions and acceptance; full-run timing limitations documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
4a38ff9c92 |
fix(ci): make Cloud readiness reruns and fixtures reliable (#18823)
## Thinking Path > - Paperclip manages agent work and preserves task state. > - Cloud readiness verifies an exact source revision before release. > - Its workspace tests must wait for execution and database cleanup to finish. > - Two attempts hit the same ten-second polling limit and read a running row. > - The second attempt also lost AWS runners and exceeded a fixture cleanup limit. > - This pull request waits for owned execution, bounds client shutdown, and gives verification reruns an independent runner pool. > - Assertions, source identity and required verification gates stay in place. ## Linked Issues or Issue Description **What happened?** [Cloud readiness run 41684102136](https://github.com/paperclipai/paperclip/actions/runs/41684102136) failed twice on master `367f29478d14003cc5c0f161b7ce8f3164993634`. Attempt 1 failed `heartbeat-workspace-busy.test.ts:473`: expected `succeeded`, received `running`. Attempt 2 failed the same helper in the inherited-workspace resume case at line 744. The helper polls for ten seconds, then returns the active row as if the wait finished. Attempt 2 also exceeded the 30-second `workspace-preparation.test.ts` teardown limit. Its client close had no deadline, and database plus directory cleanup shared the default hook budget despite the fixture's existing 90-second startup budget. Other attempt-2 failures show runner shutdown or lost communication. Several shutdown signals arrived at 18:05:17 UTC. Read-only AWS inspection confirms that all eight infrastructure-failed instances carry `runs-on-workflow-job-interrupted=true`. Five have explicit `Server.SpotInstanceTermination` reasons, including the server-1 runner that lost communication. Three were terminated through the fleet's shutdown path. CloudTrail records the Rust runner's interruption tag at 18:03:37 UTC and its termination by the RunsOn fleet role at 18:05:31 UTC, despite EC2 showing a user-initiated termination. Repeating the entire workflow reused the same Spot fleet. **Expected behavior** Wait for owned heartbeat execution before reading final state or scheduled retries. Bound database-client shutdown and allow the fixture's cleanup cost class to finish. Rerun verification on GitHub-hosted runners after an interrupted fleet attempt. Preserve all checks and report prior failures. **Steps to reproduce** 1. Delay the test adapter beyond the helper's ten-second polling window. 2. Invoke the ordinary local shared-workspace run or resume the deferred sandbox run. 3. Observe the old helper return `running` before the execution completes. 4. Evaluate the old runner expressions with `github.run_attempt = 2`; they still choose the interrupted AWS pool. **Paperclip version or commit** Base: `367f29478d14003cc5c0f161b7ce8f3164993634`. **Deployment mode** GitHub Actions Cloud readiness, with disposable PostgreSQL fixtures. Related: #11293 fixed the earlier gap between cancellation and retry-row insertion. This change waits for the complete execution instead. Duplicate searches covered the workspace test, retention fixture and runner shutdown reports. ## What Changed - Replace both workspace-test polling helpers with the existing heartbeat completion barrier. Keep final status, retry and coordination assertions. - Bound workspace-preparation client shutdown to one second. Apply the existing embedded-PostgreSQL fixture budget to its teardown. - Use AWS only for the first exact-source attempt. Route later verification, chaos and cloud-migrator planning/build attempts to GitHub-hosted runners. The publisher keeps its existing routing. The final branch also incorporates upstream #18821's CODEOWNER routing fix. - Add runner-expression coverage for numeric, string, missing and later attempt values. Document retry routing. ## Verification - The original workflow's failed assertions and interrupted jobs remain linked above. A local two-suite run also passed all 157 assertions and failed only the original 30-second teardown hook. - All 341 workflow contracts pass. After the master synchronization, all 347 routing and CI-planner controls pass. Actionlint passes for the three changed workflows. - A matched delayed-adapter comparison uses disposable PostgreSQL 17 databases, the same 11-second adapter delay, and the same 90-second experimental outer budget. The old helper fails with `running`; the completion barrier passes. Production test deadlines stay unchanged. - Earlier experiments hit a 30-second experimental outer limit and the macOS host's shared-memory limit during native database startup. The completed comparison uses task-owned databases on an existing test server. Those earlier attempts are not counted as passes. - All 31 workspace-busy cases pass with normal test deadlines. The 126-case workspace fixture also passes with its 90-second teardown budget. - Final-head cleanup measurements: main client close 7 ms, owned database cleanup 20,938 ms, and workspace-home removal 826 ms. A measured fixture held 124,494 PostgreSQL files (208 MiB) after repeated table resets. The measured database cleanup alone consumes most of the old 30-second hook budget. The separate baseline reproduced a timeout at that hook; the fixture now uses its established 90-second cost class. - Repository typecheck and build pass on synchronized head `2aceb3abd144a569efddabed3b7e0550ea23ceb2`. - All 157 focused cases pass on the synchronized branch, including complete PostgreSQL 18 fixture shutdown. [Full hosted CI](https://github.com/paperclipai/paperclip/actions/runs/41697804471) passes on the final head, including the required verification and E2E aggregates, all server/chat/workspace groups, Runner lanes and all eight browser shards. The separate GitHub quality-review workflow also passes. All 60 reported checks are terminal: 54 pass, three are informational neutral results, and three are expected skips. There are no failures or pending checks. The complete local `pnpm test:run` command was not run; targeted fixtures already encountered this host's shared-memory limit. Hosted CI supplies the full Vitest and browser matrix. - Greptile and Paperclip Review both give synchronized head `2aceb3abd144a569efddabed3b7e0550ea23ceb2` 5/5. There are no open review threads. - Module-boundary and whitespace checks pass. ## Risks - Hosted reruns can have different cache hit rates and queue times. They execute the same source and verification commands. - The first attempt retains the existing AWS Spot fleet policy. AWS evidence establishes several interruption-triggered failures; later verification attempts use an independent pool. This does not change the fleet's external capacity policy. - The completion barrier waits for owned background execution. A hung execution still fails the test's outer deadline. - The larger teardown budget remains finite. PostgreSQL's bounded stop protocol and cleanup errors still fail the hook. - No production execution, database schema or provider contract changes. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository inspection, code edits and test execution. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
367f29478d |
fix(recovery): preserve the bound remote writer before continuation (#18803)
## Thinking Path > - Paperclip manages agent work and preserves task files. > - A remote workspace records the exact run that owns its unsaved files. > - A later setup failure can become the latest task run without owning those files. > - Retry only inspected that latest run, so the original writer had no file-recovery owner. > - A released reusable source also failed the legacy retention rule. > - This change registers the bound writer, saves its files, and then admits the saved user request once. > - Existing stop, ownership, approval and execution-evidence checks remain effective. ## Linked Issues or Issue Description **What happened?** A native run prepared a remote workspace and failed before finalization. Its sandbox stopped with a dedicated stop receipt. The workspace stayed dirty and retained that run as its writer. A later setup failure became the latest task run. Retry inspected the newer run and left the original file obligation unregistered. New attempts failed at the writer fence. Registration alone was insufficient. After file recovery, continuation still required the Retry source to be one of the older recovery sources. A newer pre-provider failure did not satisfy that check. Workspace selection could also choose a fresh directory instead of the recovered files. **Expected behavior** Find the writer through the task's current workspace binding. Register only its exact prepared source. Preserve the released lease and its stop receipts. Save the files before continuation. Independently prove that a newer setup failure never dispatched work. Select the verified saved workspace and admit the Retry once. **Steps to reproduce** 1. Fail a native run with a prepared remote workspace and a released reusable sandbox. 2. Keep matching dedicated stop receipts and the workspace's dirty writer authority. 3. Record a newer pre-provider setup failure for the same task. 4. Request Retry. Before the fix, no file-maintenance record owns the actual writer. **Paperclip version or commit** Reproduced before production edits on master `f7ad1dc7ca`. The branch was rebased on master `adb49a44346849e751548b2fb640693dbfad4b6f`. **Deployment mode** Authenticated hosted execution with Paperclip Runner and remote workspace storage. Companion: #18798 (now merged upstream) fixes a pre-existing inbox test synchronization race. Its test-only commit `110aa88489cad926a451c972ad731ede469347c1` is cherry-picked here with attribution so this PR can run the complete CI matrix. Related: #17800 registers missing preservation for the selected source run. #18678 holds unfinished remote exports. #18718 repairs a prior warm-controller projection failure. #17916 addresses a separate legacy warm-copyback policy case. ## What Changed - Discover the failed writer through the company-scoped task/workspace binding during Retry and authenticated message admission. - Reuse the exact prepared-source validator for released remote workspaces. Registration changes neither the lease policy nor its stop receipts. Physical recovery retains its full allocation checks. - Check a newer failed preparation separately before allowing an older, repaired source to release continuation. Provider events, leases, results and execution ownership remain disqualifying evidence. - Select the bound saved workspace through the continuation's verified repair receipt. - Add PostgreSQL safety cases and an HTTP Retry workflow with real file capture and saved-copy reads. Document the execution contract. - Reuse #18798's completion barrier for the existing inbox cancellation test; preserve its eight-slot limit, rejection and coalescing assertions. ## Verification - RED before production edits: the exact stopped writer has no `workspaceRestoreRecovery` after Retry. Six unsafe-source controls pass on the baseline. - GREEN: the expanded Retry/message/preview selection passes 17 cases. All 627 tests in the five affected recovery suites pass on recovery commit `b572304f772d236b8a8bdf095e13e651d808f41c`, including Stop, ownership, saved messages and duplicate Retry. - The HTTP workflow registers maintenance, keeps execution held before save, reads back the saved file bytes, and admits exactly one continuation with `resume_preserved_workspace`. A registration-only repair first failed after save because it did not reconcile the newer Retry source. The incident-shaped pair passes with an independent active stranded-task action present. - Greptile identified the path where the newer setup failure already owns a reconciliation hold. Its new regression failed before the correction (saved files, zero queued continuations); the contradictory-dispatch control passed. All four HTTP variants pass after the correction, covering both hold shapes and both dispatch-evidence controls. - Repository typecheck and build passed. Server typecheck and build also passed after the recovery changes. Module-boundary and whitespace checks pass. - The full local Vitest command failed in the GitHub fast-response acknowledgement test (`delivery_unknown`: one response write instead of zero). That isolated case passes on this branch and unchanged master; the original full-run cause remains unestablished. A sibling run passed 199 test cases but timed out in workspace-suite database cleanup. Unchanged master also passed all 126 workspace cases and timed out in the same cleanup hook; all five acknowledgement outcomes passed in that baseline comparison. These local failures remain recorded rather than being reported as suite passes. - The first hosted matrix failed the pre-existing inbox cancellation assertion (expected nine computes, saw eight). The same assertion failed in [master CI](https://github.com/paperclipai/paperclip/actions/runs/41664339150/job/126015526840) and reproduced locally before applying #18798's existing repair. All six inbox HTTP/database cases pass after that repair. Final head `7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9` adds only that attributed test change to the verified recovery code. - [Final hosted CI](https://github.com/paperclipai/paperclip/actions/runs/41677310377) passes on head `7f2fafcbace74cb1ad8ea68c8375a69b42ed30d9`, including `ci / verify`, `ci / e2e`, typecheck, build, every server/chat/workspace group, Runner checks and all eight browser shards. There are 54 successful check results, two informational neutral results and three skipped results, with no pending or failing results. - Greptile and the Paperclip reviewer both report 5/5 for that final head; no review threads remain open. The Paperclip review check is informational (`neutral`) and records its local-testing limitations. - The file workflow uses real PostgreSQL, subprocess file capture, encrypted saved objects and production recovery code. Provider resume/stop calls are simulated at the environment driver boundary. This does not prove a live Daytona or Codex recovery. No deployment or original-task retry is claimed. ## Risks - File-maintenance discovery must not authorize another executor. Existing writer, coordinator and physical allocation checks still own that decision. - A false bootstrap summary cannot prove that a newer run never executed. The new path also requires its immutable native-adapter claim, unresolved preparation state, absent coordinator/process/session/lease/result evidence, and no dispatch events. - Additional database reads run only during explicit task admission or Retry strategy selection. They are company/task scoped and add no polling loop. The saved-writer query returns one matching snapshot and omits prompt bodies. - No schema, credential, provider profile or deployment setting changes. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection, code execution and test tools. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f7ad1dc7ca |
fix(workspaces): make attached project repositories writable (#18770)
## Thinking Path > - Paperclip lets people assign work to AI agents and inspect the result. > - A task can use repositories from more than one project. > - Remote workspace preparation treated every additional project as a read-only source. > - This rejected valid Git repositories before the provider started, even when no source was read-only. > - The task page then hid the preparation reason behind a generic failed-attempt notice. > - This pull request prepares authorized repositories as writable task copies and shows redacted workspace validation errors. > - People can work across both projects and see the reason if folder preparation fails. ## Linked Issues or Issue Description **What happened?** A remote task with a selected Git-backed project and another mentioned Git-backed project failed before provider execution. It reported `remote_workspace_readonly_sources_unsupported`. Neither project had a read-only setting. The task notice showed only that the attempt failed. **Expected behavior** Each attached repository should be writable inside the task workspace. This should preserve source access checks and saved work. A workspace validation failure should explain its cause on the task page. **Steps to reproduce** 1. Configure remote workspace storage and a native sandbox environment. 2. Select a Git-backed project for a task. 3. Mention another authorized Git-backed project in the task. 4. Start the task. Preparation rejects the additional source before it starts the provider. **Paperclip version or commit** Reproduced against `dc9f387edc`, before this change. **Deployment mode** Managed remote workspace storage. Local regression checks use disposable PostgreSQL and independent filesystem processes. Related merged work: Refs #18403 and #18504. Those changes established remote storage and metadata-only references. They still reject configured cross-project repositories. No duplicate writable-repository repair was found. Refs #18774 for the test-only retention-fixture repair reused to unblock CI. ## What Changed - Add every configured Git repository of an authorized referenced project to `.paperclip-repositories/` as an editable task copy. - Retain saved paths, refs, pins and files. Recheck retained project access and source trust. Keep explicit read-only, local-source import and remote feature gates. - Keep committed snapshot repository coverage separate from later requests. Honor configured default refs, resolve remote branch names, and publish secondary checkouts only after checkout succeeds. Share configured-ref resolution between root and attached-repository scripts. - Include redacted workspace validation messages in the task interruption projection and existing failure notice. Preserve Retry permissions and render messages as plain text. Explain retained-folder access and trust denials while preserving the dedicated low-trust project-access path. - Add source, authorization, redaction, UI and save/restore regressions. Update the implementation contract and workspace maps. - Reuse the reviewed test-only repair from #18774. Batch the shared-workspace history fixture, verify its complete privacy-source map, and drain owned test bodies before cleanup. Production retention behavior is unchanged. ## Verification - Red before repair: four source-resolution cases failed with the reported read-only error. The task-error projection test also failed because it omitted the cause. - Green: 49 source and transport tests, 16 task projection and redaction tests, and 10 UI tests. The assembled preparation check passes. All 48 remote workspace lifecycle tests pass. The full preparation suite passed all 126 assertions but its cleanup hook exceeded 30 seconds. The unchanged base reproduces the same cleanup timeout with all 126 assertions passing. The focused preparation check finishes cleanly. - The real PostgreSQL/Git lifecycle checks cover both first-turn and later project attachments. They verify the configured release branch, saved edits, replacement allocations, unchanged source folders, committed archive contents, and released writers. - Review regressions were red before repair: the default ref became null, later attachment failed during restore, non-default remote branch checkout failed, and invalid secondary refs left partial checkouts. They now pass. - The local browser shows the workspace validation explanation and Retry action. The fixture uses a saved failed run; this proves the error display, not a successful live sandbox run. - `pnpm -r typecheck` and `pnpm build` pass. Map, module-boundary and token checks pass. - A local two-turn acceptance probe repeated at `17ed67eaa7` used real Codex CLI 0.165.0, `gpt-6-astra`, and `xhigh` effort. Codex wrote files in both task repositories. The services saved them, removed the first allocation and its Git sources, restored another allocation, and Codex read the saved files and wrote a follow-up in both folders. This uses the real source resolver and persistence services with a local command-runner stand-in for sandbox transport. It does not qualify Daytona, managed subscription forwarding, or the original task. The first probe waited for stdin; closing the test harness pipe fixed that setup error before provider work. - `pnpm test:run` passed the first chat shard, then failed one chat response concurrency assertion in the second shard. That case passes in isolation on this head. The same isolated chat case passes on the unchanged base. Both corresponding unchanged-base chat shards also pass (351 and 324 tests). The cause of the earlier wider-suite chat failure is not established. The preparation cleanup timeout and chat failure remain part of this record. - CI at `17ed67eaa7`: all other verification jobs passed. Browser shard 6 initially reported a request callback from the preceding test after teardown; its single unchanged-code retry passed. Server shard 10 timed out the 2,100-row retention confirmation fixture at 15 seconds on both attempts. That exact case passes locally, but the repeated CI failure required a repair. Its setup repeatedly rebuilt a shared-workspace privacy map. The logs and failures remain part of this record. - Reused the test-only repair from #18774 without changing its source file. At `ab774e9418`, all 54 retention tests and all 25 ordinary production privacy tests pass locally (79 total). Full CI [41650626903](https://github.com/paperclipai/paperclip/actions/runs/41650626903) passed on that commit, and its exact-head Greptile review was 5/5. The fixture preserves its 2,100 rows, query-bound assertions, privacy-source map, and completed-history test deadlines. - The additional review identified retained-source access/trust denials that still bypassed the workspace explanation. Real-resolver regressions reproduced both plain-403 failures before the correction. Both now yield a safe workspace validation explanation, while the dedicated low-trust denial and saved workspace remain unchanged. The optional duplicate ref-resolution block is also consolidated. - Final review corrections: 65 source, transport and error-projection tests passed, followed by all 48 lifecycle cases against an owned disposable PostgreSQL database (113 total). The initial combined command skipped the explicitly gated lifecycle suite; the subsequent dedicated invocation enabled and passed all 48. Full workspace typecheck and build pass. The first standalone lifecycle launcher lacked the repository TypeScript loader; rerunning with that loader resolved its import setup error. At previous head `7be02ea09c4ac76df560e8b517050cdf398f33f0`, full CI [41654422329, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41654422329/attempts/2) passed, Greptile is 5/5 on that exact head, and all review threads are resolved. The first attempt timed out the unchanged voice-session report/history case at 15 seconds; 3,027 cases in its shard passed. That exact case passed locally, then its single unchanged-code shard retry passed. The wider-suite timeout cause remains unproven. The completed Paperclip advisory check is neutral; its earlier findings are corrected. No staging deployment or original-task recovery is claimed. - Master advanced while checks ran, producing conflicts in two adjacent test additions. Merged `27d35314be` into this branch as `d5c8d552939c86056abf8fe10833881b2ade41a4`, preserving both workspace-error tests and the audited Stop/receipt tests. Kept workspace maps to minimal corrections under the updated map rule. All 82 source, transport and interruption cases, all 48 real-database save/restore lifecycle cases, full typecheck, build, and map checks pass locally on the combined code. Current-head [CI 41660719214](https://github.com/paperclipai/paperclip/actions/runs/41660719214) and Greptile are pending; the earlier verified head has not been merged. ## Risks - Additional remote project repositories are now editable task copies. Shared source folders and upstream push permissions do not change. - Snapshot coverage uses existing workspace metadata. Older workspaces capture their previous inventory at reservation; successful save publication advances coverage and snapshot identity in one transaction. No SQL migration is needed. - A retained copy can contain files from a project that is no longer mentioned. Preparation rechecks that project's current access and trust policy before reuse. - Explicit read-only descriptors still fail. Source-less projects remain metadata-only. Local folders still require import. No migration or provider-pack update is needed. - Error display reads only workspace validation errors. The existing run-secret registry and username censor run before display truncation. A redaction failure omits the explanation. - The retained fixture repair pauses its row trigger only within the isolated seed transaction, rebuilds the exact source map, and restores the trigger before commit. A rollback regression and 25 ordinary privacy cases pass. Its paged-case test budget honors the existing bounded inner waits; production deadlines are unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution. The exact serving model ID and context-window size are not exposed by this session. The test-only fixture changes are reused from #18774, whose model record identifies OpenAI GPT-6.1 Sol. The optional live acceptance probe uses `gpt-6-astra` with `xhigh` effort through Codex CLI 0.165.0. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
27d35314be |
feat: connect experimental Cursor Cloud to native Runner (#18069)
## Thinking Path > - Paperclip helps people manage AI agents at work. > - Native Runner owns execution and task authority. > - Operators need an explicit native Cursor Cloud choice. > - Tools/files need company, task, session and run scope. > - This connects setup and lifecycle to Runner Core. > - Legacy agents retain their selected runtime/history. ## Linked Issues or Issue Description Depends on #18068. Managed-only API placement and API-owned Stop containment are in #18310. Refs #5664, #11803, #17299, and #17471. No duplicate native integration was found. ## What Changed - Add explicit native setup, Test, saved settings and dispatch with no-prompt account/catalog/callback checks. - Scope MCP to company, agent, task, session and run through authenticated HTTPS. Register verified artifacts/work products; preserve unpriced accounting, completion and budgets. - Connect Question/Plan continuation, cold handoff and admitted fresh sessions; recheck waiting admission under the task lock. - Present audited Stop intent while preserving physical containment proof; bound typed Stop metadata. Preserve title edits through preference/navigation readiness. - Select one live/history representation per run to prevent duplicate settled cards. Integrate upstream task/read-state/retry/schema contracts and repair two test fixtures. ## Verification Core #18068 merged normally as `3910352c923a`. Product head `5a60f541c982` has passing repository CI/build/browser checks, including required `ci / verify` and `ci / e2e`, plus a fresh exact-head Greptile 5/5 review with zero unresolved threads. Two non-required external app checks remain pending: Paperclip Review is queued waiting for its assigned agent; Superagent Security Scan is reviewing the diff. These are not claimed completed. The normal merge endpoint enforces the repository rules; no policy or status override is used. The dependent Top PR is addressing its fresh deferred-Stop review finding. Product `5a60f541c982` passed 270 UI and 20 selected Cloud/native/disk-full tests, UI/server typechecks, tokens and maps. Top’s clean normal merge passed tokens/maps/diff checks; its matrix is pending. Earlier receipts retain Core’s 47 Rust, 4 compiled-native retirement and 19 settlement controls, normal Runner build, and runtime 9af build/recursive typecheck. Reproduced loading/shutdown CI fixture failures passed 7/7 browser and 121/121 unit tests, including a broken-unsubscribe negative control. No timeout/oracle was relaxed. The actual-parent restack adopts 75 mainline paths, including schema/permission and generic runtime changes, beyond the prior two fixture repairs and client projection fix. The full difference from hosted `9af1a5de42a2` is therefore larger than four files; backend source also differs. The merged config-read checks match Core391, and the Cursor placement/runtime/relay/selection modules match frozen9af; selected affected regressions pass. Paid proof keeps its actual 9af source/image identity. The separately tested live/history card fix is not claimed deployed on that host. On 9af, original saved settings survived reopening and fresh no-prompt Test authenticated catalog/model/binding; no new Save is claimed. Fresh Linux admission passed. With `claude-sonnet-4-5`, Artifact downloaded exact 27 bytes and matched 121 canonical identities in complete S3-served history. First Stop’s strict quiet FAILED on a 10.373-second gap despite independent cancellation/no late artifact. One targeted unchanged-sampler retry retained all 24 samples: 35.076 seconds quiet / maximum gap 4.584 seconds, 67 ordered canonical identities, and separate CANCELLED/artifact-absent status after the original 180-second deadline. The pre-reload duplicate settled card is disclosed. Exact-ID cleanup for the three new provider agents is awaiting authorization, so full grades and the fresh Artifact’s post-cleanup download remain blocked. A historical deleted-provider artifact was newly downloaded on 9af; original F226 Question/Plan/reconnect and cleanup proof keep their identities. Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; earlier title/startup/custody/Stop/deletion failures with unknown causes/bounded recoveries; `dea023` Stop’s warning/50 duplicate identities; pre-repair ACK and original Core CI failures. WSP’s 251-assertion, 7/8-suite attempt failed physical PostgreSQL teardown; the unchanged wrapper rerun passed 128. Retry alone establishes no cause or full local-suite pass. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and checked through behavior; unpriced provider dollar cost stays UNKNOWN. Callbacks need authenticated reachable HTTPS. Scope, revocation, budgets, completion and cleanup remain authoritative. Operator Stop intent is separate from physical containment. Saved legacy agents retain their selected runtime. Prior shared title-save/deletion failures remain disclosed. ## Model Used OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools, code execution, browser verification, and delegated code review. The serving model’s exact API ID and context window are not exposed. No value is inferred from the model catalog. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; pending gates are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e16de641ba |
fix(heartbeat): retain Stop and receipt debt after setup failure (#18776)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns execution, Stop delivery, and usage settlement. > - A restored execution can fail during preparation. > - A preparation failure does not prove that earlier provider work never started. > - Replacing its result metadata can remove an existing Stop and falsely release its accounting hold. > - This pull request preserves those obligations at the terminal write. > - The benefit is honest cleanup and receipt debt without new execution authority. ## Linked Issues or Issue Description **What happened?** A recovered native run can fail a configuration check before dispatch in the new controller. The setup failure writes bootstrap metadata with `providerWorkStarted: false`. It also replaces the previous result metadata. This can remove a pending native Stop. Accounting then accepts the new bootstrap flag and releases the original hold despite an incomplete receipt. **Expected behavior** Keep the current run-Stop and receipt debt when setup fails after earlier native execution. Preserve an acknowledgement that commits during the failure write. Fresh setup that never dispatched must remain eligible for pre-provider settlement. Keep account-identity checks effective. **Steps to reproduce** 1. Save admitted native execution, an active provider checkpoint, an original held reservation, and an audited pending run-only Stop. 2. Recover the run and fail preparation with an account configuration error before the new controller dispatches. 3. Run ordinary cost reconciliation with an incomplete receipt. 4. Observe that the base removes the Stop and releases the hold as if the run never started. Related: #18682 repairs exact-session Stop cleanup after restart. This change protects setup failure before that path. #18702 handles AI connection-request waits. #18035 proposes broader accounting policy. #18778 separately handles retained cleanup across account changes. Account, budget, and receipt authority are unchanged here. ## What Changed - Mark setup-failure status writes through an internal lifecycle option. At the atomic write, a prior coordinator attempt bound to the same run, company and task prevents false pre-provider evidence, even with a stale loaded profile. Lock the coordinator before the run so an in-flight claim cannot be missed. Selected execution input alone does not imply provider work. - Keep genuine pre-dispatch setup eligible for settlement after fresh native input is persisted, whether the loaded profile is current or stale. - Preserve only exactly bound native run-Stop fields from the current database row during failure writes, including the intentional handoff marker. Keep the latest acknowledgement when the update waits behind another writer. Normal Stop delivery still advances its own metadata. Replace unrelated stale result fields. - Add real PostgreSQL regressions for receipt debt, original holds, stale snapshots, late acknowledgement, fresh setup, dispatched work, and foreign or task-scoped metadata. - Update the execution contract. ## Verification - On untouched master, the recovered-setup regression fails six assertions: Stop removal, false bootstrap evidence, false accounting settlement, lost debt, and released hold. The fresh-setup control passes. - A second regression fails four assertions with the first repair present when the loaded profile lacks the current saved execution. The atomic current-row check fixes that case. - The focused safety selection passes. It includes real row-lock contention. A controlled concurrent writer seeds audited acknowledgement state while the terminal update waits. The product code retains that state and creates no acknowledgement or provider turn. This race test does not prove physical provider termination. - Fresh native and legacy setup can settle without a receipt. Dispatched native and legacy work keeps its debt. Foreign run, company, issue and task-scope metadata stays excluded. Unrelated old result fields are replaced. - After the final review fixes, all three completion, Stop and budget-cancellation suites pass: 121 tests. The exact `pnpm run typecheck:build-gaps` entry point passes after the final source and service-test edits. Module-boundary checks pass. An ordinary adapter pre-provider receipt control also passes. - Native review also found that fresh selected input could be mistaken for prior execution. Real-admission loaded/stale cases fail against the earlier head; a prior-attempt-only control passes. The correction uses the bound coordinator attempt and includes a foreign-task negative. Reservation `providerStartedAt` is stamped at reservation creation and is not used as provider-work evidence. - A row-lock regression also exposes a first execution claim committing during the terminal write: the prior statement snapshot misses it and falsely releases the hold. Locking the exact coordinator before the run fixes that race and preserves the existing claim lock order. Concurrent Stop acknowledgement tests remain green. - Review found two preservation gaps. New real-database reds show handoff classification loss and overwritten budget Stop delivery progress. Failure-only preservation and the bounded handoff marker address them. - Initial helper-import and reservation-order fixture errors are retained separately from causal failures. No assertion or deadline was relaxed to pass the regression. - Current-head Greptile is 5/5. The optional native review stopped before completing and returned NEUTRAL; this is not substantive approval. Its prior finding was reproduced and fixed. Coordinator and independent source/evidence reviews found no additional issue; neither is a hosted recovery test. - These are local database tests with saved provider state. No hosted recovery, physical provider termination, historical-state repair, or deployment is claimed. Full repository tests and build passed in current-head CI: 55 successful checks, three expected skips, and the explicitly incomplete optional native review returned NEUTRAL. Required verify and e2e aggregates passed. ## Risks - The current database row must retain canonical Stop state while replacing other result metadata. The preservation is limited to the same native run, company, issue and run-only Stop scope. - Setup-failure history checks keep uncertain work pending. They do not create a receipt, acknowledgement, reservation or charge. - The setup-only write option leaves ordinary adapter pre-provider receipts unchanged. No account-identity, budget, dispatch, or provider authorization gate is bypassed. - Already overwritten historical state needs separate evidence and repair. This change does not reconstruct it. - No schema, public API, provider profile, or deployment setting changes are required. ## Model Used OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source inspection, code execution, and regression testing. Context-window capacity is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3910352c92 |
feat(runner): add experimental Cursor Cloud provider (#18068)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runner gives each provider durable execution, tool, and completion contracts. > - Cursor Cloud currently has a direct adapter outside that native contract. > - Cloud work needs exact ownership across HTTP, SSE, Stop, and recovery. > - This PR adds a qualified Cloud API driver and verified proxy to Runner. > - Product setup and hosted placement remain separate dependent changes. ## Linked Issues or Issue Description Refs #5664, #11803, #17299, and #17471. Product integration is #18069. Managed-only API placement is #18310. The related direct-adapter model/transcript work is not replaced. No duplicate native Cursor Cloud integration was found. ## What Changed - Add native `cursor_cloud` / `cursor_cloud_api` with explicit model/repository validation, bounded HTTP/SSE, durable per-turn checkpoints and a qualified Node proxy launched through the Rust facade. - Commit terminal custody before publication/release; coalesce consumers, preserve ambiguous create/Stop/recovery fences, reject late tools, and distinguish definitely-unsent oversized requests from ambiguous responses. - Support scoped cold attachment, current authority, reconnect and verified artifact staging. Keep unsupported warm/external and Windows paths closed. - Preserve early source cursors and verify immutable legacy prefixes before suffix replay, including supported compaction/masking. Optional proof publication failure falls back to complete cold verification. - Preserve canonical records at the durable native producer with JSON-safe privacy and the existing 8MiB bound; drain admitted successor flushes. Reconcile upstream semantic/pack/schema/lifecycle changes and retain S3 and legacy CLI checks. ## Verification Core head `37c442ec8dd4`: all 66 check contexts are terminal successful/neutral/skipped, including required `ci / verify` and `ci / e2e`; fresh exact-head Greptile is 5/5 with zero unresolved threads. Dependent Product `81e828d20a77` and Top `dd2959689627` include the later client handoff correction and are running their own fresh gates. Core: 47/47 Rust, 4/4 compiled-native retirement, 19/19 settlement and normal Runner build passed. Runtime Top `9af1a5de42a2`: build and recursive typecheck passed. Earlier source-scoped evidence includes 398 focused tests, five relay tests and token/type gates. The 251-assertion, 7/8-suite run failed physical PostgreSQL teardown; the unchanged stable WSP suite passed 128 through the documented wrapper. The downstream fixture corrections passed 7/7 browser and 121/121 startup tests without timeout/oracle relaxation; a broken-unsubscribe control fails. Hosted runtime `9af1a5de42a2` has a passing normal build and recursive typecheck. Latest dependent heads additionally contain a pure client fix for stale live/history overlap, with 48 focused tests, UI typecheck and token gates passing. Its new client is not deployed in the frozen hosted image; server/native/API/provider source is unchanged by these four downstream files. Fresh hosted Linux admission on `9af1a5de42a2` retired its owned runner/proxy/socket with zero prompts. The saved configuration survived reopening, and Test authenticated the bound key/catalog/selected model without a prompt. Artifact with `claude-sonnet-4-5` completed, downloaded the exact 27 bytes, matched 121 canonical identities against complete logs and independently reached FINISHED. The active Stop retry was clicked within its immutable boundary, produced one canonical interrupted turn and acknowledged run-only cancellation, retained a valid 35.076s quiet window (maximum gap 4.584s), and independently remained CANCELLED/artifact-absent beyond the original 180-second command deadline. All provider-observer credential callbacks, DB clients and loaders closed; accounting is settled and explicitly unpriced. The first fresh Stop retains its failed 10.373s sampling gap and every sample. On the upgraded9af server, the historical F226 artifact was reopened and downloaded with its exact original hash after its provider agent had been deleted/GET404 earlier today. Historical F226 full Artifact/Stop cleanup, Question, corrected Plan and reconnect retain their original identities. Deleting the three newly created disposable provider agents and the new-case post-delete grade remain blocked on explicit user approval after automatic review rejected expanding the two original approved IDs. No new-case cleanup or complete fresh matrix pass is claimed; this is separate from the directly observed task/Stop behavior and retained exact cleanup proof. Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; earlier startup-title, title-save, custody/Stop and deletion failures with unknown causes and bounded recoveries. `dea023` Artifact passed 17 checks but Stop failed its false warning and 50 duplicate log identities. Pre-repair ACK failure and original Core CI failure remain recorded; retry alone establishes no cause or full local-suite pass. Provider dollar cost stays UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. The caller’s explicit values are submitted and behavior checked live. Unpriced provider dollar cost stays UNKNOWN. Only qualified per-turn controllers are admitted. Uncertain work and incomplete cleanup stay fenced. Existing damaged/duplicate archives remain failed evidence. Prefix proof adds bounded object-store I/O; the 8MiB producer bound does not guarantee future providers’ output fits. ## Model Used OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools, code execution, browser verification, and delegated code review. The serving model’s exact API ID and context window are not exposed. No value is inferred from the model catalog. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; pending gates are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b22ef22d56 |
fix(runner): show provider failure details on tasks and runs (#18622)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner starts providers and records their run outcomes. > - Provider failures can contain useful details below a generic protocol error. > - Those details were lost at the sidecar, command receipt, and run projection boundaries. > - A blocked task therefore showed a generic failure even when the provider gave a clear reason. > - This pull request carries a bounded, sanitized provider failure through those boundaries. > - Operators can read the actual cause on the task and run pages. Existing recovery policy still controls retries. ## Linked Issues or Issue Description Refs #17543 for the related closed JSON-RPC classification work. Refs #17272 for provider recovery in the legacy OpenCode adapter. This change adds display evidence for native Runner providers and keeps recovery policy separate. **What happened?** A native Claude session returned an HTTP 429 message below an ACP `Internal error`. The task and run showed only a generic startup failure. Operators had to inspect the provider response to learn that the account was rate limited. **Expected behavior** The task and run show the sanitized provider message, provider name, startup or turn phase, and available HTTP status. Unknown categories keep the message. Display evidence does not grant a retry or replace a recovery decision. **Steps to reproduce** 1. Start a native Claude run whose account is rate limited during model selection, or return a structured provider failure from a Codex or OpenCode turn. 2. Let the run fail and open its task page. 3. Before this change, the page shows a generic failure. With this change, the page shows the provider explanation. The run page shows the same explanation after reload. **Paperclip version or commit** Source checkout based on `f8a888a072`. ## What Changed - Add one JSON Schema for `ProviderFailure` and generate matching Runner and shared TypeScript readers. Keep the Runner package independent of the app workspace at runtime. - Normalize known ACPX, Codex, and OpenCode error fields. Preserve unknown messages. Redact known credential values and credential patterns before applying the 4,000-character limit. Do not copy response bodies, headers, stacks, or prompts. - Carry startup failures through the sidecar, Rust Runner, durable command receipts, and private transport error details. Keep provider text out of generic transport error strings and Sentry failure messages. - Keep turn failures in authenticated run events. Recover the same evidence from persisted events during server finalization. Store the cause in the company-scoped run result. - Show one shared notice on failed tasks and run details. Keep existing retry permissions, scheduled retry state, and resolved-task behavior. - Reapply registered run-secret and current-user redaction to task display data. Batch secret lookups for only the visible provider causes. - Document the contract and update the capability maps. Add regressions for nested Claude 429 errors, OpenAI quota and rate limits, OpenCode errors, redaction, durable receipt replay, restart finalization, and UI rendering. ## Verification - `pnpm -r typecheck` passed. `pnpm build` passed. All GitHub Actions build, test, package, and validation jobs passed on `d49c79ab4a`. The security checks passed. Greptile gave this head 5/5 with no open review threads. Paperclip Review remains pending. Its review worker exhausted six workspace recovery attempts with export_interrupted. A fresh review request was accepted and is waiting for that saved workspace. The ordinary Retry action rejected reuse of the stopped request. This is an external review-worker blocker; no check was overridden. The duplicate local `pnpm test:run` was stopped after the CI test shards passed. - Focused server settlement, provider evidence, and restart tests: 93 passed. Runner admission and normalization tests: 103 passed. Focused Runner wire and terminal tests: 29 passed. The real Rust/sidecar command receipt replay test passed. - The new projection test failed before implementation because the task had no provider cause. The startup producer regression also failed before extraction moved ahead of the generic error-code replacement. Both pass with this change. - Live check: a rebuilt local sidecar received an actual Claude HTTP 429 during startup. The new cause kept the full provider message. A synthetic task in an isolated local app then reached the same provider error through the native Runner path. The dedicated run result, task API, task banner, and run detail all showed the same message after reload. The later reporting fix keeps run.error generic; a regression proves the display cause still survives restart and stays out of Sentry capture. - The first isolated app attempt failed earlier in `admission_credential`. An ordinary manual Retry reached the provider and produced the 429 above. This PR does not claim to fix that separate admission failure. An initial parallel local database test setup also failed; the serial restart suite passed. - Security review found that copying display text into run.error could forward it to Sentry. A regression failed with that behavior. The fix keeps the cause in its display field. All 38 settlement/restart tests and the server typecheck pass. - All 29 protocol contract tests pass, including bounds and extra-field rejection. An earlier full Rust core run passed. The later full run hit one 5-second receipt-limit timing assertion; its isolated rerun passed. The full Linux and Windows Rust CI jobs passed on this head. - Review regressions pass for explicit Codex driver identity, native Rust startup and resume receipt propagation, oversized OpenCode/ACPX errors, task read-time registered-secret redaction, and the standalone source closure. The focused Runner run passed 103 tests; projection/restart tests passed 16. - Read-time redaction edge cases passed: a missing decryption key omits only the unsafe optional explanation; expanded redacted messages remain visible within the Unicode character limit. All 14 task projection tests and server typecheck pass. - The package Build check found an old Pi assertion against Error.message. The probe now checks the exact closed classification in the authenticated receipt and requires the new provider cause. Both polling transport cases pass. A local credential-free Pi probe stopped at admission_installation because this checkout lacks the Pi distribution; the installed Linux package probe passed in CI on this head. - UI coverage includes startup and turn text, unknown categories, literal rendering of provider text, read-only retry behavior, and resolved-task suppression. Token gates and map checks passed. - The live check covers Claude startup on macOS. Codex, OpenCode, terminal failures, and restart paths have deterministic regression coverage. The change is not deployed. ## Risks - Provider messages are untrusted text. The boundary selects specific fields, redacts credential patterns and known environment secret values, bounds the message, and renders plain text. New provider-specific secret formats may need added redaction patterns. - Optional fields preserve compatibility with older producers and stored runs. Old runs without the field keep their current fallback text. No database migration is required. - The cause stays in company-scoped run data. This change does not send provider text to first-party Telemetry or add it to OpenTelemetry span attributes. - Normalized categories and retryability are display evidence. Existing recovery classifiers, retry limits, ownership checks, and approval gates keep their authority. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser verification. The host does not expose the exact serving variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
40ea7a9b89 |
fix(heartbeat): wait for inherited remote workspace writers (#18691)
## Thinking Path > - Paperclip manages AI agents and their work. > - A child task can inherit its parent's remote workspace. > - The remote workspace permits one writer until that writer saves its files. > - Reservation reports an active writer as a generic setup failure. > - The existing shared workspace wait does not cover inherited isolated workspaces or a race at reservation. > - This pull request sends verified active writer contention through the existing scheduled retry path. > - The child keeps a live execution path while unfinished files remain protected. ## Linked Issues or Issue Description **What happened?** A native sandbox task inherited an isolated remote workspace from its running parent. Setup failed with `The workspace still has an unfinished writer or save operation.` The child became blocked before provider execution and had no scheduled continuation after the parent saved. **Expected behavior** The child waits for the active writer through the existing workspace retry path. It keeps its workspace binding. A stale writer or a stopped writer with unfinished saves still requires reconciliation. **Steps to reproduce** 1. Start a native sandbox task with remote workspace storage. 2. Create a child that inherits its isolated workspace or operator branch while the parent is still writing. 3. Start the child. On the old code, atomic reservation raises a generic setup failure. **Paperclip version or commit** Observed on `410b0e27015495ab8951768b5b0b94727de33785`. Regression reproduced on source based on `554a770c6f8e48e327e19f5208b575a5ec5bce30`. **Deployment mode** Authenticated managed server, external PostgreSQL, native Codex with a Daytona sandbox. The failed attempt stopped before runtime selection or provider startup. Related: #18611 permits inherited remote bindings. #16371 adds waits for reused shared execution workspaces. This repair handles active writer contention at atomic remote reservation. #17156 proposes a separate change to task workspace inheritance. ## What Changed - Return a typed conflict with the exact active writer from the remote reservation owner. Keep company, binding, access, and writer checks in that transaction. - Translate a recent running writer into the existing heartbeat workspace wait. Keep stale, unknown, stopped, dirty, and uncertain save fences. The writer task lookup uses its primary key. - Preserve the admitted assignee role across preparation so reassignment cannot turn an old owner into a guest retry. - Add PostgreSQL regressions for isolated workspaces, operator branches, native writer identity, unchanged saved authority, admission after save, durable retry, Stop, and reassignment before contention reaches the wait handler. - Update the development guide and workspace verification recipes. ## Verification - Red: both inherited isolated/operator preparation cases fail on the old code. They receive `HttpError(remote_workspace_writer_pending)` instead of a workspace wait. No provider or allocation starts. - Green: those same cases pass after the repair. The assembled public heartbeat cases also pass: one scheduled retry owns the task lock, no sandbox lease is created, the original writer and saved authority remain unchanged, and Stop prevents retry promotion. - Ownership race red/green: changing the assignee during preparation produced `wasIssueAssignee: false` before the review fix. The same regression now preserves its original role. A real transaction fault boundary also reassigns the task after reservation rejects it and before the wait handler runs; promotion cancels that retry with `issue_reassigned`. - Test isolation: an initial combined placement cached the synthetic S3 run-log backend and failed twelve later preparation cases. The assembled native tests now have their own process. The ordered reproduction and all four complete affected suites pass after that change: 171 tests, zero skipped. - Four additional saved-file lifecycle cases pass against disposable Docker PostgreSQL 17 with independent filesystem execution and local object storage. They restore inherited shared, isolated, and operator-branch files on a new allocation and retain the second-writer fence. The other 42 cases in that file were outside the selection. - `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs`, and `git diff --check` pass on the final source. - The earlier `pnpm test:run` passed all six chat shards: 1,828 tests. Its remaining general-server phase was interrupted after final-head CI passed. That local command began before the review fix and is not a complete local full-suite pass. - Final head `096235d8fc80033f92d8de1fcfa9835795ba0456`: 54 successful checks/statuses, three skipped checks, and two neutral checks. None is pending or failed. All build, typecheck, general/serialized tests, Runner checks, and eight browser E2E shards pass in [CI](https://github.com/paperclipai/paperclip/actions/runs/41544310900). The PR is mergeable with no conflicts. - Greptile is 5/5 on that exact head. Its ownership finding is resolved and no review thread remains open. Paperclip Review also reports 5/5 with no findings; its check conclusion is neutral. Its earlier status-conflict recovery completed through the ordinary review request path. - Local environment: macOS arm64, Node 26.4.0, pnpm 9.15.4, isolated native PostgreSQL and the disposable PostgreSQL 17 lifecycle fixture. The acquisition boundary is controlled in the focused preparation cases. No live Codex/Daytona replay is claimed: the local environment and test configuration lack the required test credentials. - No deployment, merge, or original task recovery is claimed. ## Risks - Incorrect contention classification could retry a stopped or unverified writer. Only a running writer with a same-company task and recent activity enters the wait path. Stale and terminal dirty cases retain the conflict. - Retries must not start another executor or release unsaved files. Reservation remains atomic. The tests check unchanged writer authority, no allocation before release, one durable retry, retained binding, Stop, and reassignment before the conflict is caught. - No API schema, database migration, provider configuration, model, credential, trust, or approval changes. ## Model Used - OpenAI `gpt-6.1-sol` through Codex. The context window size is not exposed in this session. Used reasoning, tool use, code execution, and browser diagnostics. No sub-agents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — targeted and chat results are recorded above; the full local command was interrupted - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
16643a6e03 |
fix(runner): recover pending Stop cleanup after controller restart (#18682)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server owns execution, Stop delivery, and usage settlement. > - A Stop can remain pending when the controller restarts. > - Recovery must reach the existing session without starting more paid work. > - Startup and budget fences currently block that cleanup, and interrupted runs leave the restart selection. > - This pull request gives exact audited Stop cleanup a shared authority check and records its real acknowledgement. > - The benefit is safe cleanup while usage debt remains honest. ## Linked Issues or Issue Description **What happened?** A pending native run-only Stop can retain an unfinished execution owner after controller loss. Its startup cancellation fence prevents recovery admission. If teardown marks the run interrupted and clears its task lock, restart selection also omits it. A recovered session can receive cancellation without the durable acknowledgement that the normal Stop request would save. **Expected behavior** Recover only the existing remote session for the audited pending Stop. Keep fresh provider work blocked, including during budget pause. Keep incomplete usage pending until a real complete receipt arrives. **Steps to reproduce** 1. Start a native run with a held reservation and an authenticated remote checkpoint. 2. Persist a run-only Stop and its startup and execution cancellation fences before acknowledgement. 3. Lose the controller. Leave the coordinator unfinished, mark the run interrupted, and clear its task lock. 4. Attempt restart recovery and observe the missing claim or cancellation admission rejection. Related work: #14733 binds caller Stop requests. #18658 handles plugin startup cleanup obligations. #18035 proposes broader accounting changes. This change repairs existing-session Stop recovery and preserves current accounting guards. ## What Changed - Add one shared check for the exact run, company, issue, pending Stop, intent audit, live recovery lease, generation, and remote reattachment disposition. - Include eligible interrupted Stop owners in restart selection. Keep generic interruptions and another task execution owner fenced. - Reuse only the original held reservation for cleanup. Keep budget pause effective for fresh, bootstrap, dead-runner, and unproven recovery work. - Apply the same cleanup authority at heartbeat runtime selection and native entry. Skip attachment staging and warm-owner retirement only for that exact cleanup. Revalidate at session publication and fail closed if the proof changed. - Send the existing Stop through the audited cancellation producer before turn submission. Close the coordinator only after acknowledgement, including system-owned run-only Stops. - Give verified cleanup a transport purpose that retains pending semantic inputs without replay, dispatch, or a semantic reply. Add an explicit Core retain-only intake mode; preserve authenticated identity, schema, digest, sequence and replay checks. Ordinary execution still requires its dispatcher. - Add PostgreSQL, native protocol, and startup-fence regressions. Update the execution contract and recovery maps. ## Verification - The combined Stop regression fails on unchanged production at the cancellation reservation guard. Interrupted restart selection also fails before the repair while its running sibling passes. - An assembled `heartbeatService.recoverNativeRunsAfterRestart` regression exposes the earlier runtime-selection fence with the rest of the repair present. The fixed path reaches selector/claim, runtime selection, original held-budget reuse during pause, dispatch, real native protocol, audited cancellation, and heartbeat settlement. It starts no provider turn, acquires no new sandbox, and does not reclaim the task lock. - Fail-if-called attachment staging and turn boundaries remain untouched. A retained warm-owner regression prevents retirement. Invalidation at session publication fails closed. Actual backend/driver constructor tests verify cleanup-purpose forwarding, absent default options, and denial under foreign ownership. - Missing usage retains accounting debt and the hold. A later complete receipt records one 25-cent charge and settles that same hold. Concurrent duplicate cleanup is denied. Both user-owned and system-owned Stops pass. - Real transport tests: 9 pass on Node 24.22.1. They preserve ordinary saved-tool and completion recovery, prove the authority-denial race before cancellation, and qualify cleanup with pending ordinary/completion inputs and either handler ordering. - Real authenticated Core intake: 11 pass. Callback-present and callback-absent retain-only cases persist and acknowledge unanswered input without dispatch, recovery or result. Default missing-dispatcher rejection and digest-integrity/suspension guards remain effective. Omitting the callback alone first failed by disconnecting; the explicit retain-only mode fixes that defect. - Repository `pnpm -r typecheck`, the server/runner-vendor build, map checks, and module boundary checks passed locally before integrating the latest retained-startup change. That integration passes 11 native startup/cleanup compatibility cases and 3 assembled recovery/accounting cases. The final broad affected-suite attempt was stopped without a verdict; it is incomplete. Full repository tests and build are delegated to current-head GitHub CI. - Review exposed an additional real-process close failure with unanswered input. The corrected path requires a fresh exact provider-stop receipt confirming exit, followed by durable event drain and exact runner suspension. All 11 focused transport cases pass, including missing/false exit-proof negatives and ordinary recovery siblings. Runner TypeScript and map validation pass after this correction. - The assembled test controls provider, workspace-transfer and GitHub bootstrap I/O. It does not qualify hosted credentials, physical provider termination, workspace preservation, or historical receipt recovery. Local socket-permission setup failures are separate from semantic regression results. No hosted replay has run on this repair. - CI exposed an ordinary factory-call compatibility regression and an obsolete authority-read test fixture. The repair preserves single-argument ordinary calls and updates the race fixture without changing its Stop assertions. The complete native executor suite now passes all 1,049 tests; the ordinary factory boundary and all five Stop-accounting cases pass. A separate local environment-default case twice exceeded its five-second wait while other work ran; its prior-head CI case passed. The same unchanged case passed an isolated diagnostic without changing its assertion or timeout; the precise contention source remains unproven. - The final fixture edit initially used the wrong overloaded promise type. CI rejected it; the annotation-only correction passes the exact `pnpm run typecheck:build-gaps` entry point locally, including server service tests and remaining workspaces. No runtime assertion changed. - Final exact head `007fd4d04a2d6338c6dc073e97ab470a277fc3d0`: required `ci / verify` and `ci / e2e`, build, typecheck and test groups pass. Greptile is 5/5 with both findings resolved. Three expected checks are skipped; the optional native check is neutral, while its final current-head comment gives 5/5 from source inspection without a checkout or test execution. The first server-shard attempt lost its self-hosted runner connection; one scoped rerun passed. Its original annotation and unavailable-log evidence are retained. ## Risks - This changes execution authority. Cleanup requires an audited, exactly bound pending Stop, an unfinished coordinator, a live remote reattachment claim, and the original held reservation. It never authorizes a replacement provider turn. - Missing checkpoint, original held reservation, termination, or usage evidence can keep cleanup or accounting pending. The repair does not manufacture acknowledgement, replies, receipts, or zero cost. - Retain-only intake is an internal trusted constructor option. It changes no agent tool permission. Ordinary callback requirements and semantic dispatch remain unchanged. - No schema, public API, provider profile, or deployment setting changes are required. ## Model Used OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source inspection, code execution, and regression testing. The context-window capacity is not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run affected tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open findings - [x] I will address all reviewer comments before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
a9b18e1cbe |
fix(runner): preserve cleanup ownership across startup failures (#18658)
## Thinking Path > - Paperclip controls which run owns an agent task. > - A plugin can continue accepted work after Core stops waiting. > - A late task-creation reply can therefore arrive after startup cancellation. > - Core must keep that work owned until its resource identity and cleanup outcome are known. > - This PR supplies the cleanup and recovery support before shared startup is enabled. > - The dependent activation PR can then reuse transport data within a bounded startup context. ## Linked Issues or Issue Description Refs #18405. This is the independently mergeable prerequisite for that stacked activation PR. **What existing behavior does this improve?** Cleanup of explicitly recorded startup work that outlives its caller. **Current behavior** Caller timeout, worker exit, or failed cleanup-intent persistence can leave work without a complete recovery path. **Proposed behavior** Retain the cleanup obligation, use existing Stop and recovery queues, and release ownership only after the exact task and accepted work have stopped. Ordinary calls keep their existing startup payloads and budgets. This PR does not create startup contexts or advertise transport reuse. **Reason and benefit** Separate the failure-handling contract from activation so each can be reviewed on its own. This support PR has **no standalone TTFT speedup**. ## What Changed - Add the startup cleanup marker, exact ownership checks, verified completion, and acknowledgement helpers. - Connect recorded cleanup obligations to existing Stop, deadline reconciliation, restart recovery, and retry scheduling. - Retain typed cleanup responsibility when a database write fails. Keep the nonterminal run from being treated as ordinary completed cleanup. - Observe late acquisition results and save resource identities before retiring them. Retain real host writes and received results across worker exit or persistence failure. - Queue abandoned replacement tasks through the existing creation journal and cleanup machinery. Require the replacement's own Stop proof. - Accept an explicit startup context in the internal bridge for lifetime tracking. Automatic context creation, marker production, capability advertisement, transport capture, and executor activation belong to #18405. - Add PostgreSQL and worker IPC regressions, including legacy payload assertions. ## Verification Initial split source `8fb482d8be9e70101b0cfdd0830f41cf5b8e3cf4` had tree `95508bd1859e991670cad8afd4de8d2167009ac5`. **190 distinct targeted tests pass**, along with server typechecking. The selected checks cover real PostgreSQL cleanup/ownership, worker IPC, the bridge and creation journal, ordinary/distributed Stop, restart and deadline recovery, idle-sleep behavior, and the unchanged legacy warm-task suite. Independent recovery and worker reviews found no blocking issue. The first correction, `b6342d42042601b376da38dc8e1dba9ca80e9539`, fixes the replacement cleanup finding with the existing startup-cleanup marker, scoped to contextual abandoned task creations. The replacement was never prepared, so preparation-stop proof is not required. No dispatch is fabricated and no normal-startup query is added. The actual queue/sweep/driver/receipt/cancellation regression fails before the fix and passes after it: void, running or foreign-task status retains ownership; exact-task stopped status permits completion even when Release returns void. **64 affected targeted cases and server typechecking pass**. Independent review found no blocker. Current source is `f53af72af76f263096c3c2e788506e6faef9474f`. It fixes both recovery findings from the prior head's Apex 3/5: orphan recovery now preserves pending or completed-but-not-cancelled cleanup through its existing final-write fence, and exact late identities can reach verified cleanup after reconciliation or restart. The regression selection went from four failing assertions to six passing cases. Affected selections cover 217 distinct cases; the exact final six-case regression and server typecheck pass. Independent reviewers found no blocker in either fix. Current-head standard review is 5/5 and all CI checks pass (54 successes, three expected skips, plus Snyk). Apex also completed at **5/5** on this exact head, with zero unresolved threads. The requested independent availability audit is complete on this head. It found an activation-dependent operability gap: an acquisition response that is never received can leave an unknown creation identity with no supported task inspection/reconciliation path, while cleanup keeps the run owner reserved. This does not occur in ordinary lower-only calls. The contract and recovery policy need follow-up before activation; the external review scores do not resolve that availability tradeoff. The split preserves the accepted implementation at `21db7b3851` after its clean merge with frozen master `946735d03a55c1fb4cc5434d6bf1d87742daa9df`. The initial dependent activation PR restored exact aggregate tree `f68c6895f87c7b93ae03e6324d955d2659a99dce`. Both subsequent cleanup corrections are integrated into upper head `27978f294a02da3992585c7f4daa73cd8fcf99f8`; its current aggregate tree is `a0b66c33f5e22745f034f45d24d213f5c210cf16`. Lower-only payload tests cover the absence of capability advertisement and transport projection. Only targeted local tests and typechecking are used. Full regression coverage runs in GitHub CI. No deployment or ROM qualification is part of this change. The lower audit covered connected worker, creation-journal, Stop, restart, deadline and orphan-recovery paths. A new isolated real-PostgreSQL characterization reproduced the unknown-ID hold through actual recovery, cleanup verification and run-control cancellation. It models a lost acquisition response; it is not a live provider or complete IPC test. No source edits were made by the audit. The upper/composition audit remains required after its review cycle. ## Risks - Recording a replacement result now commits with the existing cleanup notification. This conditional replacement path adds transaction/notification responses; common retained startup is unchanged. - This adds cleanup and recovery state handling. A missing receipt or permanently inaccessible old backend can keep ownership held indefinitely. Temporary outages retry; an ordinary retry is not evidence that a deleted or replaced backend stopped. - A caller timeout does not undo work already accepted by a plugin. Unknown resource creation remains unresolved until its identity/outcome is established. Current task creation has no supported inspect-by-operation or operator reconciliation API for a permanently lost identity; repeated Stop alone cannot resolve it. - The support code understands records produced by the dependent activation PR. It does not enable the shared startup policy by itself. - Existing legacy callers must retain their behavior; context-only automatic replacement cleanup remains explicitly scoped. ## Model Used OpenAI Codex, GPT-6 family. The exact model ID and context window are not exposed in this session. Tool-assisted implementation, targeted testing, and independent subagent review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, with available version and capability details - [x] I have checked ROADMAP.md and confirmed this is the requested split of existing work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue with the required fields - [x] I have not referenced internal issues or private links - [x] My branch name describes the change and contains no internal ticket ID - [x] Targeted local tests pass on this source - [x] I have added or updated relevant tests - [x] I have updated relevant documentation - [x] I have documented the risks - [x] All current-head GitHub CI gates are green - [x] Current-head standard Greptile and Apex are 5/5 with no unresolved findings - [x] I will address all review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b4315943e8 |
fix(heartbeat): keep remote retry workspace strategy stable (#18660)
## Thinking Path > - Paperclip preserves agent work across separate execution services. > - A remote retry can create a new isolated Git workspace. > - Later admissions compare its saved configuration with the current policy. > - Fresh preparation also creates a temporary controller-side strategy. > - Remote preparation saved that temporary strategy in its fingerprint. > - The next unchanged turn then reported a workspace strategy replacement. > - This change saves the admitted remote strategy so unchanged reuse stays stable. ## Linked Issues or Issue Description Related preparation work: #18543 and #17924. I searched open and merged workspace/session repairs and public workspace-strategy issues. None covers this fresh remote retry case. **What happened?** After Retry created an isolated remote Git workspace, each unchanged follow-up reported `workspace_config_freshness` with `action: replace` and `changedCategories: [strategy]`. It retained the same workspace. The stored fingerprint contained `project_primary`, which was only a temporary controller provisioning choice. **Expected behavior** Save the configured remote strategy. An unchanged follow-up must reuse the same workspace without a false replacement report. A real strategy, base-ref, mode or environment change must retain its replacement checks. **Steps to reproduce** 1. Prepare a fresh isolated remote Git retry with a `git_worktree` strategy. 2. Complete its runtime handoff. 3. Admit another turn with the same workspace policy. 4. Inspect the recorded workspace freshness decision. **Paperclip version or commit** Reproduced on `b9e94b85a69bf679818eeef9ecf4655554a89c6b`. The regression also fails on the unchanged preparation code in the current base. **Deployment mode** Authenticated server with remote workspace authority and a sandbox execution target. ## What Changed - Keep the admitted strategy when a fresh workspace uses remote authority. - Retain the temporary controller strategy for existing host-side seed preparation. - Extend the PostgreSQL preparation test through a fresh admission and its next ordinary admission. Assert the same workspace, current writer and stable fingerprint, with no false drift operation. - Cover a genuinely configured `project_primary` fresh remote Git retry followed by a real strategy change, preserving its replacement decision and stored fingerprint. - Document the execution rule and the separate live verification recipe. ## Verification - The final regression was run before and after the production change. Before: the next admission recorded a strategy replacement. After: unchanged reuse records no drift and retains the workspace and fingerprint. - All 312 targeted cases pass on `48b3e548afa42e740871feae2763d88fd5a00431` across workspace preparation, workspace/session policy and inherited session preparation. These include actual strategy/base-ref changes and the rule that explicit reuse must not hide real replacement-class drift. - All eight remote preparation cases also pass with the compatibility regression: a genuinely configured `project_primary` fresh Git retry creates a `git_worktree` row, and its later real strategy change retains the replacement decision and saved fingerprint. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Full recursive typecheck and `pnpm build` pass locally on this head. - The [full current-head CI matrix](https://github.com/paperclipai/paperclip/actions/runs/41524105868) passes: typecheck/build, all general server, chat, workspace and serialized test groups, plus Runner and E2E checks. The workflow and `run-vitest-stable.mjs` partition the complete suite. - The duplicate local `pnpm test:run` passed all 1,828 chat cases and was intentionally stopped during general-server after the same complete CI matrix passed. This is not recorded as a full local-suite pass. - Greptile and Paperclip Review both assessed this head at 5/5 with no new findings. Paperclip Review accepted the compatibility boundary after the counterexample; its check conclusion is neutral. The addressed review thread is resolved. - The regression uses real PostgreSQL and preparation policy. Runtime acquisition is controlled at its external boundary. The new fix has not been deployed; live provider resumption remains unverified for this patch. ## Risks - Existing incorrect fingerprints are not rewritten. The same stored shape can come from a genuinely configured `project_primary` remote Git retry; a Git workspace row alone cannot prove the old admission policy. Rewriting it would hide a real policy change. After preserving and inspecting retained work, an explicitly admitted fresh recovery after rollout establishes a correct new fingerprint. Ordinary reuse of the old workspace does not migrate it. Normal recovery gates remain required. - Real policy changes still use the current replacement guard. No schema, credential or authority change is included. ## Model Used OpenAI GPT-6 Codex with reasoning, repository editing, shell execution, PostgreSQL tests and browser inspection. The host does not expose the exact model variant or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b4a66027b7 |
fix(server): deliver chat completions after nested workspace saves (#18665)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent Chat returns completed worker tasks to the original conversation. > - Delivery must retain its wake receipt when execution gates refuse admission. > - A workspace save can contain a nested snapshot operation. > - The old checks lost the receipt and treated that saved snapshot as unfinished work. > - This pull request preserves the receipt and recognizes a successful owning save. > - The benefit is that completed work can reach the chat through the normal queue. ## Linked Issues or Issue Description Refs #18652. Related: #17206 covers failed finalize recovery. This repair handles a successful finalize with an enclosed snapshot and does not change failed-save recovery policy. **What happened?** A delegated worker finished and saved its result. The original chat received no update. The dependency gate compared operation start times. A nested snapshot started after its enclosing finalize and therefore left a permanent hold. The completion wake also lost its exact idempotency receipt when the queue merged a refused wake into a generic dependency wait. The outbox reported `Completion wake returned no run or durable receipt`. **Expected behavior** A successful finalize covers a completed operation enclosed by that save when company, workspace, issue, and run match. Completion wakes that disable coalescing keep their own durable receipt even when admission waits. Dependency, budget, Stop, and pause gates still apply. **Steps to reproduce** 1. Hand a task from a chat to a worker and make the chat wait on that task. 2. Finish a workspace finalize that starts before its nested snapshot and ends after the successful snapshot. 3. Mark the worker Done and deliver its completion outbox entry. 4. On the parent revision, the dependency stays blocked and the completion receipt is absent. **Paperclip version or commit** Reproduced against `53f480d1b3` with real disposable Postgres rows and the normal queue/outbox services. ## What Changed - Honor `allowRunCoalescing: false` when recording execution waits. - Accept an enclosed successful operation only when the successful finalize has the same company, workspace, issue, and run and covers its whole recorded interval. - Preserve the existing rule that a strictly later successful workspace finalize clears earlier work. - Add failure controls for foreign runs, unfinished snapshots, failed saves, snapshots that finish too late, and later operations. - Exercise the real outbox and queue together. Hold provider dispatch at the existing drain boundary and verify server-recorded Done facts reach the admitted turn. - Document receipt identity and workspace-save coverage. ## Verification - Before repair: the two causal regressions fail and five safety controls pass. The missing exact receipt and empty wakeable-dependent list match the observed failure. - After repair: those seven cases pass without assertion changes. - The full affected set passed 288 tests and failed one existing scheduling test. Its unchanged Node24 baseline also failed. A closed-phase probe showed the fake adapter finished successfully at 3.453 seconds, after the incidental 3-second poll. The test now joins the existing execution promise and retains its status and dependency assertions under the unchanged Vitest timeout. The synchronized full dependency suite passes all 13 cases. - `pnpm -r typecheck` and `pnpm build` pass on pinned Node24/pnpm9. `pnpm test:run` is running. The combined real outbox/queue case passes. Fresh Greptile is 5/5 on `d7cee24b7032c7fdf171bfd1246cd90d41db1cba`. Exact-head normal CI and required `ci / verify` and `ci / e2e` pass. - The original scheduling failures on Node26 and Node24, the unchanged-source baseline failure, and the closed-phase probe are retained in the validation record. No model, deadline, production assertion, or paid attempt was retried to change the failure grade. - Current-head CI attempt1 passed all product and workspace coverage but failed the unchanged serialized retry-worker case at `issue-scheduled-retry-routes.test.ts:606` (first batch not observed before the poll deadline;410 other shard tests passed). The exact unchanged case passes locally on Node24 with real Postgres. The one failed-jobs-only retry passed on the same head; the original grade is retained. - The full local command completed all1828 chat integration tests. Its general stage remains running and has retained failures in unchanged `workspace-runtime.test.ts` (30-second local port-conflict fixture) and `muse-control.test.ts` (current binding authentication fixture). These are not counted as passing local full-suite evidence. Their current-head hosted coverage passed. - The separate advisory Paperclip Review app is paused and has no code finding. Its in-progress state is preserved and is not counted as a pass; it is not a required repository status check. - The live Cloud completion journey remains failed and unqualified. It must pass at the deployed fixed revision before daily-plan admission. No QA oracle, model, budget, or run limit changes are included. ## Risks - The workspace coverage query must not clear a save from another owner or new work after finalization. The guard checks the full interval and exact owner identity, with negative controls. - Refused noncoalescible automatic wakes retain separate receipts. Replaceable automatic signals still merge. No new polling or provider retry path is added. - No migration or API change. Deployment and live acceptance are separate remaining gates. ## Model Used - OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The exact deployment variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a92f616cd0 |
fix(db): attribute bounded captures to execution owners (#18634)
## Thinking Path > - Paperclip manages agent work through shared server and database services. > - High query volume can spend main-thread time on SQL construction and result decoding. > - The existing bounded response capture counts rows and bytes, but it cannot name the operation that executed a query. > - Bundled callers can be missing, and measuring every parsed JSON value adds work to the diagnostic. > - This pull request carries a semantic execution owner into each measured query and bounds response sizing. > - Operators can rank query volume by owner before choosing a performance fix. ## Linked Issues or Issue Description Refs #17504, #17522, #17537, #17564. **What existing behavior does this improve?** The opt-in `PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW` diagnostic. **Subsystem affected** Database diagnostics and existing server execution boundaries. **Current behavior** The capture groups by table, operation, and builder caller. It cannot distinguish execution owners that reuse a prepared query. Bundled deployments need source maps for caller locations. The capture visits every returned value and serializes parsed JSON again to estimate its size. **Proposed behavior** Version 2 captures use fixed semantic owners for HTTP requests, delivery queues, execution phases, and budget work. They count completed calls, returned rows, errors, and completion wall time. They sample successful results 1, 17, 33, and so on, with fixed cell and string/byte limits. Complex values remain explicitly unmeasured. Semantic ownership works without source maps. **Reason and benefit** A short census can identify which operations issue the most queries or return the most rows. It preserves each query's execution owner through pooled connections, concurrent operations, and prepared-query reuse. It reduces diagnostic work without claiming per-query CPU attribution. **Breaking changes** Opt-in capture output moves from `entry.v1`/`summary.v1` to `entry.v2`/`summary.v2`. Sampled primitive byte lower bounds replace all-result byte estimates. Consumers must use the new fields and must not extrapolate them into a total. There is no database migration or application API change. ## What Changed - Added `withDatabaseOperation(owner, action)` to the DB package. The server owns semantic labels. Queries retain execution ownership in their completion closure. - Added fixed owners at existing HTTP, delivery, budget, and execution timing boundaries. Delivery work replaces the context of the producer that woke it. - Added complete call/row/error counters and completion wall durations within the existing bounded window. Durations include waits and are not CPU measurements. - Sampled and bounded primitive-value sizing. Complex and oversized values remain unmeasured. Known owners skip repeated stack extraction. - Preserved the deadline, group limit, paced output, query outcomes, and existing coverage exclusions. Updated the operator documentation. - Added real PostgreSQL owner-isolation and prepared-query tests, server scope tests, and plain bundled-runtime proof. - Moved projection-test and benchmark full-response sizing into an explicit test helper. Existing payload-size thresholds remain unchanged; these synthetic checks do not use the production sampler. ## Verification - Final-commit DB capture suites: 43 tests passed in 13.86 seconds against the actual PostgreSQL driver, including concurrent owners, prepared reuse, transaction failure, a dedicated client, sampling, and a query crossing the deadline. DB typecheck passed. - Server attribution and existing delivery/execution/budget suites: 53 tests passed. Final concurrent HTTP GET/JSON POST scope refinement: 5 tests passed. - Runtime bundle suite: 8 existing tests passed. The new no-source-map owner test passed after its fixture was corrected to use a driver thenable. An ordinary native Promise can bypass an overridden `.then`; the real PostgreSQL tests separately cover driver integration. - Observer-only experiment: three alternating pairs per case on Node 24.22.2, macOS arm64, 3,000 calls per run after warm-up. For an already-materialized synthetic JSON result, median capture CPU fell from 82.37 to 1.87 microseconds/call. For a 100-row primitive result, it fell from 8.10 to 4.40 microseconds/call. Every result retained identity and all calls were counted. The new diagnostic intentionally measures less byte data; these are observer costs, not database latency or a hosted application speedup. Disabled-case timings were below one microsecond/call and too small for a performance conclusion. - Final lifecycle regression: 28 unit tests and DB typecheck passed. Closing the last capture disables and replaces owner storage; overlapping captures retain it. - Final-commit full recursive typecheck passed in 38.38 seconds and full build passed in 40.81 seconds. The build stamp matches `83ed4ba9cebd68d48929c46f15ab32d5c0b9687b`. Focused server attribution tests also passed 5/5 before the test-helper-only migration. Full test acceptance will use the repository's parallel CI gates; the unsharded local command takes hours. - Verification failures retained: initial sandbox runs could not open local IPC/PostgreSQL listeners; permitted reruns exercised the real services. One old test expected JSON serialization and was updated to assert explicit unmeasured complex values. The first bundle fixture used the wrong promise boundary, as described above. A retained-callback regression also demonstrated that disabling and reusing the same AsyncLocalStorage instance could resurrect an old owner; replacing the instance after the last capture closes fixes that case. - The first CI run found older projection tests consuming the previous full-response fields. Those consumers now use the full-response test helper, preserving their byte thresholds. The two complete affected route suites plus helper tests passed 115/115 in 73.08 seconds. A temporary full-row query negative control then failed the unchanged 2,048-byte limit at 5,000,812 bytes and was restored; five other service suites passed 34/34 and the affected chat cases passed 3/3. All three previously failing CI shards passed on the corrected head. A local skill-list benchmark smoke check completed 255 captured queries with no in-flight queries or overflow; this was integration verification under concurrent test load, not a timing benchmark. - Two final-head runner jobs lost communication with GitHub before reporting a test result. One targeted retry of those jobs and their dependent aggregate passed. All 53 final-head checks passed, with 3 intentional skips; previously passing suites were preserved. - Hosted capture remains pending until the reviewed revision is deployed and a bounded window is explicitly configured. Reproduce the focused checks with Node 24 and the repository pnpm version: ```sh pnpm exec vitest run packages/db/src/response-capture.test.ts packages/db/src/response-capture-postgres.test.ts pnpm --dir server exec vitest run --config vitest.config.ts src/__tests__/database-operation-attribution.test.ts scripts/bundle-runtime.test.mjs pnpm -r typecheck pnpm build ``` ## Risks - Owner context is active while a valid future capture window is armed, up to 30 minutes before accounting starts. No-window and expired calls take the direct path. Diagnostics still add bounded work while enabled. - A nearest nested owner is reported. Broad `http.request` and unattributed groups remain visible; this does not identify every service callsite or SQL shape. - The existing census excludes tagged/raw explicit-execute/streaming paths and internal protocol commands. Its counts are not PostgreSQL transaction totals. - Completion wall time includes queue/database/network/scheduling waits. Concurrent durations overlap. Sampled bytes exclude complex values, large strings, and values beyond the sizing limits. - Activation is startup-only. Operators must choose the window after artifact preparation, allow rollout/warm-up time, and remove the expired setting afterward. ## Model Used OpenAI Codex, GPT-6 family, with code execution, repository inspection, and parallel agents. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0710c56842 |
fix: retry saved chat input after a successful run (#18621)
## Thinking Path > - Paperclip manages AI agents and the work they receive. > - Connected conversations save incoming requests before they can run. > - A request can remain queued when the previous turn finishes successfully. > - Task Retry sent that case through the failed-run retry guard, which rejected the successful run. > - This change resumes the saved request through its existing admission owner. > - The operator can continue saved work while preserving the prior result and current permissions. ## Linked Issues or Issue Description **What happened?** Task Retry returned a conflict for a restricted connected conversation when its latest run succeeded and newer input remained saved. A GitHub review can reach this state when a new PR head arrives during the previous review. The prior review cannot publish against the new head, but its completed turn is not a failed run that can be replayed. **Expected behavior** Retry should resume the saved request with its original source identity and current authorization. It must preserve the previous successful result and must not replay that turn. **Steps to reproduce** 1. Start a review through a restricted GitHub endpoint. 2. Send a PR synchronize event while that review runs and retain its deferred input receipt. 3. Let the original run finish successfully with the task still open. 4. Click Retry. Before this change, the failed-run guard rejects the request. With this change, the saved input creates one successor through normal admission. Related work: #17725, #16965, #15033. ## What Changed - Select the newest saved input by its canonical chat action and wake receipt. - Store its receipt ID in the task retry intent and resume it through the existing chat owner. - Preserve caller permissions, source authorization, task state, agent state, cleanup waits, and duplicate-click handling. - Exclude saved-input intents from generic board-message replay. Keep board messages in separate queued receipts and allow independently saved input after successful board turns. - Add signed webhook, HTTP endpoint, accepted-result preservation, and refusal tests. Update execution semantics and capability maps. Stabilize the existing Teams lifecycle test with explicit database result ordering. ## Verification - Current-head [CI passes](https://github.com/paperclipai/paperclip/actions/runs/41482727909): 54 successful checks, one neutral review, three expected skips, and a successful Snyk status. Greptile gives commit `53f02e9105` 5/5 with no unresolved review threads. - Reproduced the original HTTP 409 before the fix. Reproduced both review findings with three failing cases before fixing them. - Passed 25 focused and related GitHub retry cases, 52 ordinary queue tests, and three native startup cases against disposable PostgreSQL. Earlier related verification also passed 83 GitHub retry cases and 109 recovery/receipt cases. - Passed `pnpm -r typecheck`, `pnpm build`, `node scripts/check-maps.mjs`, and the module boundary check. The final server typecheck passed after the review fixes. - The full local `pnpm test:run` attempt stopped on an existing Slack queue deadline assertion with a 1 ms database/host clock difference. That case passed in isolation. A second attempt was stopped when review changes superseded its source. Full local completion is not claimed; all chat, server, browser, build, and runner CI jobs passed on the final commit. - The runner CI job needed one rerun after a two-second startup timing assertion failed. That case and its two related cases passed locally; the rerun passed without code changes. - Live GitHub review publication with this patch is not verified. The signed provider fixture uses mocked GitHub responses. This branch is not deployed. ## Risks - This changes how Retry dispatches deferred external input after a successful run. It relies on the existing inbound owner to recheck source authority before execution. - Tests cover paused connections and agents, changed source bindings, completed tasks, pending cleanup, duplicate clicks, and preservation of the accepted result. - No schema migration or public API shape change is required. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code execution, and browser tools. The exact deployed model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c0550e7010 |
fix(server): preserve sessions and defer busy overlap recovery (#18552)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A web server replacement must preserve remote execution and later task continuation. > - The active-run handoff now survives deletion of the departing server home. > - The next admission still compared disposable instruction and skill paths as configuration. > - A native receipt could also settle without the old adapter returning its terminal metadata. > - A replacement can also collide with a run row locked by the serving controller. > - This change preserves durable session identity and terminal evidence, and defers locked recovery work so overlap can reach readiness. ## Linked Issues or Issue Description Refs #18543, #18366, #9977, #18118. **What happened?** A real Codex/Daytona journey replaced both control-plane servers and deleted their homes. Reads, saves, saved files and active execution passed. A later useful answer used a new provider session. All four run records lacked the native terminal contract. The whole qualification therefore failed. A later attempt also exposed startup failure: the replacement waited for a row lock held by the serving controller and aborted on its one-second lock timeout. **Expected behavior** An unchanged managed instruction entry and immutable skill version retain the session across server cache relocation. Actual content and configuration changes still reset it. A committed native receipt retains its terminal contract even when the originating adapter never returns. A busy recovery row defers through the existing scheduler while the replacement reaches readiness. **Steps to reproduce** Run the `rolling-web-deployment.runner-codex.daytona.recover-controller` Product E2E cell with separate controller homes and shared PostgreSQL/S3. The focused regressions reproduce the failures without a model call. ## What Changed - Read the managed instruction entry digest from the company-scoped durable file head. Do not restore or read the cache for this check. - Omit disposable instruction paths only when that durable authority exists. Fingerprint selected immutable skill versions in both runtime skill inputs. - Preserve the existing local and external instruction path checks, and unversioned skill identity. - Publish the accepted native terminal contract in the same transaction as the canonical result receipt. - Skip locked recovery rows and schedule the same run for retry. Avoid an audit write that would block on the same row. No ownership is transferred while busy. - Keep replacement readiness independent of an in-progress skill selection; migration and cache eviction still require exact pins. Missing durable skill versions still refuse startup. - Project only the instruction digest in SQL instead of loading the entire agent-file manifest per admission. - Add real PostgreSQL lock contention tests for the issue, run and coordinator, including the heartbeat startup caller and later claim. - Limit native test teardown to its owned database on a shared test host. The old behavior also fails on clean master. - Add cache-loss, scope, changed-content, changed-version and finalization regressions. Update the deployment qualification record. ## Verification - Both focused regressions failed before the source changes. - All 249 tests pass across `heartbeat-workspace-session`, `agent-file-durability`, and `paperclip-control-plane-port`. - Server typecheck and exact-revision build pass. - Assembled admission ran 59 passing cases but failed during native-suite teardown. The teardown failure reproduced on clean master. All 16 native resumption cases and teardown pass after the database-scope fix. - All three row-lock regressions fail with the pre-fix lock timeout and pass with deferred recovery. The full recovery suite passes 74 cases, including the startup caller; eight Docker-specific cases were not enabled. - The real attempt at `d2165dec5d` failed when B aborted on the run lock. Its credential and bucket were removed; all three provider sandboxes were independently confirmed absent. The next real attempt at `aae8a53d01` failed a different readiness gate while a run was preparing skill pins. That race now has a failing-before regression; all 33 readiness, durable-file and managed-skill tests pass after the fix. Its owned resources are cleaned. The `a15a12651c` live attempt passed both web replacements: 765 reads, 78 saves and 78 verified saves with zero HTTP probe failures. The first three turns preserved the provider session and native terminal contract. The fourth continuation failed because a safely suspended remote Runner was treated as live/idle; that distinct recovery gap is being handled in a follow-up. All three owned sandboxes were independently confirmed absent. No claim of full Railway qualification yet. ## Risks - The new fingerprint shape may reset a session once when upgrading from older fingerprint metadata. It is stable across subsequent cache replacements. - A versioned skill is identified by its immutable selected version; materialization and admission still enforce availability. Unversioned skills retain their existing path boundary. - A native terminal describes the accepted execution result. Workspace and issue settlement remain separate authorities. - This does not yet prove the ten-deploy staging campaign, natural database autosuspension or the production canary. ## Model Used OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution and live integration-test tools. The exact runtime model variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7751fc3107 |
fix(server): complete volume-free controller admission and recovery (#18543)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its volume-free web server must admit remote warm execution without
depending on a local disk.
> - Fresh run rows retain the initial legacy label during workspace
preparation.
> - The native runtime selector commits the execution mode later.
> - The earlier guard rejected valid remote runs before that selection.
> - This PR checks storage during preparation and the resolved lifecycle
during runtime selection.
> - Agent-file recovery also has to reconstruct disposable cache paths
from the retained remote directory.
> - The deployment test removes the complete predecessor home, including
immutable caches, to enforce that contract.
## Linked Issues or Issue Description
Refs #18518, #18529, #18533.
**What happened?**
A real Codex/Daytona deployment test failed during setup because the
volume-free workspace guard checked the initial run label. After fixing
admission, controller retirement first failed on an immutable skill
cache. Removing that cache exposed a production recovery failure: B
rejected the saved agent-directory path from A, even though the remote
execution identity was unchanged. The next live attempt recovered that
directory and transferred the same runner, but adoption then tried to
recreate provider launch assets from the deleted controller home.
**Expected behavior**
Allow remote workspace preparation before native selection. Reject
actual legacy and per-turn execution at the runtime boundary. Delete the
entire stopped predecessor home, including immutable caches, without
following links to shared dependencies. B must reconstruct its local
agent-file projection from the exact registered remote lease while
preserving the save baseline and unsaved remote edits. Reattaching an
already-running provider must not prepare a new launch or require the
old launch files. The full backend must recover the admitted
instructions from the matching durable run checkpoint.
**Steps to reproduce**
Run the explicit
`rolling-web-deployment.runner-codex.daytona.recover-controller` Product
E2E cell with a complete server/UI build, disposable PostgreSQL/S3, and
an owned Daytona fixture. The new boundary regressions also reproduce
the premature guard and missing runtime rejection without a model call.
**Paperclip version or commit**
The failures were observed on
`247552fe0719dcc92f7b42c50b658be6f624986d`. This branch is based on
`e122478368`.
## What Changed
- Validate Runner and remote workspace authority before runtime
selection without trusting the initial run label.
- Reject resolved legacy execution and local-source fallback, preserving
the native warm-policy check before admission.
- Rebind remote agent-file caches on ephemeral controllers and retrieve
their validated contents without uploading over the retained execution.
Preserve pending saves when the remote collector is unavailable.
- Restore over-quota remote directories without applying save quotas to
run admission; ordinary saves still enforce quotas.
- Skip provider launch asset materialization when adopting an existing
Runner. A real-process regression deletes the old skill/instruction
sources before reattachment.
- Recover admitted system instructions through Runner-owned checkpoint
validation for the same run, session and context digest. Fresh launches
still compose current instructions.
- Make the attention-retention test advance past its database-assigned
deadline, fixing a clean-master clock mismatch.
- Serialize remote collection and cleanup across controllers with an
operation-scoped Postgres advisory lock; fence stale cache-path updates.
- Keep synthetic deployment probe tasks in a separate disposable company
so they cannot block the business completion oracle.
- Share the launcher's directory cleanup with rolling retirement. Reject
a symlink root and skip linked descendants.
- Test admission with real PostgreSQL transactions and cleanup with real
read-only directories and external symlinks.
- Update developer documentation and the canonical qualification record.
Broader deployment acceptance remains in progress.
## Verification
- `pnpm -r typecheck` and `pnpm build` passed.
- All 156 workspace preparation, runtime selection and ephemeral
contract tests passed. Two new regressions failed before the source fix.
- Seven focused Product E2E cleanup/replacement tests and `pnpm
test:e2e:runner:typecheck` passed.
- `pnpm test:run` stopped at an unrelated 15-second chat integration
timeout after 338 passes. That exact test passed in isolation without
increasing its timeout. The later full-suite shards did not run.
- `pnpm test:e2e:runner:unit` reported 5,898 passes and 13 failures.
Clean master reproduced 11 failures in unchanged onboarding
expectations, operating receipts, Pi catalog assertions, workflow
bootstrap counts and embedded PostgreSQL startup. The other two workflow
assertions were fixed by the intervening master commit. These failures
are retained, not counted as passes.
- All 122 workspace-preparation tests pass after the reviewed
host-fallback fix. The first broader attempt exposed a test environment
leak; restoring the ephemeral flag between cases resolved it.
- Agent-directory and instruction regressions passed 179 tests. Five
focused cache-loss cases pass: different home, same path on an empty
filesystem, foreign lease, concurrent controller, and missing collector.
- The live fixture at `a18f6b85a3` gracefully stopped A and deleted its
entire home, then failed the 420-second remote reattachment deadline on
the saved local agent-directory path. That failure is retained; exact
owned sandbox cleanup was independently confirmed. The next live attempt
at `e14ca5f3e9` restored the cache and transferred the same runner, but
failed while rematerializing launch assets from A. Both failures are
retained and their owned sandboxes confirmed absent. No active-overlap,
ten-deploy, or production-canary acceptance is claimed.
- All 71 related Runner adoption/recovery cases pass, including the new
removed-context regression (failed before the source fix).
- The revised full agent-directory suite passed 110 cases and failed
three cleanup timing cases. All 20 cleanup/cache-loss cases then passed
in isolation with unchanged timeouts, including over-quota recovery.
- Server and Runner TypeScript checks pass after the latest corrections;
the live retry remains pending.
- The live test at `67714af9ff` adopted the same remote execution after
A was deleted, then failed while the full backend read the retired
instruction bundle. All owned sandboxes were independently confirmed
absent. Its regression now passes with actual backend recovery and no
new thread or turn. All 59 backend factory tests and focused checkpoint
identity tests pass. Runner and server TypeScript checks pass; the exact
committed server build passes. A fresh paid-model retry followed.
- The attention-retention CI timeout reproduced on clean master. The
deadline correction passes all 53 tests in that file with unchanged
timeouts.
- The real `cd690de4d3` run transferred the same runner/session after
deleting A, completed, and completed the next revision. During handoff,
372 reads and 38 saves had zero failures; all 38 saves were verified.
The full attempt was interrupted because the probe's open task
contaminated the business company oracle. This remains incomplete
evidence. The scope regression fails before the fixture fix, then
passes; all nine deployment support tests and E2E typecheck pass. All
three owned sandboxes were confirmed absent and temporary
credentials/storage were cleaned up. A clean corrected attempt is
running.
## Risks
- The runtime guard must still reject legacy and per-turn execution
before dispatch. Tests cover these refusals and a valid resolved warm
policy.
- Directory permission changes apply only during cleanup of stopped,
test-owned homes. Traversal does not follow symlinks. This changes no
production cache permissions.
- Cache reconstruction must not change remote execution authority or
rebase the original save baseline. Tests verify foreign-lease rejection,
preserved receipts, and absence of local-to-remote upload.
- Postgres locking is scoped to active remote file work and adds no idle
polling; local/durable-server locking remains unchanged.
- No schema or public API changes. External conversation exemptions
remain unchanged.
## Model Used
OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution
and live integration-test tools. The exact runtime model variant and
context-window size are not exposed in this session.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Latest real qualification at
|
||
|
DottaandPaperclip
|
006d5691a6 |
feat(server): support volume-free web server handoff (#18518)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. **Current behavior** The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. **Proposed behavior** Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. **Reason and benefit** This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1e8e32290c |
Give metadata project tasks private remote workspaces (#18514)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task can belong to a project without using that project's files. > - The specification gives a task without a selected or configured source its own private folder. > - The remote source planner instead rejects such a task when its project has no configured workspace. > - This stops an ordinary delegated task before native execution starts. > - This pull request selects an empty, task-owned remote source for that narrow default case. > - Explicit source choices, existing workspace authority, and repository requirements retain their current checks. ## Linked Issues or Issue Description Refs #18504 and #18351. The earlier reference-project fix covered additional project mentions. This fix covers the task's own organizational project. **What happened?** A task in a project with no configured workspace fails with `remote_workspace_source_unavailable`. Its project has no workspace policy, its four workspace selection and binding fields are null, and it requests no repository, branch, or local folder. The failure occurs before native runtime selection. A derived project folder exists in the project facade even when no source is configured; that derived path does not grant file access. **Expected behavior** Use the task's private remote workspace. Keep the project on the task for organization. Do not adopt the project folder, agent home, or another repository. Preserve explicit workspace and source requirements. **Steps to reproduce** 1. Create a project without configured workspaces or a workspace policy. 2. Assign a normal task in that project to a native sandbox agent. Leave all workspace selections and bindings unset. 3. Start the task with remote workspace storage enabled. 4. On the prior source, observe the source-unavailable error before allocation. On this source, observe a company-owned remote workspace whose `sourceIssueId` is the task and whose filesystem project and source IDs are null. **Paperclip version or commit** Reproduced against the deployed source planner at `756599e4ea069c9338c320fe8846523d3dfb92a0`. This branch starts at `987e0932bf66e703750cd04698b3c7e8993580a7`; the relevant planner and preparation inputs at that parent match the deployed source. **Deployment mode** Native sandbox execution with remote workspace storage. The regression tests use real PostgreSQL with the current migrations. ## What Changed - Allow an implicit empty source only after a company-scoped query confirms the project has no workspace rows and no explicit workspace policy. - Keep existing bindings, pending selections, explicit source choices, Git requirements, and local folder choices outside this default. - Add real PostgreSQL source controls and an assembled heartbeat preparation control through workspace reservation and the environment acquisition boundary. - Clarify the existing task-owned folder contract in the implementation specification. ## Verification - Causal RED: the new source control and the assembled allocation control both fail against the unchanged deployed planner with the observed source-unavailable error. - GREEN: 19 focused PostgreSQL controls pass across the source planner and assembled heartbeat preparation suites. The allocation control checks the persisted company/task owner, null filesystem project IDs, private remote root, original organizational project, and unchanged project and host sentinel files. - Negative controls retain configured repository sources and reject a configured source without a URL, an explicit missing source, an unknown selection, a required project policy, a required branch, an explicit local folder, a foreign project, and unsupported read-only project sources. - Current-source transitive TypeScript checks and `node scripts/check-maps.mjs` pass. The map check is structural evidence only. - Fresh exact-head Greptile is 5/5 with zero unresolved threads at `680461e180232d0f4a057528617e39b64b4f0e3a`. [Review](https://github.com/paperclipai/paperclip/pull/18514#issuecomment-6552946468). The official [Paperclip assessment](https://github.com/paperclipai/paperclip/pull/18514#issuecomment-6552931726) is 4/5 and remains recorded unchanged. A separate non-author source review found no actionable issue after the full producer/consumer trace, with manual confidence 4/5. Neither static review ran the tests or proved deployed native execution. - Full repository tests, build, and type checks passed in [canonical CI](https://github.com/paperclipai/paperclip/actions/runs/41269733360). All 59 checks are terminal: 55 success, three skipped, and the official Paperclip check neutral; no failed or pending checks. The successful build checked out merge preview `03dce9f9bc708e778dd214ca85dfa1ba58e0a806`, with parents current main `9dbbfa4fb298448db83ace946b338bff321d8b2a` and this exact head. Its four helper blobs match this head; the four added mainline document-harness paths remain unchanged. Local tests use source aliases with the current DB, shared, and Runner source, rather than borrowed generated declarations. - Actual QA staging qualification is pending deployment. The previous failed journey and cleanup evidence remain failed; these local checks do not relabel them. ### Review follow-up - Fresh workspace retry: the normal producer sets `taskDirectory:true` for this empty-source case. [Heartbeat](https://github.com/paperclipai/paperclip/blob/680461e180232d0f4a057528617e39b64b4f0e3a/server/src/services/heartbeat.ts#L2288-L2296) separates the saved filesystem project from the organizational project. [Environment selection](https://github.com/paperclipai/paperclip/blob/680461e180232d0f4a057528617e39b64b4f0e3a/server/src/services/heartbeat/environment-selection.ts#L506) selects the fresh task directory. Preparation passes that flag, and the planner's existing task-directory return runs before the new binding guard. The proposed fresh failure omits that producer flag. The existing fresh-intent and binding checks stay strict. This is source evidence, not an extra runtime test claim. - Empty policy object: the [public project policy schema](https://github.com/paperclipai/paperclip/blob/680461e180232d0f4a057528617e39b64b4f0e3a/packages/shared/src/validators/project.ts#L24-L43) requires `enabled:boolean`. The project PATCH schema makes outer fields optional and does not make nested policy fields optional. A clear uses `null`; `{}` is rejected. A defensive parser's treatment of unknown stored data does not establish a valid `{}` producer. Keep the narrow fallback closed for non-null policies. - Boolean grouping: JavaScript evaluates the existing predicate as `sourceId || (projectId && !implicitTaskDirectory)`. The explicit-source and metadata-default controls cover those branches. The optional parentheses suggestion does not identify a behavior change; this reviewed source remains frozen. - The official 4/5 assessment and its static verification limits remain visible. The independent manual review withdrew its initial fresh-retry concern after the full producer trace. Actual staging qualification still requires deployment and a new real journey result. ## Risks - The new query runs only for a source-less default admission. It reads one authorized project and uses `NOT EXISTS` for that project's source rows. It adds no recurring poll, write, provider call, or local folder access. - A project with an explicit workspace policy remains subject to the prior source requirement. This conservative rule does not reinterpret policy fields. - The allocation control uses the real planner and PostgreSQL reservation. It stops at environment acquisition, so it proves workspace ownership rather than provider filesystem execution. Live staging qualification is still required. - No API, database schema, permission, model, budget, run limit, or QA journey oracle changes are included. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. This session does not expose a more precise backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
756599e4ea |
fix(workspaces): keep project references without source folders (#18504)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can receive project references in normal task and chat messages. > - Remote workspace preparation checks those references before a native run starts. > - A project with no workspace source has no folder to attach. > - The current check rejects that project as an unsupported read-only folder. > - This pull request keeps the project reference and reports the missing source without attaching files. > - Projects with configured sources keep the current remote access restrictions. ## Linked Issues or Issue Description Refs #18403 and #18351. This is a product prerequisite for the hiring journey. It changes no journey code. **What happened?** A native chat run failed before execution when its message referenced an authorized project with zero configured workspaces. The error was `workspace_validation_failed`: “This remote runtime cannot safely provide read-only project folders.” The project had no repository or checkout. The failed QA run and original grade remain preserved. **Expected behavior** Keep the task workspace when a referenced project has no configured source. Report that no project folder was attached. Continue to reject configured cross-project sources that require an unsupported read-only mount. **Steps to reproduce** 1. Use a native remote task folder with multi-project references enabled. 2. Mention a same-company project that has no configured workspaces in a normal comment. 3. Prepare the run workspace. **Paperclip version or commit** Reproduced on `b889ac39b7fbed558c5f4cfbbc2b73c4ca916810` with the real workspace resolver and PostgreSQL migrations. The staging failure was recorded in workflow [41253428446](https://github.com/paperclipai/paperclip-cloud/actions/runs/41253428446). ## What Changed - Distinguish authorized project references with a known empty workspace list from configured sources. - Return a resolution warning and failure receipt for an absent source. Do not attach a folder or create a workspace. - Preserve project authorization, company filtering, the disabled remote-source gate, and the read-only restriction for configured sources. - Add real PostgreSQL regression cases and document the reference contract. ## Verification - Causal RED on unchanged `b889`: the metadata-only project case failed with the same read-only error. The real repository negative already passed. - GREEN: eight PostgreSQL cases pass. They cover an absent source with the remote-source switch both on and off, a saved existing workspace, a repository, a configured source without a repository URL, the disabled remote gate, mixed references, and a foreign company. - The existing assembled workspace test also passes: a real read-only project source is rejected before reservation or environment acquisition. Total: 9 selected tests pass; 119 unrelated tests were not selected. - Tests use PostgreSQL 17 and the current source migrations. The local Vitest configuration selects current DB, shared, and Runner sources because the borrowed dependency output predates the remote workspace feature. - Current-source TypeScript checking of both changed TypeScript files and their dependency graph passes. It uses the repository's real Express actor declaration. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Canonical reproduction: `pnpm exec vitest run --project server server/src/services/heartbeat/remote-workspace-source.test.ts` with the existing embedded PostgreSQL support or `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` set to a disposable test cluster. - Historical `0f2b` passed all 59 CI checks. A separate non-author manual source audit reported no actionable findings at confidence 4/5; it did not rerun tests or qualify live QA. The official 4/5 review requested a flag-scope test and saved-workspace control. This test-and-documentation update adds both, with production source bytes unchanged. - Current-head canonical [CI 41258109282](https://github.com/paperclipai/paperclip/actions/runs/41258109282) passed: all 59 checks are terminal (55 success, 3 skipped, 1 neutral), with no failures or pending checks. Build and typecheck passed. The eight-case regression file passed in the canonical Linux server 1/12 lane. Its checkout `6f69ccca70d10e78885b133fb54a80940bf175a5` has parents `b889` and this exact head; its tree equals this head. - Fresh Greptile and official Ponytail assessments each give 5/5 for exact `27660027d12ff31427d611019e90dabfb6d8503a`, with zero unresolved review threads. The separate manual audit remains 4/5. - Live QA remains pending on a reviewed app build. This regression proof does not claim that the hiring journey passed. ## Risks A project source could be added after project hydration. This change uses the same run-preparation source snapshot as the existing authorization and source check. It only permits a positively known empty workspace list. Configured sources, unknown shapes, foreign projects, and existing execution authority keep their current checks. No schema, credentials, provider, model, budget, or filesystem permissions change. ## Model Used OpenAI Codex, GPT-6 family. The session does not expose a more specific backend model ID or fixed context window. Used reasoning, repository tools, TypeScript and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
66e8fab944 |
fix: complete native chat responses and improve API discovery (#18364)
## Thinking Path > - Paperclip must perform work and report its outcome accurately. > - A chat response and its delegated or operator-owned work are separate stages. > - Native chat contracts instead required literal completion of the user's whole request. > - Agents kept searching or waiting even when a truthful handoff should finish the response. > - OpenCode also has separate acknowledgement backlog and empty-response failure paths. > - This PR aligns chat completion with its current stage and makes OpenCode completion truthful. ## Linked Issues or Issue Description Refs #18098 for retained OpenCode evidence and #18377 for separate eval setup, oracle and reporting repairs. **What happened?** Real pinned OpenCode 1.18.34 reproduced an MCP timeout behind 800 text fragments. Pilot [41222196525](https://github.com/paperclipai/paperclip/actions/runs/41222196525) separately exposed a fabricated successful reply after a reasoning-only response. Later restart attempts timed out without an answer. Their native criterion literally required restarting the instance, although the chat instructions and authority required an operator handoff. General authority guidance alone did not resolve it. **Expected behavior:** A standing chat completes its current authorized response. It performs and verifies permitted actions, verifies delegated handoffs, or explains the specific unavailable action and operator path. The worker's task remains separate. Ordinary execution tasks must still complete their requested work. Empty provider responses must not become invented success. **Version and reproduction:** Actual pinned OpenCode and native Rust Runner, with retained full-stack AWS attempts. Product baseline master `51ea624344`. This PR adds no historical catalog to master. ## What Changed - Build standing-chat completion contracts from the server-owned conversation identity. Scope each current request to the response stage; preserve request text, criterion IDs, order, human-response scope and source revisions. Require verified authorized actions and truthful handoffs. Ordinary task contracts stay strict. Existing admitted inputs retain their saved contract during recovery; new contract content uses the existing revision/hash mechanism. - Guide prospective defaults to their owning resource and preserve existing work unless backfill is requested. Guide unavailable board/operator operations to the supported handoff. These are general chat instructions with no eval identities or expected answers. - Batch adjacent assistant text for at most 50 ms and 4,000 UTF-8 bytes. Preserve text, metadata and order; flush before non-text frames and shutdown. - Give an empty OpenCode response one correction within its existing turn, model, reasoning, context and deadline. Accept actual completed answer parts with incomplete headers. Preserve actual active tool work, bound idle evidence grace and fail truthfully if correction stays empty. - Reject empty OpenCode terminals in Rust without actual prose or an accepted semantic result. Never invent successful replies or evidence. Refresh generated project-discovery contracts. - Preserve accurate public API-discovery descriptions for existing project defaults and managed instance restart operations, including their field scope and authority limits. ## Verification **Qualified: [pilot 41256592937](https://github.com/paperclipai/paperclip/actions/runs/41256592937) passes 3/3, with all 35 independent checks passing.** | Case | Checks | Full case duration | Evidence and cleanup | | --- | --- | --- | --- | | Instance restart | 9/9 | 140.1 s | complete | | Project defaults | 11/11 | 204.2 s | complete | | Project work | 15/15 | 204.0 s | complete | All three are original-attempt results from source `c51e079f7f2364305b131ffd549e263ff78c5335`, containing the exact product files from `13c2b61f31d873a475ea0c3c026b19926ff42301`. Actual saved conversation contracts were inspected, as were ordinary native worker contracts where present. Each protected archive includes the actual API state and final PNG; cleanup completes. Original queries, `openrouter/deepseek/deepseek-v4.1-flash-1017`, finite case clocks, concurrency one and zero eval retries are preserved. Earlier passes do not transfer into this qualification. Both current heads pass all checks: product **62** (56 success, one neutral, five skipped), catalog **56** (54 success, one neutral, one skipped). Both have exact-head Greptile **5/5**, zero unresolved review threads and clean mergeability. The three trusted private-evidence protocol files remain byte-identical to the dispatch workflow. Raw evidence stays protected. - **2,044 API discovery tests pass.** Red controls showed that “restart instance” selected unrelated drain/settings operations and “save project defaults” selected repository replacement. Accurate public summaries now rank the existing operations correctly; route registration preserves descriptions. No endpoint, permission or field behavior changes. - **112 real-database, authenticated HTTP, native Runner and OpenAPI consumer tests pass locally**, using disposable task-owned Docker PostgreSQL. It was removed afterward; shared IPC was untouched. These cover persisted contract selection, admitted native input and the model envelope together. The isolated CI suites also pass. - **77 focused contract/input/context tests pass.** Red controls catch the wrong conversation predicate; canonical identity requires both agent and user IDs, including the actual heartbeat projection. Ordinary tasks, agent-only issues and planner tasks remain strict. Coalesced requests preserve criterion identities, source hashes and human-response scope. - **144 OpenCode driver tests pass**; native core **441 pass, one ignored**; packaging **56 pass, six skipped**. Server typecheck and build pass. - Real pinned OpenCode/native Runner: a delayed 25-second acknowledgement completes within the original 30-second MCP deadline. The text-fragment control changes 817 notifications / 54.1 seconds / timeout to 45 / 19.0 seconds / accepted finish. Separate fast, six-minute and real authenticated API probes complete 32 tool calls plus finish. These do not reproduce every remaining historical MCP timeout, and no common cause is claimed for all such failures. Retained failures remain attributed to their original source. Earlier runs exposed fabricated empty-response success, prospective-default backfill, a literal whole-task chat criterion, fixture readiness and oracle/reporting defects. The immediately preceding [41250502608](https://github.com/paperclipai/paperclip/actions/runs/41250502608), product `c0bf249807` / source `7b1de24ce1`, is **1/3**: work 15/15; restart 4/9 with the repaired contract actually admitted but no answer; defaults timed out before settlement, then exceeded the unchanged 256 MiB original-evidence allowance. Defaults domain behavior on that attempt remains inconclusive. #18377 retains the full attempt ledger, including missing archives and pre-paid cancellations. The fresh 3/3 establishes these selected outcomes; it does not establish full-catalog reliability. ## Risks The shared contract change applies to standing native chats across providers. It changes model instructions and completion scope, while permission, approval and review gates remain enforced by their existing owners. Incorrect model claims remain possible; the product evals independently require actual mutations, native work and delivery. The correction adds at most one provider request within the existing deadline; text batching can delay adjacent text by 50 ms. Three qualified attempts do not establish full-catalog reliability. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, tool use, code execution and live eval tooling. Exact hosted model revision and context-window size are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge - [x] Final same-source three-cell pilot passes 3/3 with required evidence and cleanup --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9ca9af544a |
Drive remaining live run and resource recovery from durable work (#18419)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
eb12cfc96b |
perf(github): activate connection work from events and deadlines (#18395)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
5e4cc01073 |
feat(workspaces): keep remote files off the webserver disk (#18403)
Keep native sandbox workspace files independent of the webserver disk. Use encrypted saved versions and Postgres ownership so a replacement server can recover files and continue work. Enable the path by default for new eligible workspaces when durable storage is configured. Preserve explicit opt-outs and existing host workspaces. Include safe snapshot repair, Files access, import, migration and retention. Bound saved-Files resources, preserve imported Git recovery, reject unfinished Git operations, publish immutable concurrent captures, and verify permanent object-version deletion. Validated 471 targeted local controls, build/typechecks, all 40 CI partitions, fresh Greptile 5/5 and security review. The PR records original failed attempts and scoped retries. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9a3f395737 |
fix(server): settle routine history without idle scans (#18399)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
dce383c733 |
perf(server): stop recurring attention retention scans (#18388)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
b6b57079b9 |
fix: evaluate status cards on committed changes and deadlines (#18401)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
3ee4c5b613 |
Make saved execution receipt recovery react to durable work (#18409)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
baf651c8d0 |
perf(heartbeat): drive retry promotion from durable deadlines (#18387)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
492d537340 |
fix(native): recover controllers across webserver replacement (#18343)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native execution runs in a remote Runner while the webserver owns its controller. > - Encrypted PRP checkpoints and session snapshots already survive a webserver restart in Postgres. > - Ordinary warm and per-turn admission still needs local markers and supervisor metadata to find that state. > - Replacing the webserver with an empty native controller directory can therefore lose a usable retained session. > - This change adds bounded recovery metadata to the existing snapshot transaction and reconstructs local projections from the selected durable source. > - The benefit is native execution continuity across webserver replacement, with the existing ownership and agent lifecycle policies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Native remote controller recovery and retained-session continuation during a webserver upgrade. **Subsystem affected** server/ — native runtime, checkpoint persistence, recovery scheduling and internal upgrade reporting. Product E2E qualification also changes. **Current behavior** Without both the local PRP marker and supervisor file, normal continuation can return before discovering the saved Postgres checkpoint. The snapshot already exists in Postgres, but its local reuse compatibility envelope cannot be reconstructed. **Proposed behavior** Select the existing durable source, validate its identity and ownership, restore its PRP state and supervisor projection, then use normal authenticated admission. Missing or inconsistent evidence holds recovery. It does not authorize fresh provider work. **Reason and benefit** A replacement webserver must recover active execution and later continue its retained session without the old server's native files. This is one component of deployment continuity. HTTP traffic switching, workspace storage migration and Railway rollout remain separate. **Breaking changes** No new public endpoint, schema migration or database pool. The private run-profile envelope is additive. Disk-independent rollback also requires a reader for the staged controller format and compact checkpoint part; the first production release containing these readers is the floor. Older file-backed sessions need an owner checkpoint or the bounded internal importer. **Additional context** Related foundations: #17401, #17916, #17254, #14621, #14622. The implementation contract and open acceptance gates are in [the dated plan](doc/plans/2026-10-30-native-controller-checkpoints.md). ## What Changed - Persist a versioned recovery envelope, at most 4 KiB, beside the redacted session snapshot in its existing transaction. Keep it out of public run projections. - Compare frozen admission separately from prepared attachment instructions during staged recovery, preserving exact template/identity/authority checks. - Bound the existing lazy checkpoint pool at three connections after measured two-slot burst backlog; keep request/renewal capacities and SQL operation counts unchanged. The tuning is a separate commit. - Discover and restore the exact selected retained source without local native markers. Preserve the 32-candidate window, progress barriers and current authority checks. - Fence snapshot and result writes. Acquire coordinator, issue and run locks in order and check expiry after lock waits. Combine snapshot ownership/read work into one statement. - Reuse existing recovery deadlines and drain/resume scheduling. Retained-owner waits preserve the original unstarted run and actual lease deadline. Stop, ownership and provider-progress fences precede deferral. Add bounded native evidence reporting, retained-source import and retention protection. - Admit a settled warm source with an authenticated idle goal-read suffix while retaining active-work and mixed-authority holds. - Permit lifecycle `turn.stop` checkpoints after accepted results under existing ownership fences. Claim the existing successor coordinator before retiring the old warm session, and stage shared attachments only after verified retirement. Preserve omitted isolated-home defaults and existing fresh skill-selection order during recovery without hiding real configuration changes. - Carry the exact durable six-field PRP binding during active recovery after a healthy warm handoff. Keep the Runner mismatch guard and preserve opaque attachment IDs. - Preserve an already verified active or idle warm process during remote preparation; revalidate its proof/liveness and retain later authenticated PRP attachment. Ordinary resumed leases retain suspension checks. - Drain retained source events under their original authority, reconcile immutable source accounting and block unknown source tool dispatch before target work. - Persist compact staged handoff intent through authenticated adoption and recover direct/listener command-loss windows without another Runner or duplicate attachment. - Rebuild suspended lifecycle receipts for resumed prepared tasks, and retry missing lifecycle reads after partial PRP materialization without overwriting contradictory evidence. - Add normal warm/per-turn diskless continuation coverage, real Postgres races and SQL-cost checks. Add a reproducible paired load workload. - Give only the explicit outage fixture a 15-minute provider idle window, with a red/green catalog regression preserving all production timing and lifecycle policies. - Publish the goal-less task-session pointer in the existing post-acceptance terminal checkpoint transaction, with current ownership, accepted-result, reset and newer-progress fences. Keep a private bound receipt so late completion cannot duplicate or resurrect publication. - Use a retained warm sandbox in the dedicated continuity fixture, and accept an exact output token followed by an explanation. Keep the strict identity oracle. - Extend the explicit Daytona cell to a five-minute active outage and another replacement after completion. Require retained provider identity, a new tool receipt and a useful follow-up result. Add an owned loopback Postgres E2E fixture for hosts that cannot start embedded Postgres. ## Verification **Merge recommendation — `fc68584ce6231a02e47ecd743f55b036ccea034f`: ready for the qualified native-controller component.** Full CI, fresh review, local application paths, controlled performance qualification and the strict live Linux/Daytona case pass. Independent resource cleanup is complete. Broader rollout, browser reconnection and the unexecuted transport/platform combinations remain integration gates. Integrated 484 passed full local recursive typecheck/build and ordinary active→completion→cache-loss→C component qualification. Fresh review then found a valid attached-wake failure: preparation appends attachment instructions before the verifier compares execution to the frozen admission. 9f4 fixes that by separately checking the pre-preparation admission and exact prepared execution. Both hashes are captured privately per admission; the full template, identity, configuration and authority checks remain. Its regression fails before the fix; 36 focused cases, 42 staged source/target cases and server typecheck pass afterward. The review thread is resolved. Exact-head full CI passes ([run 41168228822](https://github.com/paperclipai/paperclip/actions/runs/41168228822): 48 successful jobs, one intended skip); the complete `pnpm test:run` partitions are forced, with the five native suites in the Runner lane. This is not a claim that an unsharded local command passed. The fresh exact-head Greptile check succeeds with 5/5 and all six threads resolved. The original two-connection database comparison closes **steady PASS and burst FAIL**. Every relative CPU/WAL/request-p99 gate passes and all operations/ownership remain correct, but candidate arm 3 drains in 1,015.99 ms and baseline arm 4 in 1,059.67 ms against the unchanged 1,000 ms absolute gate. Those results remain failed. A separate 2→3→4→2 diagnostic reproduces drain times 1,051.57 / 956.79 / 882.02 / 1,065.72 ms: three is the smallest tested passing cap, and the repeated two-cap failure survives run-order/cache effects. Separately reviewable `fc68584ce6` changes the existing lazy checkpoint pool default to three connections. Request capacity stays ten and liveness two; idle release, shutdown fencing, transaction counts and pool count are unchanged. Thirteen DB lifecycle/cap tests and DB typecheck pass. Full recursive typecheck/build pass at exact |
||
|
DottaandPaperclip
|
59da15066a |
feat(runner): deliver native Kimi Code (#18391)
Deliver native Kimi Code setup, exact models, scoped credentials, verified packages and runtime packs. Preserve current upstream release attestations and restore verified same-run remote retry dispatch without weakening process ownership. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
9e904ee519 | perf(server): overlap independent heartbeat preparation reads (#18372) | ||
|
DottaandPaperclip
|
0e15c92073 |
feat(skills): make managed content independent of web-server disk (#18334)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use installed skills and exact version pins during execution. > - Managed skills still depended on mutable directories on the web server. > - A replacement server could lose acknowledged edits or remove an unassigned skill whose directory was absent. > - This pull request makes immutable versions the authority for managed content and installed repository reads. > - The server can rebuild delivery files from durable content after replacement. > - This establishes skills readiness for ephemeral web servers. General server draining and workspace recovery remain separate work. ## Linked Issues or Issue Description **Subsystem affected** Skills services, shared contracts, database receipts, Skill Studio, and runtime delivery. **Problem or motivation** A fresh web server must discover, read, edit, and deliver protected installed skills using the same database, object storage, secrets, and supported application artifacts. It must not need its predecessor's files or process memory. **Proposed solution** Use `managed` ownership with a null source locator. Read installed content by immutable version. Prepare packages before a short publication transaction. Persist execution selections before materialization. Recover exact pinned runtime assets and reject deletion while active or recovering pins exist. **Alternatives considered** Reconstructing a mutable installed directory preserves the old source-of-truth ambiguity. Permanent dual writes create two competing authorities. This change uses one managed model after a bounded legacy conversion. **Roadmap alignment** This improves the existing Skills Manager, Studio, version history, and Cloud execution capabilities. The maintainer supplied the implementation contract. This does not add a separate skills product or deploy fleet changes. Related work: #18082 supplies immutable package storage. #17273 addresses edits outside the API for linked local skills; this PR keeps that source category separate. #2648 is a closed earlier ephemeral-filesystem proposal. ## What Changed - Add managed ownership and focused content publication and execution materialization modules. Keep the archive format and package storage tables. - Publish managed create, save, delete, fork, rename, restore, and catalog content with version checks. Add durable create receipts. Preserve file-save receipt responses. - Return file version identities. Keep Studio drafts after a conflict and provide an explicit reload action. A delayed reload cannot overwrite newer typing or navigation. - Persist exact runtime selections, including empty selections. Rebuild pinned assets and verify their admitted digests on a replacement server. - Preserve source access on forks and disconnected copies. Keep the existing workspace policy category for managed skills. - Add bounded conversion, eligibility, and independent verification phases. Load persisted Cloud configuration in the standalone CLI without seeding or changing it. Reject partial current versions against installed inventories; recover only complete saved content or the exact source. Document storage migration blockers and the compatible rollback floor. - Add separate-process PostgreSQL/S3 qualification and update domain, retention, policy, and UI regressions. - Pass the plugin worker manager into Studio execution, admit restricted `skill_test` mode through native Runner contracts, and preserve admitted scope across native and REST tools. Native tests cannot create ordinary tasks, browse company data, or use external tools. - Select the exact durable Studio revision even when unassigned or Beta skills is off. Persist Studio output and audit atomically with native task completion, preserving the result on replay. ## Verification - Final revision `13e8af480b88ba6fea5737bac4bd81104d604792` passed full CI: all 49 workflow jobs completed successfully ([run 41127396098](https://github.com/paperclipai/paperclip/actions/runs/41127396098)). This includes typecheck, test suites, Runner checks, browser checks and build. Greptile review 15 is 5/5 on that same revision, with zero unresolved threads. The PR is ready for review. - Local workspace typecheck and build passed after integrating master. Focused checks passed: 33 execution/completion cases, 52 Runner HTTP cases, 16 managed-content cases, 17 Studio cases, the new real-process CLI regression, map checks and UI token gates. The CLI regression proves inspection does not seed missing Cloud configuration. Server typecheck passed after that fix. - The independent-process PostgreSQL/S3 matrix passed through normal HTTP skill routes. It covers A → B → C replacement, lost commit responses, upload crashes, concurrent saves, binary contents and executable delivery. Metadata listing and warm selected delivery retain zero object downloads. Local timing measurements are recorded; they do not establish hosted latency. - Hosted A → B → C passed on the owned disposable test tenant with external Postgres, S3 and a real remote Paperclip Computer. Predecessor temporary homes were absent. C read and executed B's acknowledged managed save. New runs used repository v1 while v2 was installed. Hashes matched an independently checked exact upstream commit. - Compatible image rollback to `6518b323` passed. Studio retained the unassigned fork's C save. A fresh remote run executed the newer managed content and historical repository pin. The actual shell event confirmed expected hashes, stdout, exit 0 and mode 0555. - Runtime candidate `9200e3fcb4` deployed successfully ([Cloud run 41124147876](https://github.com/paperclipai/paperclip-cloud/actions/runs/41124147876)). The final Studio test delivered only the unassigned fork's exact v3, executed its script and persisted a successful output snapshot. Independent command-event inspection confirmed all three hashes, stdout, exit 0 and mode 0555. All 14 installations were version-backed. Independent verification checked all 25 versions: 23 remote objects, two bundled database snapshots, zero blockers. - The deployed image is `9200e3fcb4`. Later master integration, the client-only delayed-reload draft guard and the CLI bootstrap have separate automated proof. The CLI bootstrap was also exercised against the tenant's stored configuration and storage through the same read-only services. This PR does not claim that the deployed image is the final PR revision. - Live Studio checks exposed worker wiring, admission, access scope, exact selection and completion-publication gaps. They are fixed. Failed attempts remain in the evidence. The full local suite exceeded 80 minutes under host contention and was stopped. Later combined/isolated local attempts also timed out; they are not counted as passes. Full CI covers the integrated suites. Earlier CI found an obsolete restart-scope fixture, which was corrected and passed all 52 HTTP cases. - Reproduction and migration: [`doc/skill-ephemeral-web-servers.md`](doc/skill-ephemeral-web-servers.md). Source revisions, deployments, versions, run IDs, hashes and failure history: [`doc/plans/2026-10-30-ephemeral-web-skills.md`](doc/plans/2026-10-30-ephemeral-web-skills.md). ## Risks - Legacy conversion requires stopped admission and retired filesystem-authoritative writers. Pre-conversion binaries are not supported rollback targets afterward. - Cloud readiness requires remote object storage. Local provider receipts and provider mismatches remain blockers until objects are copied and verified. - Linked local sources retain filesystem semantics and are outside the cloud guarantee. - Cold reads still fetch the existing archive format. Warm runtime preparation retains the verified-cache path. - Bundled reconciliation preserves an installed head. Existing release snapshots remain selectable. An older server must not replace a newer head with its shipped files. - No production volumes, fleet defaults, or general deployment coordination change in this PR. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and browser use. The exact backend model ID and context-window size are not exposed in this session. No subagents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9f524ec1a7 |
fix: persist native Runner usage before transport acknowledgement (#18354)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent execution must preserve usage so Paperclip can price work and enforce budgets. > - Remote Runner execution can outlive its web receiver. > - The receiver previously acknowledged Runner events before its later usage recorder saved the accounting receipt. > - This pull request saves required native usage receipts before that transport acknowledgement. > - A replacement receiver can accept the same retained report and create one ledger charge. ## Linked Issues or Issue Description **What happened?** Native remote Runner usage could leave its durable outbox before the web receiver saved its accounting receipt. The later recorder required a local receipt directory. A replacement web receiver could lose the only remaining report when the database write failed. **Expected behavior** The Runner retains unaccepted structured usage. Postgres owns accepted receipts. Repeated delivery preserves one receipt for each source report and one charge for the run. Missing usage keeps accounting pending. **Steps to reproduce** 1. Start a supported built-in native remote Runner. 2. Make the receipt database insert fail after the provider reports usage. 3. Replace the web receiver and its local directory. 4. Reconnect the same Runner and inspect the saved receipt and ledger. **Paperclip version or commit** The initial investigation used `ebd05e9f4c88c66af7473db86bbaacfd1544aa0a`. **Deployment mode** A web server with a remote native Runner and shared Postgres. Related work: #18327 owns output logs and provider traces. This change has no dependency on its S3 export path. #18031 covers interrupted Codex receipt recovery. #18005 covers legacy spool reads. #17516 covers Codex stop-boundary retention. ## What Changed - Add native producer receipt delivery through the existing Runner callback before transport acknowledgement. - Reuse `run_usage_receipts`, stable source sequences, and deterministic report identities. Reject conflicting replay and retry a lost commit response with the same receipt. - Restore per-turn usage from structured receipt facts. Preserve provider turns that share one PRP envelope turn. - Bind retained warm transport callbacks to the current run authority epoch, allowing old attachment control acknowledgements while preserving accounting ownership. - Persist Codex session baselines and cumulative totals in the existing provider checkpoint. Mark qualified whole-run deltas, preserve them across resume/restart/attachment, and leave old unmarked reports incomplete and unpriced. - Normalize the bounded final usage tail before the terminal event. Retain later increases for explicit reconciliation, block further attachment, and fail an affected successor without changing its baseline. - Complete runs from the accepted immutable receipt and its exact saved rate-card amount; preserve existing live-budget enforcement. - Skip the local receipt recorder for supported built-in native remote Runner execution. Keep local, legacy, and external Dot/Muse behavior unchanged. - Keep pricing, reservations, ledger settlement, source fencing, and budget holds with their existing owners. - Put the actual Runner integration test in the existing Rust-cached native test lane. Build before opening Postgres and give cold compilation its own bounded budget. ## Verification - Fresh Greptile review is 5/5 on `e7376a50c8162770da1d581ecf25c8cffac64520`, with no unresolved findings. Parent and independent peer review approved the final changes. - Real PostgreSQL 18.6 and a real Rust Runner exercise failed receipt insertion, receiver-directory loss, replay from the same producer, and exactly one 25-cent ledger charge. The actual accounting spool path stays absent. Optional logs/traces are disabled. The provider is controlled; no paid request is made and the fixture never pre-saves the expected receipt. - Producer checks passed 32 usage/provider tests,37 terminal controls, corrected completion, all 3 previously failing exit cases, and both full transport goal-continuation cases. Rejected active-goal completion retains its 120-token tail before failure. Baseline 100→successor150→33 duplicate old snapshots→old120 retains explicit incomplete reconciliation evidence and positive budget exposure. - Heartbeat executor checks passed 982/982. PostgreSQL tests preserve exact 1.002 USD pricing, the 100.2-cent ledger amount, budget holds, source takeover, partial/final receipts, cost-only corrections, equal rate-card replay, and stale-completion interleaving. Newer incomplete usage remains pending rather than settling an older snapshot. - Runtime replay and strict usage-schema checks passed 2/2; generated contracts and Rust formatting pass. Test-partition contracts passed 32/32. - [Final combined-head evidence](https://github.com/paperclipai/paperclip/pull/18354#issuecomment-6531526158) records full workspace typecheck/build and 41/41 accounting tests against all five cloud PR heads, including this final accounting commit. It includes commands and evidence limits. - Exact-head CI is green: 53successful checks, 6intentional skips, and the legacy Snyk status passed. Typecheck, build, Runner, server, browser, and aggregate gates pass. Runner Vitest1/2 initially failed an unchanged mocked Antigravity login fixture; its focused case passed locally, then one unchanged job retry passed 131 files/1995 tests, including all 11 Antigravity tests. The initial failed attempt remains recorded in CI. Initial local attempts exposed the host SysV semaphore limit and corrected harness path, IPC readiness, and queue-counter defects. A temporary test loader selected an existing POSIX-semaphore PostgreSQL 18.6 build without changing repository dependencies. An optional expanded debug provider run was interrupted at its unrelated 8192-turn rollover stress case; no full-file local pass is claimed. Required failure cases were run directly, and the full Runner CI remains required. ## Risks - The guarantee covers built-in native remote Runner execution. Local native execution, legacy adapters, external Dot/Muse execution, and server-originated model calls retain their existing contracts. - This does not provide traffic switching or general web-server failover. The existing controller identity, checkpoint, and generation gates still apply. - The receipt JSON gains optional structured producer facts. No schema migration or accounting S3 spool is added. Text output and raw trace payloads are excluded. - Codex accounting authority requires a Runner artifact built from this provider baseline implementation and qualified by digest. Update receivers and strict usage-schema readers before releasing that Runner; PRP does not negotiate this payload field. Existing unmarked Codex reports remain pending/unpriced. No deployed Runner artifact is claimed fixed. - Usage arriving beyond the bounded terminal tail may require manual reconciliation or a fresh session. Its evidence remains in the existing producer checkpoint/outbox; no adjustment ledger or recurring scan is added. ## Model Used OpenAI `gpt-6.1-sol` assisted with the implementation, tests, and documentation through code execution and repository tools. The context window size is unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
dbff3747b8 |
feat(server): persist run logs and provider traces in S3 (#18327)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators use run logs and restricted provider traces to inspect agent work. > - Local log and trace files depend on the web server's persistent volume. > - That dependency prevents overlapping volume-backed Railway deployments. > - This PR reuses the instance S3 store for batched log and trace payloads with conditional publication. > - Existing Postgres rows retain their contracts; native export recovery reuses events already committed before Runner acknowledgement. > - S3-configured servers automatically copy and verify retained history before a later deployment cutover. ## Linked Issues or Issue Description **Subsystem affected** Server run logs, restricted provider traces, storage providers, and migration tooling. **Problem or motivation** Replacing the web server and its disk must preserve committed log output and retained traces. The current S3 mirror can repeatedly upload a growing file. Moving those payloads into Postgres would add database writes, WAL, replication, and backup work. **Proposed solution** Reuse the existing instance storage configuration, S3-compatible provider and credential chain. Store new run logs under run-logs/ and opted-in raw traces under provider-traces/, below the existing prefix. Cloud keeps its existing bucket and bucket-scoped credentials per stack. Add no new configuration switches. Store immutable payload segments and bounded binary range indexes directly in S3. Publish each batch with an ETag-conditional head. Keep existing database ownership and lifecycle records. Recover canonical native exports from their existing validated control event source, checkpointing export progress in S3. Add no new payload writes to Postgres. Preserve local defaults and migrate settled history with verified byte counts and digests. **Alternatives considered** Postgres payload chunks add control-plane load. Repeated whole-file mirrors amplify transfer work. The first index prototype left superseded index nodes; the final binary index keeps all normally published nodes reachable. **Roadmap alignment** This is a storage prerequisite for replacing the main web/API server. It does not enable deployment overlap, detach a volume, or change native Runner protocol acknowledgements. The maintainer approved this implementation scope. ROADMAP.md has no duplicate implementation item. Related PR: #16770 improves the existing in-flight mirror. This PR preserves that legacy backend and uses the existing instance S3 profile for new journals. ## What Changed - Automatically select S3 segments for new logs and opted-in traces when instance file storage uses S3. Reuse bucket, endpoint, region, prefix, path style and credentials. Remove separate backend switches. Keep local-disk instances local. - Share the existing storage configuration resolver with product files. Preserve config-file settings, environment precedence and the existing root prefix. Keep pre-existing run-log mirror settings for legacy history. - Add bounded buffering, conditional ownership fencing, immutable range reads, stable byte offsets, and completion hashing. - Dispatch all existing log consumers through the recorded backend. Coalesce output progress writes and publish live log messages after storage receipts. - Recover interrupted canonical native exports from their existing durable source. Exclude separate credential, cleanup, and reprojection records. Replay missing callback ranges and deduplicate late delivery. Repair settled archives after controller lease expiry. Bound shared repair work and stop it when all readers cancel. Serve the published prefix when best-effort repair fails. - Add encrypted optional trace channels, replacement reads, durable deletion tombstones, and deadline-driven local/external expiry cleanup independently of heartbeat scheduling. Stream historical source and remote verification. Propagate worker cancellation through trace open, conditional-write probing, upload, restore and reads. Fence deletion even before the first object head exists; abort and drain owned captures. - Retry transient storage errors, distinguish 409 from failed preconditions, and backpressure bounded log admission. Track interrupted trace imports explicitly so ordinary local deletion stays independent of S3. - Automatically migrate retained history through the existing delivery coordinator. Process one log and one trace per batch. Retry unresolved history with backoff and stop scans after completion. Keep the inventory/import command as optional diagnosis and repair. Verify source and remote bytes before switching references. Verify finalized native size and digest before setting its export cursor. Require ordered source coverage when completeness metadata is missing. Preserve original trace expiry. Block active, missing, corrupt, and unproven sources. - Preserve identity and run-secret redaction in live and recovered native logs. Register runtime log redaction secrets once through the existing encrypted registry. Keep agent private keys in their existing owner. - Add storage, migration, retention, UTF-8, and route-privacy coverage, plus operating documentation and qualification tooling. - Add no schema, payload table, per-record SQL, production dependency, or lockfile change. ## Verification - Final fixes at `b3a44c898e` passed all 116 selected startup, retention, attribution and cancellation controls. The disabled-scheduler cleanup and concurrent-repair defects reproduced red before the fixes. A timed-out repair still prevents later storage reads. - The merged source passed 124 storage and real-Postgres Cloud configuration tests. These include automatic history migration, existing environment/file vault keys, source-directory removal, five real-SDK stalled-request cancellation cases, shared capability-probe cancellation and multi-chunk trace ordering. Saved Cloud S3 settings work after the legacy config file is gone. - Workspace typecheck and full build passed at the merged parent `19f702d780`. Server typecheck and build passed again at `b3a44c898e`. - Actual app acceptance: finish a process-agent run through the normal API, stage its retained local-file input, then restart. Automatic migration reports one migrated item and zero blockers. Remove the original web home and restart at `b3a44c898e` with an empty home, the existing S3 profile and the existing vault key file. The normal log API preserves all five producer lines and exactly 788 bytes, with SHA-256 `74d9ae469da28bfe89b9b50a0e132aec24fe9e6d686496ed5e4ffa6167151036`. No log-specific storage settings are present. - Actual expiry acceptance: with heartbeat scheduling disabled, a trace remains unexpired after startup and later reaches its fixture deadline. The worker marks the row deleted and tombstones both S3 channel heads. The retained run log stays byte-identical. The process fixture emits no provider protocol frames, so nonempty encrypted payloads are qualified separately through the real SDK round trips. - Isolation tests use identical run identities in two stack buckets. Cleanup in one existing prefix leaves another prefix, another company and product files intact. Read-only Cloud source inspection confirms a bucket and bucket-scoped credentials per stack. Four existing Cloud storage environment-contract tests passed. No Cloud provisioning changed and no new live vendor permission probe was performed. - Earlier qualification passed 1,014 affected storage/native tests and 27 privacy-route tests with isolated Docker Postgres. An abrupt process test commits 1,000 records, receives receipts, dies by SIGKILL and loses its directory; another process reads every byte and continues. The real Rust Runner experiment acknowledges 32 records while the export cursor is zero, then continues the same Runner PID after controller loss. All 64 records export once with exactly 64 existing control rows. Repeated snapshot/Stop commands completed in 162/81 ms. - Emulator workload: 1,000 burst records used two batch PUTs. Eight steady streams produced 240 records with 104 conditional PUTs and 72 GETs over 6.9 seconds. These are synthetic operation counts, not fleet sizing or provider prices. - Failure evidence is retained. Compact native events over 64 KiB, missing file-key support, disabled-scheduler retention and repair failure handling all reproduced before their fixes. Fixture repairs addressed cold module setup, localhost/IPC permissions, the S3 bucket-name limit and a mode-0700 crash-test directory. Full local `pnpm test:run` was stopped after host PostgreSQL semaphore exhaustion; affected suites used isolated Docker databases instead. Native package library symlinks were restored with the package's standard hydration helper. - The `19f702d780` browser session-recovery case timed out while saving a task title. All eight browser shards passed on `b3a44c898e`, including that case, with no UI behavior or assertion change. Fresh-head CI passed at `b3a44c898e`: 55 successful checks, three skipped checks and one neutral check. The branch is conflict-free. - Map and module-boundary checks passed. Current-head Greptile is 5/5 with zero unresolved review threads. Reproduce the opt-in storage suites with `PAPERCLIP_LOG_TEST_S3_ENDPOINT` and a dedicated `PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL` ending in `_test`. See `doc/run-evidence-storage.md` for operating limits, automatic migration, optional diagnostics and rollout qualification. ## Risks - Keep the existing per-stack bucket, prefix and vault key stable across replacements. Relocating that storage is a separate whole-instance operation. - This release automatically changes new log storage for existing S3-configured instances. Local-disk instances remain local. Canonical native source replay is qualified with a deterministic real Runner; actual remote execution and production-provider qualification remain rollout gates. Generic server-only diagnostics and optional raw trace tails without a surviving source can be incomplete after a crash. This does not establish full zero-downtime deployment. - Before deploying this release, qualify the actual S3 provider. It must enforce conditional writes and pass read, version, deletion, workload, and latency qualification. A capability probe rejects providers which ignore preconditions. Moto is emulator evidence only. - Dirty output is bounded and unconfirmed until S3 publication. Storage failure leaves persistence incomplete while task results and usage can settle. - Trace expiry is not yet represented by Cloud's external wake contract. Outstanding trace deadlines conservatively block sleep. - Automatic history copying requires the original source volume. Keep it until all retained resources pass verified import and clean-filesystem reads. Stop old binaries from creating new local-file runs. Missing or corrupt history remains blocked and visible in server logs. Rollback requires a compatible reader against the same external state. Ordinary object deletion does not erase retained historical object versions or backups. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, code execution, and browser verification. The runtime did not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f5e5b05a33 |
Preserve native workspace recovery across Stop and sandbox reuse (#18332)
Land the thirteen-file native recovery fix independently from #18245. Retain verified Stop evidence and queued user comments. Retire stale preparation ownership atomically and require paired sandbox lease handoff receipts before workspace recovery. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
fba0ab8162 | fix(runner): preserve GitHub helpers and Codex probe policy (#18212) | ||
|
DottaandPaperclip
|
ed8fae8dbb |
fix(workspaces): preserve completed shallow sandbox exports (#18221)
## Thinking Path > - Paperclip coordinates agent work and preserves its results. > - Remote execution copies completed workspace changes back to the host. > - An agent can fetch a published branch with shallow Git history. > - A full Git bundle omits the shallow boundary metadata and can fail host connectivity checks. > - This pull request carries the boundaries through the full-bundle fallback and imports through normal Git fetch. > - Recovery must also recognize completed native work on a reusable lease without treating it as ordinary executor reuse. > - Completed work can be saved without another provider turn, while existing host history remains connected. ## Linked Issues or Issue Description **What happened?** Sandbox finalization can fail with a missing parent commit after an agent checks out a separately fetched shallow branch. Re-exporting a full bundle repeats the failure because Git bundles do not encode shallow boundaries. **Expected behavior** Import the completed branch and saved files. Preserve existing host ancestry and concurrent host edits. Keep normal Git connectivity and workspace merge checks. **Steps to reproduce** 1. Prepare a sandbox from one host commit. 2. Fetch a different published branch with depth one and check it out. 3. Save a result file and restore the sandbox workspace. **Paperclip version or commit** Reproduced against master before this repair. The regression fails with the missing-parent error before the production change. Related public context: #15268 covers rewritten history integration. This repair handles shallow bundle transport before integration. Searches found no equivalent open repair. ## What Changed - Export shallow boundaries beside the full bundle when the delta import lacks history. - Unpack into a temporary shared bare repository. Keep parent connections already available on the host and import with Git's shallow fetch protocol. - Reject malformed boundary values and retain existing connectivity, branch, and concurrent-work merge guards. Keep boundary probes within 15 seconds without consuming the separate 60-second Git import budget. - Let source-bound preservation recover reusable leases only for the finalizer-selected, accepted successful native result and matching completion contract. Retain provider-stop, physical ownership and host-writer checks; ordinary reusable sources remain blocked. - Classify invalid Git metadata separately from omittable unsafe archives. Keep native finalization failed and preserve service-owned file recovery; carry the sender's dedicated rejection across the command boundary without transient retries. - Share the failure allowlist across bounded run/activity projections and cancelled-preservation selection. Migration `0416` rebuilds the matching partial index; task chat retains the non-retryable recovery notice. - Add real Git, filesystem sandbox and database recovery regressions. Update execution semantics and affected verification maps. ## Verification - Red: real filesystem sandbox restore failed with the same missing-parent/revision-walk error before the export repair. - Green: 240 focused Git, workspace restore, sandbox transport and native-finalization tests pass. Coverage includes both transfer paths, malformed/oversized metadata, a genuinely shallow host with its original boundary intact, and concurrent host work with both merge parents verified. - Red/green: a 14-second simulated probe left the final real Git fetch only 45,893 ms before the budget split. It now retains over 55 seconds after the probes; a 16-second probe exhausts its own budget before host import. - Red/green: an oversized read progress report was retried three times before its distinct metadata classification. It now aborts the first read before payload consumption, leaves the host tip unchanged and does not retry. - Red/green: the real metadata error was swallowed by best-effort export and native finalization incorrectly returned success. Both regressions now pass; all 240 focused Git/restore/native-finalization tests pass. A PostgreSQL/filesystem fault-injection test also verifies retained files, failed run, service-owned recovery barrier, no preservation receipt and no new wake. - Red/green: duplicate metadata produced 47 Git process starts before deduplication; the bounded-work regression now passes. Oversized byte and boundary inputs fail before Git probes. - Recovery red: both reusable lease policies exhausted recovery despite a selected accepted successful result. The assembled PostgreSQL/filesystem path now passes for retained and both reusable policies, including idempotent re-admission after exhausted automatic recovery, saved files, no agent wake, and once-only existing-result finalization. - Source verification controls reject ordinary reusable leases, rejected/failed/unselected results, another physical allocation owner, and an active host writer without changing recorded state. - The 78-test PostgreSQL/filesystem recovery/source suite passed before the metadata classification correction. Its new persisted metadata-failure regression also passes. Full workspace typecheck and build pass at final head `69f10daab2`. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Full local `pnpm test:run` attempts did not finish. They are not counted as passes. Early focused runs exceeded 5-second and 30-second fixture deadlines; the final serial suite passed all 176 tests with a 30-second per-test budget in 56.8 seconds. - Red/green: both bounded PostgreSQL projections dropped the new code in oversized results. All 19 source/projection cases now pass, plus an actual index-predicate test that excludes a metadata repair hold and still selects ordinary cancelled work. - Red/green: an oversized sender `.git/shallow` lost its error type before host validation. The real transport test now verifies one typed rejection, no shallow-payload read, and an unchanged host tip. All 465 command/sandbox/task-chat tests pass, and token/migration gates pass. - Full typecheck/build pass after these final contract corrections at `69f10daab2`; all 20 PostgreSQL source/projection/index contracts pass. All current-head CI checks passed; Greptile is 5/5 with no unresolved review threads. Merged #18230 supplies the CI-cache and Linux shell-fixture baseline repairs; their duplicate changes have been removed from this PR. The final Runner lane passed after one justified rerun of an unchanged authentication fixture timeout (all 11 fixture tests also pass locally). Previous CI failures and incomplete local runs remain recorded; live deployment and retained-run recovery remain separate verification. ## Risks - The fallback creates a temporary shared bare clone. Metadata is limited to 64 KiB and 256 unique boundaries, duplicate/parent probes are cached, and Git import has a 60-second command budget plus a separate 15-second boundary-probe budget. It runs only after a missing-history import failure. - Incorrect boundaries could hide real host ancestry. The importer retains available parent connections, and regressions cover that case. - The full-bundle fallback can transfer a bundle twice. This preserves the existing fast delta path; removing that extra transfer is outside this correctness repair. - Preservation eligibility expands only for completed native exports on reusable leases. It creates no provider turn and leaves ordinary reusable leases ineligible. Database/filesystem regressions cover result selection and competing physical/host ownership. - The migration rebuilds one partial index; no tables, columns or user records change. There are no agent execution admission, model, trust, recovery permission, workflow or lockfile changes. ## Model Used OpenAI Codex, GPT-6, with reasoning, code execution, and browser tools. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2aa3ea47e3 |
Reduce restored workspace overhead and trace cache publication (#18240)
## Thinking Path > - Paperclip runs agents in isolated workspaces. > - A native sandbox turn must restore its workspace before the provider starts. > - Restored Git metadata can cause expensive work in the provider's first snapshot. > - Cache replay also hashes the same archives twice, and workspace helpers open sessions that do not need output. > - This pull request removes that repeated work and refreshes the restored index before OpenCode starts. > - It also adds cache-publication timings so the next measurement can separate quota checks, lock waiting, and archive IO. ## Linked Issues or Issue Description **What existing behavior does this improve?** Native sandbox workspace preparation and copyback. **Current behavior** Cache replay validates shared archives, copies them, and validates them again. Exit-status-only helpers use persistent sessions. A restored checkout can make OpenCode's first snapshot recheck much of the repository. Cache publication has an unexplained interval after archive verification. **Proposed behavior** Validate the private archive copy that staging will consume. Use the existing one-shot command path for helpers that only need an exit status. Refresh the Git index after workspace transfer for a new native OpenCode sandbox. Keep the refresh inside the awaited inbound barrier and use the existing command timeout. Add nested publication spans. **Reason and benefit** Two matched local component pairs reduced preparation plus provider time from 8.563/8.553 seconds to 7.313/7.511 seconds, including the refresh cost. Two alternating remote copyback pairs reduced the median from 9.110 to 7.943 seconds. These are component results. They do not establish an end-to-end hosted improvement. Related: #18181 and the earlier closed #18195. This implementation uses the existing fully awaited transfer boundary. It does not use an optional preparation deadline. ## What Changed - Materialize cache hits into caller-owned staging and validate those exact bytes once. Remove incomplete copies on failure. - Select the existing one-shot path for exit-status-only run and removal helpers. Output-reading helpers keep their existing behavior. - Refresh restored Git stat metadata before OpenCode starts. Enable this only for newly acquired native OpenCode host-current workspaces. Skip recovery and retained sandboxes. Keep dirty files unstaged, accept Git's dirty-worktree status, and propagate real command errors. - Trace cache capacity checks, eviction, the body after lock acquisition, archive copy, sync, and hashing. Keep the operator endpoint gate and bounded phase names. ## Verification - On the rebased branch: 202 focused tests pass: 152 workspace seed/cache/runtime tests, 27 managed command tests, and 23 native workspace/tracing tests. Adapter TypeScript build and server typecheck pass. - Tests cover corrupted copies, cache changes after private materialization, dirty working trees, quoted paths, recovery, the awaited refresh barrier, real command failure, and company-lock contention. - A real-filesystem probe through the server phase mapper verified parentage and endpoint-off behavior. A controlled lock wait stayed outside the locked-publication span. No external export was used. - A full local Core/Daytona replay on the pre-rebase implementation saved its synthetic reply. It retained the pinned large public workspace, all 209 skill files, positive Git context, and 66 tool schemas. Both model calls stayed on sandbox loopback. Index refresh took 0.644 seconds and the first provider snapshot took 0.499 seconds. The run took 162.004 seconds, including large laptop uploads; it is not a hosted acceptance result or a matched end-to-end comparison. The failed installer attempt remains recorded separately. Normal cleanup removed all owned sandboxes. - The current head passed 53 CI checks, with two optional skips. Greptile is 5/5 on the exact head, with no open review threads. No deployment or hosted latency claim is included. ## Risks - Index refresh adds one awaited command. It uses the existing timeout. A real Git failure stops preparation. - Cache hits need temporary disk space for a private copy. Existing corruption fallback and per-run recovery archives remain. - One-shot command transport must preserve exit status and timeout behavior across providers. The agent's native session and output-reading helpers are unchanged. - Local component gains are small relative to the remaining hosted latency. Batch the fixes with the diagnostic spans; measure the actual deployed path before claiming success. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The session does not expose the exact model SKU or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
61bd4ac691 |
feat(agents): set up Antigravity with saved Google subscriptions (#18189)
## Thinking Path > - Paperclip manages agents and their work. > - Users choose providers and managed accounts while adding agents. > - Antigravity was missing from the ordinary additional-agent picker and its saved Google path did not complete native readiness. > - A user needs a successful Test, an enabled task action, and a stable account binding. > - This pull request connects the ordinary setup flow to the native candidate. > - It also preserves retired credential refreshes across database failures and server restarts. ## Linked Issues or Issue Description Built on merged #18188 and #18187. Refs: #12503, #14938 for related legacy CLI proposals. **Agent or provider** Google Antigravity ACP 1.4.3. **Why this adapter is useful** Users need to set up an Antigravity agent and run real work through their managed Google subscription. **How the agent is invoked** Native Paperclip Runner uses the pinned Antigravity ACP runtime. Exact operator admission remains required; the local candidate result does not enable general release. ## What Changed - Show Google Antigravity in the ordinary adapter grid and preserve the draft choice when setup resumes. - Reuse the same user’s saved personal default Google subscription and discover the exact low model through the native account. - Run a prompt-free native Test and wait for authoritative readiness before enabling task assignment. - Carry the managed account through native runs and recover retired refreshes through the existing encrypted, fsynced journal with exact generation checks. - Retain the latest rotated credential through verification, storage, and partial home-removal failures. Retry the exact account’s retained cleanup before another provider use. - Reconcile cleanup before taking the credential mutation lock, fence owners arriving during lock acquisition, and support nested lock holders without a join cycle. - Bound sign-in retirement outside database transactions and preserve claims for the existing cleanup owner; one pending sign-in does not block unrelated accounts. - Cover Gemini and Pi API-key recovery, keep API-key bulk manual-model editing usable, and keep adapter discovery available after candidate expiry. - Skip taskless native timer work before run or credential admission. Preserve current master monitor and deadline scheduling. ## Verification - The registered local saved-Google-subscription journey passed all 16 matchers in 196,520 ms using eval source `12aba` against actual serving source `4a37ee7c`. Both turns stayed in one conversation, produced exact downloaded files of 61 and 119 bytes after reload, and used distinct native primary processes. All observed owners retired, leases released, and credential fences settled. Independent readback confirmed Done work, no active owned runs, a connected saved account, and the retained agent paused for inspection. - The measured runtime is production-equivalent to pre-restack `65ef31dc` except its test-only transport fixture correction. Current-master head `ff7c9249` preserves all 99 product paths: 98 exact blobs and only eight inherited plugin-panel lines in `agents.ts`. Its native binary was rebuilt for inherited Codex-only diagnostic changes; the changed Rust behavior is outside Antigravity's ACPX execution path. Shared native/UI changes have focused regression coverage. The recorded live result is not represented as a measured replay of `ff7c9249`. - Full workspace build and typecheck passed on `ff7c9249`. All 376 focused tests in 14 files passed, covering Google credential/retirement controls and the inherited native-tool, redaction, route, and UI recovery changes. Token gates, maps, and diff checks passed. The required-close fixture was separately demonstrated red, then passed all 317 selected provider-wiring cases. - [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/40959387972) is green: 55 successful checks, two skipped workflow checks, and an optional neutral review check. Required `ci / verify` and `ci / e2e` passed. Greptile and Paperclip Review both report 5/5 on exact `ff7c9249`, with zero unresolved review threads. - This qualifies the selected operator-authorized local saved-subscription candidate. The original exploratory journey remains historical evidence; the repeatable fixed-driver eval is in the next PR. ## Risks - Google refresh recovery must preserve the exact grant, account, and credential generation. An uncertain native owner or failed encrypted handoff retains the private home and blocks cleanup. - A successful local saved-subscription candidate does not qualify fresh OAuth, API keys, Cloud, Linux, other models, or general release. - Timer filtering changes only taskless native Runner admissions. Existing task-bound monitors and legacy adapters retain their owners. ## Model Used OpenAI Codex (GPT-6). The host did not expose the exact model revision or context limit. The model used tools for code edits, tests, and review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f1911a4c53 |
perf(skills): move repository packages out of Postgres (#18082)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/205/changes/paperclip/205 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents receive selected company skills before each turn. > - Imported skills can need files from the whole source repository. > - The current path stores those files in Postgres and reads large JSON payloads during preparation. > - This pull request stores verified packages in the configured file or object store and keeps preparation queries small. > - It also converts old rows automatically and restores missing files from the exact saved commit. > - The result reduces database traffic while preserving skill versions and repository dependencies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Company skill import, version storage, and preparation before agent turns. **Current behavior** Repository snapshot bodies live in JSONB. Skill preparation can read those bodies again even when the files already exist on disk. Historical packages and native assets also need a bounded cleanup path. **Proposed behavior** Store package archives and manifests through the existing storage service. Keep version identities, receipts, hashes, and recovery metadata in Postgres. Convert legacy rows after the stored package passes read-back verification. A missing package can use legacy bytes or its exact Git commit under current repository authorization. No separate data-migration command is required. **Reason and benefit** The warm path sends no repository bodies through SQL and reads no object bytes. Cleanup runs through the existing deadline coordinator. Current versions, active runs, explicit pins, and authored content retain their protection. **Breaking changes** The normal application schema upgrade is required. Single-version and skill-file responses keep their existing shape. Version lists, current-version summaries, and test-run version summaries now contain file names and kinds without file bodies. Diff and restore fetch selected versions through the existing single-version endpoint. Older server builds must not write after conversion unless they support the new package format. The storage override and rollback limits are documented in `doc/skill-package-storage.md`. **Additional context** Related work: #15731 preserves repository dependencies; #17923 changes version-cache scheduling; #18080 changes delivery to remote Codex homes. This change owns package persistence, acquisition, and expiry. It does not replace those delivery or scheduler changes. ## What Changed - Add company-scoped archive receipts, exact-commit recovery recipes, migration deadlines, and a durable object-deletion queue. - Add bounded archive upload, read-back verification, extraction, concurrent request coalescing, and verified local caches. - Convert old snapshots and versions automatically. Keep legacy bytes when verification fails. - Recover missing files through current source authorization and the original commit. Validate repository identity, file hashes, modes, and the aggregate hash. - Expire unused recoverable Git packages and native assets after 30 days. Preserve active pins and the durable copy of authored or unproven content. - Keep warm preparation queries metadata-only. Bound file digest caches and avoid repeated body hashing when file identities are unchanged. - Extend the existing storage contract with cancellation. Register package work with the existing coordinator. Optional conversion retries retain the original bytes and may wait for ingress without blocking sleep. - Add regression tests, operating guidance, a deep trace, benchmark receipts, and an optional maintenance command. ## Verification ### Final master integration and review repair — October 29, 2026 Current head: `a6065424249e763dfeac37161f1b52a9f25bd9b7`. Commit `f8bbdd3dbd` resolves the conflicts with master `76a796e964`; the final two commits repair the Apex deletion findings below. - Keep both package-maintenance and version-retention notifications in their write transactions. - Keep the shared run-pin predicate and master's controller-lease deadline metadata. Package expiry reads the updated protected-version result. - Preserve every master migration entry. Generate migration **0415 (`0415_zippy_zemo.sql`)** after master's 0414. Its skill SQL is byte-for-byte unchanged from the previously tested idempotent migration. Retain only the five newest schema snapshots. Migration numbering and safety checks pass. - Repair failed file/folder deletion: preserve a complete temporary subtree until transaction commit, restore on upload or SQL failure, and restore callback failures before releasing the mutation lock. Both PostgreSQL regressions reproduced missing files before the fix and now pass. They cover binary data, executable permissions, empty folders, unlisted files, unchanged database/version rows, successful retry after upload-claim expiry, and historical reads. - Reconcile lost commit replies under the same advisory and row locks. Check the saved version and source identity before restoring. Keep committed deletions intact and preserve backups when the database outcome is unknown or a newer edit exists. Four real-PostgreSQL outcome tests pass; the prior code failed three of them. - Full workspace typecheck and production build pass on the final commit. All **204** focused PostgreSQL package/cache/retention/authoring and filesystem-boundary tests pass, with none skipped. - Apex review **18 passes 5/5**, with zero new comments on the exact final head. All **34** review conversations are resolved. - [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/40914116508) completes successfully on attempt 2. The head has **55 successful checks**, two neutral reports, two skipped checks, and a successful Snyk status. No checks are pending or failing. All eight browser shards pass. - Runner Vitest 2/2 timed out once in the unchanged ACPX controller-replacement fixture. The same job passed on the preceding commit. One unchanged CI retry passes all five ACPX recovery cases, **2,253** Runner package cases and **126** native server integration cases; their standard one and five skips remain. The initial timeout's root cause is not established. Local diagnosis also hit the fixture's eight-second session.open timeout with an authenticated Runner; no local Runner pass is claimed. - Both full local `pnpm test:run` attempts were stopped when Apex identified deletion defects and source repair became necessary; neither is claimed as a passing full local run. The final **204** focused tests, full workspace typecheck, and production build pass. Full-suite acceptance uses the final-head CI jobs. The superseded merge-head CI included runners terminated by shutdown signals; each source repair started a new full run. - Auto-merge is enabled for squash merge. The final head is conflict-free and all checks pass. GitHub's active master ruleset still requires a code-owner approval for the dependency files. The existing review requests are with `devinfoley`, `nickyleach`, and `forgottendev`; no policy bypass was used. - The earlier hosted stress and full native receipts below remain tied to `a2e597d800`; they are not relabeled as a deployment of this final integration. The earlier local PostgreSQL startup failure remains recorded below; PostgreSQL ran successfully for this integration verification. ### Full native skill execution — October 29, 2026 The full native-agent acceptance gap is closed on candidate `a2e597d800eb989d565780e7aa947d5368676444`. Two qualifying native turns used OpenAI API / `gpt-5.4-mini` in real Daytona sandboxes, on the same disposable hosted TEST tenant as the stress test. A follow-up on the same task consumed a newly saved skill version. The task text asked for the assigned skill workflow. It did not contain the proof nonce or expected receipt. The assigned authored skill contained a Python verifier and a random package-local nonce. The agent read the actual installed skill files and executed that verifier against the imported, automatically converted 1,360-file repository package. It checked the repository file count and reference/binary bytes, and directly executed the repository's executable probe. Independent verification matched the emitted JSON to the nonce and expected byte hashes. The agent persisted the same receipt as its task response; native finalization committed, workspace finalization succeeded, and task state became Done. | Qualifying turn | End-to-end run time | Recorded skills.prepare | Repository files | Binary hex | Shell receipt SHA-256 | |---|---:|---:|---:|---|---| | Initial task | 47.881 s | 1822 ms | 1360 | `00ff07800a` | `a1deb30dc095f2306c7aa9cc5ec396da12c68961cf5b809f4d7c20f7fc68493b` | | Corrected follow-up, new nonce/version | 37.974 s | 377 ms | 1360 | `00ff07800a` | `dbcdc33fa1376b4edc136feae1cdf770560a73292467e93619efdda20e4282a7` | Both successful shell receipts reported `probe: STRESS18082` and `assetSha256: e8ea7a2f15f7bb3b332a46adbef848465884c8996595aa234748befd544d1f3a`. Their proof nonces, selected version IDs and receipt hashes differed, and both ran on the same task. These are observed timings, not latency SLOs; they include real provider work. The service stress-test timings remain a separate measurement boundary. Earlier failures remain in the evidence: - The original five-minute Daytona allocation timeout never reached model execution. After normal owned-stack restore, its durable cleanup record was `expired` / `success`. The cause of that original allocation timeout remains unknown; a later success establishes recovery only. - Initial continuation setup used unsupported generic agent-update fields and returned HTTP 409 before dispatch. Setup was corrected to use the supported skill-assignment and versioned instruction commands. - The intervening completed native turn read repository bytes and ran the probe, but omitted the verifier JSON under the older test agent's overly-specific instruction file. Its strict receipt validator failed; it is not counted as a qualifying turn. Correcting the test agent's instruction file and sending an explicit task-scoped wake produced the qualifying follow-up above. - The pinned application's immutable rate card omits `gpt-5.4-mini`, so completed usage triggered the normal block-on-unpriced budget policy. Supported append-only accounting corrections estimated the three completed runs using the [official OpenAI model rate card](https://developers.openai.com/api/docs/models/gpt-5.4-mini): $0.75 input, $0.075 cached input, $4.50 output per million tokens, assuming standard processing. Estimates totaled 12.482655 cents. Original usage and missing-price records are preserved; the $1 hard stop and unpriced-usage guard stayed active. - Model shell attempts included a missing `xxd` command and incorrect guessed file paths. Those failed calls and subsequent successful actual verifier calls remain visible in complete event streams. No zero-tool-error claim is made. No product source changed during this continuation. The previously measured contention, cold-start and memory limits remain. The later merge resolution is covered by the separate integration checks above; these hosted receipts qualify their recorded candidate. ### Hosted stress test — October 29, 2026 **Hosted acceptance:** full native-agent execution and a changed-skill follow-up pass on the recorded hosted candidate. The final integration section above records the subsequent conflict resolution and its separate checks. Tested baseline `929e840595c8933ca485ac7a2e99413e1e11ca64` and this exact candidate sequentially on one disposable TEST tenant, with external Neon PostgreSQL and actual S3. The fixture contained **1,360 files / 22.4 MB / ten skills sharing one repository**. Measurements cover deployed skill-preparation and native-context services in an adjacent diagnostic process, not a complete agent turn. The container had **24 vCPU / 24 GB**. | Warm workload | Median before → after | p95 before → after | DB returned value bytes/preparation before → after | |---|---:|---:|---:| | One skill, serial | 1,060 → 125 ms | 1,303 → 155 ms | 22,610,389 → 1,771 | | One skill, four concurrent | 4,415 → 297 ms | 5,143 → 328 ms | 22,610,389 → 1,771 | | Ten skills, serial | 5,723 → 535 ms | 6,057 → 600 ms | 224,739,030 → 10,671 | | Ten skills, four concurrent | 24,410 → 1,986 ms | 24,833 → 2,221 ms | 224,739,030 → 10,671 | - Candidate warm preparation performed **zero S3 GETs**. Ordinary-pool queries fell from 16 to 9 for one skill and 61 to 9 for ten, excluding dedicated lock connections. DB numbers measure returned values before Drizzle decoding, not wire bytes. Baseline/candidate sample counts were 30/100, 32/100, 5/50 and 8/48 respectively; the small baseline ten-skill samples are comparative evidence, not SLO estimates. - **Application-UID soak:** 800 ten-skill preparations over 10.3 minutes, four workers paced by 2.5 seconds; **zero failures**, median **559 ms**, p95 **775 ms**, p99 **910 ms**, zero warm S3 reads. All 726 accompanying health/company-list requests returned HTTP 200. Their end-to-end p95 was 569/502 ms, including network latency. The probe overlaps most of the soak. Soak DB-response capture was limited to its first 60 seconds. - **Important remaining costs:** an unpaced 16-way ten-skill burst had p95 **8.32 seconds** because company-level preparation serializes. Cold preparation took approximately **1.1–2.5 seconds** for one skill and **7.4–10.0 seconds** for ten. Ten-skill diagnostic RSS remained about **1.17 GiB** (baseline 1.18 GiB), and the soak peaked at **1.35 GiB**. A concurrent whole-container sample was 3.45 GB. This does not establish suitability for a small-memory host or eliminate per-entrypoint native asset duplication. - **Upgrade without a manual data migration passed:** legacy snapshot JSON shrank from 22,465,691 bytes to `[]`, with a stored package and zero pending conversions. - **Recovery passed:** sixteen concurrent cold requests coalesced into two S3 GETs; full bytes/binary/executable modes matched; same-size corruption repaired; an injected storage failure preserved warm access and failed cold access safely. A disabled public-Git source denied recovery, and after the actual five-minute backoff its exact SHA reconstructed all 229 files. Cross-company descriptor access was denied. Private-Git credential revocation was not tested. - **Retention passed:** reclaimed 27,754,496 allocated cache bytes, deleted an unreferenced package row and both objects, preserved aged authored history, and expired/reacquired reconstructible Git content. A version pinned by the failed native run was correctly retained. Only owned fixture timestamps were aged. - **Browser UI passed:** authored skill creation, saved edit surviving reload, version history, and restoration of earlier content. - **Initial native attempt (superseded by the qualifying turns above):** one real run failed at Daytona allocation after five minutes (`setup_failed`), before any model execution. Provider cleanup also reported failure. That attempt does not count as a passing native turn. Its receipt and all evidence remain preserved; its cleanup later recorded success after normal restoration. - Initial A/B commands ran as SSH root. Diagnostic-owned lock/cache permissions caused one setup HTTP 500 and the first UID-1000 soak to fail; exact fixture paths were repaired. The final soak, retention and recovery ran as application UID 1000. A fresh UID-1000 one-skill batch also passed (30 samples, median 116 ms, p95 321 ms, 1,771 DB value bytes/preparation). Earlier harness-export, SSH, npm provenance, backoff and UI-wait failures remain recorded alongside their successful retries. - Private evidence includes raw timings, assertions, HTTP probes, screenshots and failure/cleanup receipts. Initial failed-campaign cleanup suspended and retained the owned TEST tenant and released the shared QA account lease at **15:22:41 UTC**. The tenant was subsequently resumed under its existing ownership for the native execution qualification above. Final cleanup completed at **15:53:09 UTC**: the account lease is released and the tenant is suspended/retained under the original October 30, 12:24:52 UTC retention deadline. Its separate stack lease remains held; no active or pending-cleanup environment lease remained. The original retention window is deliberately preserved even after successful continuation. - Earlier hosted-tested head: `a2e597d800eb989d565780e7aa947d5368676444`, integrated with master `929e840595`. The earlier merge conflicts were resolved at this head. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/40884087629) passed; all checks are complete with 55 successes, two neutral reports, two skips, and no failures. [Apex review 15](https://github.com/paperclipai/paperclip/pull/18082#issuecomment-6480257258) passed 5/5 with zero findings on this exact head. All review conversations are resolved. - Ten real-filesystem boundary cases and server typecheck pass locally for the security fix. Coverage includes cache roots, descendants, individual files, symlink aliases, recursive ancestors, a future concurrent cache fill, and allowed sibling directories. [Current-head private-project integration tests](https://github.com/paperclipai/paperclip/actions/runs/40884087629/job/123483306405) passed all 29 cases, including authorized and unauthorized imports of a cached private package. Full typecheck, build, server suites, all eight browser shards, and Runner checks passed in current-head CI. Local database tests could not run because PostgreSQL could not initialize; this limitation is recorded below. - The earlier generated migration preserved the newest five snapshots. The final integration renumbers its unchanged SQL to 0415. Migration safety/numbering checks pass. Its SQL is byte-for-byte identical to the previously verified idempotent skill migration; repeated application on this latest schema was blocked by the same local database startup failure. - The broader verification before the final migration-only rebase passed 1,025 package, lifecycle, remote-launch, native-session, and asset-retention tests, 52 focused UI tests, full workspace typecheck/build, token gates, and map checks. The linked complete CI run passed before the latest focused fixes. - A disposable MinIO instance passed multipart upload, verified read-back, and cold restoration of an 18 MB archive. - Twenty warm samples with a synthetic 1,349-file, 22.8 MB repository retained five metadata queries and zero object reads. Median preparation was 256 ms for one selected skill and 691 ms for ten, with 5 ms emulated network delay each way. These are local measurements, not production latency guarantees. - No production instance was changed. Earlier verification failures and their disposition are retained below. <details> <summary>Earlier failures, review findings, fixes, and rerun evidence</summary> - The first CI pass found a stale inventory-span expectation and a missing repository-read span. Both are fixed; all three observability tests pass. One browser test copied page text into its YAML parser after reload; its first unchanged-head retry passed, but the race recurred after rebasing. The test now waits for the renamed URL and completed reload before copying the editor, retaining its original assertions. The earlier full local suite did not complete cleanly: it reported setup timeouts and assertions, while isolated relevant reruns passed. The complete current-head CI run closes that broad verification gap. No production instance was changed. - The first Apex review scored 2/5 and identified six issues. The fixes protect authored history even when it shares an in-flight Git upload, recheck upload expiry during orphan deletion, bound failed-upload cleanup, retain per-version kind/encoding metadata, add sanitized failure diagnostics, and correct the migration guide. The final focused run passes all 140 tests; the preceding eight-file run passed all 149 tests, including local storage and S3 diagnostics. Server typecheck and migration safety/numbering checks pass after the fixes. The second Apex review scored 1/5 and found unbounded maintenance storage calls and mismatched package/Git batch limits. Those are fixed with a 30-second cancellable cleanup deadline, shared 100 MiB/10,000-file scanner and storage limits, and bounded Git read batches. The proposed manual lockfile edit was rejected with evidence of the enforced CI-owned lockfile policy. All 19 storage/recovery unit tests and server typechecking pass. The new database-backed cleanup/conversion tests were locally skipped because the host exhausted its PostgreSQL semaphore capacity (confirmed directly by initdb). That gap is closed by current-head CI: [package-store tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109300889) pass all 18 cases and [package-worker tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109301157) pass all 7, with no skips in either suite. The corrected YAML browser shard passes. A separate planning-fixture test failed its cleanup DELETE with an activity-log foreign key violation after its behavior assertions passed; its one unchanged-head retry passed. The third Apex review scored 1/5 and identified a one-slot pool deadlock, projected path limits, cleanup holding the preparation lock during archive reads, and a publication/expiry race. These are fixed with transaction-local reads, prefix-aware bounds, isolated verification, and shared package row locks with a publication grace. The separate Paperclip review also found stale historical inspection, eager history downloads, and conversion retries blocking sleep. Those are fixed. Local PostgreSQL capacity recovered: the new database regressions ran, including the one-slot fork, paused storage during a warm turn, and a locked publication during expiry. The expanded run passed 144/145 tests; its remaining assertion still expected file bodies from the version list and was updated to fetch the selected version. The corrected company-skill suite passes all 81 tests, and the final package/projection run passes all 27. All 52 focused UI tests pass after updating their query-provider fixtures; the first UI run exposed a fixture remount, and the follow-up exposed an editing syntax error, both corrected. Full workspace typecheck and build, token gates, and map checks pass. The sandbox initially blocked the build tool’s local IPC socket; the authorized build rerun passed. Master then claimed migration 0401. The branch is rebased on |
||
|
Nicky LeachandPaperclip
|
247de89eae |
fix(runner): preserve warm tasks during recovery (#18169)
## Thinking Path > - Paperclip manages agent work and the lifetime of its execution resources. > - A warm native task can keep one Runner alive between successful turns. > - Foreground completion selects whether that resource stays running or stops. > - Background recovery must use the same selected policy. > - Recovery currently reads that policy only from workspace copy-back records. > - A task without such a record can be stopped immediately after foreground retention. > - This pull request saves the resource policy with the native runtime and permits the retained task to refresh its control connection. ## Linked Issues or Issue Description **What happened?** Background native finalization can stop a successful warm Runner when its run has no Core workspace-sync record. Foreground completion retains that same task. The stopped Runner cannot receive the later drain and suspend commands. Cold resume then correctly rejects its unproven checkpoint. Hosted diagnostics confirmed the recovery reconciler attempted environment release immediately after foreground retention. The provider connection returned HTTP 404 within the same second. A database-backed regression reproduced the stop without a workspace-sync record. A separate authorization gap also prevents the current retained task from refreshing its connection endpoint after the run succeeds. **Expected behavior** Foreground completion and recovery must use the selected resource policy. A successful warm task stays running until its idle shutdown can suspend it. Failed or cancelled work still stops. The current retained task can refresh its existing control route, but cannot prepare or submit new execution. **Steps to reproduce** 1. Select a warm native runtime on a reusable remote execution environment. 2. Complete a turn without a Core workspace-sync record. 3. Let native finalization recovery settle the still-active run lease before foreground retention finishes. 4. Before this change, recovery defaults to `stop_and_retain` and invokes the provider stop path. 5. Separately, request the retained task's connection endpoint after successful completion. Before this change, Core rejects that request. **Paperclip version/commit** The hosted failure used Core |
||
|
DottaandPaperclip
|
6f278dd6d7 |
fix(runtime): settle recovery lease and connection lifecycles (#18167)
## Thinking Path > - Paperclip manages AI agents and their work. > - A remote runner can survive a controller restart. > - Recovery renews its execution lease during slow preparation. > - The native executor then shares that renewal timer. > - Preparation kept its reference until all finalization finished. A tick after lease release treated completion as ownership loss and detached the retained runner. > - A Linux continuity test also found that buffered wire events could arrive before their connection was registered. Cleanup then kept a completed promise under an undefined key and never finished. > - This pull request transfers renewal ownership after execution acquires its reference. It also registers each connection before wire callbacks can run. > - Recovered turns can complete, retain the warm runner, and settle connection cleanup. ## Linked Issues or Issue Description **What happened?** A recovered remote run could complete, release its coordinator lease, then detach during instruction collection. The next turn could find the old runner still using the ingress port. Recovery then stopped because it could not prove process ownership. **Expected behavior** Preparation keeps the lease alive until execution owns renewal. Execution stops renewal before releasing the lease. Actual ownership loss still detaches the runner. Buffered wire callbacks must use the registered connection, and cleanup must finish. **Steps to reproduce** 1. Recover an in-flight remote runner after a controller restart. 2. Let the recovered turn complete. 3. Hold instruction collection open through the next lease renewal tick. 4. Send a follow-up that needs the retained runner. Related: #17439 introduced remote controller recovery. #18011 preserves checkpoint authority through warm preparation. This change fixes the lifetime of preparation's renewal reference. ## What Changed - Pass an internal lease-acquired callback from recovery through heartbeat dispatch. - Release preparation's renewal reference after the executor registers its own. Keep final cleanup for exits before dispatch. - Add a PostgreSQL regression for completion while finalization is still pending. Cover callback forwarding, acquire-before-release ordering, and failure before dispatch. - Register the authority connection before subscribing to wire events. Handle an immediate close before accepting buffered data. - Add regressions for a message or close delivered during subscription. - Extend Linux continuity coverage with preparation renewal and delayed instruction collection after recovery. - Update the continuity guide and session verification recipes. ## Verification - The new recovery regression failed before the fix because renewal detached the runner after lease release. The 76 recovery, renewal, and dispatch tests then passed. The final ordering and early-exit checks passed again after replacing fixed sleeps with controlled interval ticks. - Both buffered-wire regressions failed before the connection fix. All 170 control-plane tests passed afterward. Existing tests cover stale authority, ownership loss, pending operations, and closed ingress. - Linux ARM64 acceptance passed at `a7daefacceb8bd994d0b79d284cdf54f3e3829d1`, with Node 24.19.0, PostgreSQL, the real Rust runner, TLS ingress, and a deterministic Codex protocol fixture. The test interrupted the first turn, recovered it, held finalization open for a renewal interval, and completed two follow-ups. It retained one provider process, three turns in one history, and the same runner identity. Final suspension and connection cleanup completed. The ordinary warm-reuse case also passed three turns with one provider process. - Earlier Linux attempts completed the turns but hung during cleanup. A debugger found a fulfilled promise keyed by an undefined connection. The regression reproduces that defect. One later setup attempt timed out while staging an unstripped 373 MB debug binary. The passing run used the same binary with debug symbols removed. - Reproduce Linux acceptance with the `same-server` case in `doc/testing-native-remote-continuity.md`. - `pnpm -r typecheck`, `pnpm build`, map validation, and `git diff --check` passed at the final revision. The full local `pnpm test:run` command was attempted, but the serial run had not finished after 13 minutes. It was stopped after the equivalent CI lanes passed. It is not counted as a full local pass. - Greptile and the Paperclip review both scored the final revision 5/5 with no findings. All Runner verification lanes passed. The first CI attempt hit timing failures in voice recovery, agent YAML configuration, and the task-title outage browser test. The full YAML and voice-recovery files passed locally (88 and 19 tests). These paths were unchanged and passed on the prior revision. All three passed on the single retry. Final current-head status: 55 successful checks, one neutral check, and two skipped Storybook checks. No pending or failed checks remain. - Live Daytona/model acceptance and recovery of the hosted task remain unverified. This PR does not deploy the change. ## Risks - Releasing preparation before execution starts renewal could leave a gap in authority. The callback runs after the executor registers its reference. - Preparation errors must not leak renewal. The recovery promise retains idempotent final cleanup. - Wire callbacks can run during subscription. Message and close regressions check that registration is complete before either callback runs. - No schema, API, model, permission, lease duration, or retry policy changes. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and browser diagnostics. The exact serving model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
ba07f85cad |
perf: schedule agent timers from durable deadlines (#17876)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/58/changes/paperclip/58 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent timers must start due work and preserve queue admission rules. > - The server woke their scan on every heartbeat interval, even with no eligible timer. > - Idle scans keep the database and instance active. > - This pull request derives the next deadline from saved policy and work records. > - Supported producer transactions notify the shared coordinator when eligibility changes. > - An idle timer worker stops scanning and resumes when real work or a deadline requires it. ## Linked Issues or Issue Description Refs #17863. This is slice 63 of the recurring-work migration and follows the merged issue-monitor deadline PR. Related: #16632 changes timer issue scope; #16881 changes live-run eligibility. This PR preserves the current admission policy. Those policy changes are separate work. A future time-sensitive predicate must expose its next transition deadline. **What existing behavior does this improve?** Agent timer heartbeat scheduling in the server. **Current behavior** A recurring heartbeat wakes the timer scan even when companies are paused, timers are disabled, or no actionable task exists. **Proposed behavior** Register the earliest eligible interval or positive daily-cap reset deadline. Re-read on supported producer commits and startup. Leave no recurring database scan when no timer obligation remains. ## What Changed - Keep policy and deadline calculation in the heartbeat queue. Scheduling retains company, reporting-chain, worktree, and CAS gates. - Use the existing deadline coordinator. Remove only the timer wake from the server heartbeat loop. - Notify before existing transaction writes that change eligibility or deadlines. Cover configuration, lifecycle, company budgets, task ownership, recovery, conversations, accounting corrections, and completion. - Use the lifecycle status-change hook for saved hire rejection. Cover checkout's second adoption transaction after an initial failed CAS. - Preserve pending work through idle and maintenance drains. Check cancellation between due agents. Reconstruct on restart. - Document the supported writer model and producer inventory. Add no schema, table, or provider branch. ## Verification - **1,710 passing cases across 49 complete files**, no skips, on accepted source `32db86c1974dcbf794f01a8c5e42024e2ad071fc`. Coverage includes real timer policy/CAS/producers, monitor and shared deadline consumers, native reconstruction, Slack lifecycle, skills/settings/configuration, queue isolation and agent lifecycle. - Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. Twenty-nine fresh PostgreSQL 17 schemas plus per-file fixtures applied all 412 journal entries through index 413; all selected source hashes and temporary fixture imports were restored exactly. - Four new deadline composition cases verify future/overdue timing, minimum scan spacing and disarming. A controlled mutation dropping only the inherited continuation option fails; the composed source passes with identical assertions and timers. - Normal parent reconciliation preserves task-drain expiry and both deadline contracts. The final monitor-parent publication changes only its audit, preserving every qualified runtime and selected test source. - Historical failures and review corrections remain in [the timer audit](doc/idle-agent-timers.md). No full local repository-suite, live provider, measured suspend/wake, external-alarm or fleet-savings claim. Final-head CI/security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - Process-local notifications cover supported service/API writers in one server process. Direct SQL and independent replicas need an explicit ingress contract. Startup reconstruction is not live cross-process notification. - Notifications can batch for up to the configured heartbeat interval. Known deadlines retain their scheduler path; failed reconciliation retains bounded retry. - Enabled timers without an actionable-work requirement remain real recurring obligations. Zero caps require policy changes; positive reached caps use UTC midnight. - The timer fixture substitutes adapter enqueue. Existing admission tests also pass, but this is not deployed provider or sleep qualification. - Cloud does not gain legacy agent timer alarms in this PR. Existing sleep-safety blockers remain. No database/host suspension or fleet savings claim is made. - Parent-stack CI and review must pass before merge. This PR does not authorize merging its dependencies. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, normal Git merges, composition proofs, a controlled red/green regression and complete PostgreSQL integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
DottaandPaperclip
|
b81b870732 |
Add QA smoke journeys and OpenCode run traces (#18098)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E journeys verify saved application behavior across deployments. > - Task creation and questions need reusable smoke selections and exact run attribution. > - Operators also need to inspect the OpenCode execution behind a measured question. > - Company observability does not currently support native OpenCode. > - This change adds the journeys and a bounded projection of normalized OpenCode run events. > - The projection follows the configured capture policy and retains the current run identity across warm reuse. ## Linked Issues or Issue Description Related foundation: #17262. Related alternative: #17989 exports metadata from OpenCode's native OTLP stream. This change uses normalized run events to support the existing content switches and per-run linkage. It does not forward the native stream. The two PRs overlap in launch registration and must be reconciled before both land. **What existing behavior does this improve?** Reusable Product E2E smoke selections and company observability for native OpenCode. **Current behavior** The journey catalog lacks reusable task creation and attached-agent question checks. Native OpenCode is marked unsupported for company observability, so a successful destination connection test does not prove that agent traces will appear. **Proposed behavior** Add explicit task-create and ask-question selections. Verify saved state and exact company, issue, agent, model, environment and successful run attribution. Export normalized OpenCode turns, tools and usage to the configured OTLP destination. Honor each capture switch and redact known secrets. **Reason and benefit** CI can run a small measured smoke selection, retain its screenshots and checks, and link results to traces belonging to the same run. **Breaking changes** Existing selectors remain unchanged. OpenCode export is opt-in through existing company settings. Legacy OpenCode stays unsupported. ## What Changed - Add reusable task creation and question journeys, configured-agent capability, owned-fixture checkpoints and strict evidence validation. - Add an OpenCode-owned projection of normalized run events. No provider SDK or raw payload schema enters heartbeat coordination. - Extend the private launch union and reserved environment allowlist for native OpenCode. With export disabled, no receiver request runs. - Keep destination headers in the parent wrapper and out of the actual provider child environment and configuration. - Bound each turn to 64 tool and 64 usage entries, 8 KiB per text field and a 1 MiB request. Use a three-second request deadline, four concurrent exports, and no retries or idle worker. - Pass the rotated runner authority to each proxy turn. A retained process no longer attributes later turns to its original launch run. - Update settings copy, observability documentation and capability maps. ## Verification - `pnpm -r typecheck` and `pnpm build` passed after the implementation changes. - Adapter launch/redaction: 9 tests passed. Projection and actual proxy: 12 tests passed before the review fix; the expanded projection suite passed all 12 cases after the fix. The proxy test exercises two runs in one process against a real local HTTP receiver and a fake provider. - Full OpenCode driver/projection suite: 72 passed, one existing response_wake test timed out at five seconds. A focused check then passed both existing omitted-continuation cases and the new trace export case. The timeout cause remains unconfirmed. - Runner package TypeScript and Rust `cargo check` passed. Map and UI token checks passed. - Prior journey validation: 273 tests across 12 files and the journey TypeScript check passed. - Broad local `pnpm test:run` encountered native-runtime integration failures with repeated embedded PostgreSQL initialization failures and was stopped. All 16 failing cases passed in an isolated run outside the filesystem sandbox (961 unrelated cases unselected). The broad local run is not a clean acceptance result. A later focused database-backed chat/heartbeat run passed all six selected observability cases, including native OpenCode launch and successful allocation (1,809 unrelated cases were unselected). - All CI gates passed for `21085f880d8babc9efeb22d0623db99613d33093`, including packaged-runner, clean-install, browser and build checks: https://github.com/paperclipai/paperclip/actions/runs/40884658995 . Greptile reviewed the same commit at 5/5 with no unresolved findings. - Verified OTLP JSON acceptance and indexing with an explicitly synthetic transport marker. No live OpenCode model trace or deployed smoke acceptance is claimed yet. ## Risks Export is best effort. Abrupt termination, recovered in-flight turns without a new submitted event, limits or receiver failure can lose traces. Export failure does not change task success. The projection represents observed normalized events, not raw native model spans or hidden reasoning. Known-secret redaction is not an OS isolation boundary. UI changes can break journey locators. Deployment adapters retain credential, ownership and cleanup responsibility. Both the controller and native runner bundle must be updated before live qualification. ## Model Used OpenAI Codex, GPT-6, with reasoning, code execution and browser tools. The exact serving model ID and context size are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted suites; broad local limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |