mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-02 16:58:52 +01:00
1d348bbef04bfd229459a707a98e67690096e444
2922
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
1d348bbef0 |
fix(server): restore durable agent folders with bounded concurrency (#18531)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A replacement web server restores agent files from shared storage. > - The old restore reads each object in sequence. > - Modest network latency makes a 1,024-file folder exceed its deadline. > - This pull request uses at most eight concurrent restore streams. > - Every file must pass its size and hash checks before the folder is published. ## Linked Issues or Issue Description Refs #18518 and #18526. Local deployment qualification found that a 64 MiB folder could not be restored after the old server filesystem was removed. ## What Changed - Restore at most eight files concurrently within the existing file authority service. - Cancel sibling transfers on failure. Await all workers before removing the temporary folder. - Keep the existing deadline, file-transfer database pool, authorization and publication rules. - Add regression coverage for bounded concurrency, corruption, settled cancellation, retained previous bytes and retry. - Document the behavior and update the deployment plan with local qualification evidence. ## Verification - Real PostgreSQL and S3-compatible storage: 1,024 files of 64 KiB each, plus instructions, with 25 ms added to each object GET. Delete the entire predecessor home and disable local adoption before reading through the successor. - Before: the read failed after 30.065 seconds. After: the same case passed in 4.180 seconds and fetched 1,025 objects. All owned test resources were cleaned. - The durable-file suite passed eight tests against PostgreSQL. Server TypeScript compilation passed. All 56 latest-head CI checks are complete and green or intentionally skipped, including full typecheck, build, unit and browser suites. Greptile is 5/5 with no review findings or open threads. - The plan separately records a released-container overlap proof: 130 reads, 26 saves, zero failures, two deleted predecessor containers and a retained authenticated session. That image predates this restore change. ## Risks - A cold folder can use eight simultaneous streams instead of one. Two file-transfer database connections bound the simultaneous restore callers. Streams verify bytes without loading entire objects into memory. - The 30-second deadline is unchanged. The measured size and latency do not qualify every folder at the maximum storage limits. - No schema, API or UI changes. Paid-model and hosted deployment acceptance remain separate work. ## Model Used OpenAI Codex, GPT-6. Tool use, code execution and repository review. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d941ed3879 |
feat(server): verify existing-volume ephemeral enrollment (#18529)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud web servers can now keep durable state in PostgreSQL and S3.
> - Existing servers still have files and a decision signing key on
their volume.
> - The volume must stay attached until every data owner verifies its
migration.
> - This pull request adds a dedicated maintenance process for that
transition.
> - Cloud can preserve the original key and remove the volume only after
a matching receipt.
## Linked Issues or Issue Description
Refs #18518 and #18526. Existing-volume enrollment is a required step in
the zero-downtime web deployment work. This is the Core maintenance
side; the Cloud transition is a separate private change.
## What Changed
- Package a standalone maintenance entry that starts no normal app
routes or execution workers.
- Coordinate the existing agent-file, run-evidence and skill migration
owners. Follow every page and verify shared bytes before final
eligibility.
- Require stopped writers. Refuse active execution, missing local bytes,
local workspaces and external instructions.
- Read the existing decision key without generating a replacement.
Preserve normal key creation for ordinary server startup.
- Require an attempt token for status and export. Export the original
key only after successful migration. Keep status and errors free of
secret material.
- Document the Cloud ownership boundary, deadline, retry and retention
requirements.
## Verification
- Full workspace typecheck and build passed before the review fixes. The
final server TypeScript compilation passed after correcting Drizzle's
optional predicate type.
- Production runtime bundling passed: 2,702 inputs, 122 chunks,
including the maintenance executable.
- The focused suite passed 21 cases using PostgreSQL and real
S3-compatible storage. It covers multi-page migration of 26 agents,
exact reads after deleting the predecessor home, corrupt-object
rejection, active/missing-data refusal, and the bundled plain-Node
process with authenticated export and clean SIGTERM exit.
- Regression cases refuse unsaved terminal working copies, require
explicit stopped/cleaned receipts for unavailable copies, clean 101
expired traces through the owning service, and enforce the deadline
after migration succeeds.
- Initial integration attempts exposed fixture cleanup ordering
problems; corrected runs pass. CI also caught a Drizzle predicate type
error, now fixed and compiled locally.
- All repository CI gates passed on
|
||
|
DottaandPaperclip
|
73cd8eb9a5 |
fix(server): deliver answers promptly during web handoff (#18530)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its web servers can overlap while Railway replaces one deployment.
> - An answer can reach the replacement while the original server owns
the native execution.
> - A competing delivery claim can make the owner miss the first
notification.
> - The normal retry interval then delays the answer by 30 seconds.
> - This pull request gives pending answers a one-second retry in
ephemeral mode and verifies delivery between separate processes.
## Linked Issues or Issue Description
Refs #18518 and #18526. This closes a measured native-question delivery
delay during control-plane web deployment overlap.
## What Changed
- Register question delivery through its owning service. Use a
one-second retry for ephemeral servers and retain the configured cadence
for ordinary servers.
- Keep the existing durable claims, response identity, authorization and
continuation behavior.
- Add a two-process PostgreSQL regression for a claim held by the
non-owning server. Verify one command, settlement, idle behavior and
reactivation by another committed answer.
- Keep the assembled startup fixture on the real registration function.
Update the deployment plan with current evidence and remaining
acceptance work.
## Verification
- The same local two-process race took 30.157 seconds before the fix and
1.197 seconds after it. The command sink is simulated; PostgreSQL,
notification transport, interaction response, delivery claim and worker
are real.
- The cross-process regression and assembled startup suite passed 122
tests. No further sweeps or pending inspections occur after settlement.
A second answer wakes the owner through PostgreSQL without a test-owned
sweep.
- Server TypeScript compilation passed. All PR CI gates passed on
|
||
|
DottaandPaperclip
|
16787e4167 |
Qualify rolling web deployments and preserve Stop through handoff (#18526)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server owns request handling and remote execution coordination. > - Deployment must start a ready replacement while the old server still serves. > - An outage test does not prove that two live servers can overlap safely. > - This pull request adds a separate Product E2E cell for that boundary. > - The existing run, file, recovery, and retained-session checks still apply. ## Linked Issues or Issue Description Refs #18518. The runtime prerequisite is merged; this PR qualifies its ephemeral server and deployment contracts. ## What Changed - Add the explicit `rolling-web-deployment` Codex Daytona workflow. - Start independent A/B/C homes and compiled builds behind a test ingress. - Wait for readiness before routing, then drain and delete the old home. - Observe active authority before handoff and use the existing read-only recovery oracle. - Require ordinary reads and saves to succeed throughout replacement. Read back every accepted save exactly once. - Preserve the five-minute outage suites and their existing validators. - Add a model-free supervisor smoke and an unready replacement control. - Require graceful exit before deleting an old home; forced cleanup cannot satisfy the oracle. - Preserve a durable Stop through lease release/expiry until the recovered execution owner acknowledges it. Permit only fenced reattachment of the original execution for Stop cleanup; submit no new turn. - Exclude events saved by A during shutdown from evidence of B making progress. ## Verification - E2E TypeScript check passed. - Affected harness checks: 143 passed. Catalog, fixture and report checks: 120 passed. - Actual supervisor smoke passed two replacements: 436 reads, 45 saves, zero request failures, and confirmed owned resource cleanup. - B became ready in 18.7 seconds; C became ready in 109.2 seconds. A prior C timeout remains failed evidence. - Fresh browser proof: 11 checks passed, 246 reads, 50 acknowledged saves, zero failures, 1,058ms reconnect, both old homes deleted and cleanup confirmed. - Stop lease-gap regression: three PostgreSQL delivery tests passed with a simulated provider edge. Current server TypeScript build/vendoring passed. - Full harness: 5,814 passed, 17 failed, four skipped. Two shutdown races passed the focused file; two report timeouts passed an isolated 42-test rerun. Onboarding/workflow-security/Pi/date-expired fixtures are unchanged on inspected master; the host embedded PostgreSQL fixture remains a setup limitation. This is not a green full local suite. - CI at `172509b8be` passed all gates. Greptile is 5/5 with every review thread resolved. The earlier access-service race failure passed its isolated regression and the subsequent full CI. - Final review corrections: six focused executor cases, three PostgreSQL command-delivery cases, 19 harness cases and server/E2E typechecks passed. - Three live attempts failed before agent creation: two harness setup gaps were fixed, then provider authentication failed. The saved OpenAI key returns 401; a refresh is pending. The Daytona probe reported cleanup uncertain, and no sandbox with that attempt label was found through the authenticated staging manager. No live model execution is claimed. - Live Runner overlap, released-container qualification and hosted deployment proof remain pending. Merging this opt-in qualification lane does not claim those results. ## Risks - The source lane shares workspace dependencies and does not establish container packaging correctness. - The live workflow needs an explicitly owned S3 bucket, PostgreSQL, pinned remote image and provider credentials. - Qualification cleanup stops all owned processes after failure. A failed replacement must not be reported as a pass. ## Model Used OpenAI Codex, GPT-6. Tool use, code execution and repository review. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2b2b5a43b7 |
fix(daytona): restore the recorded directory for file recovery (#18527)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A stopped sandbox can hold workspace files that still need a durable copy. > - File recovery must reopen the exact allocation and directory recorded by the run. > - Workspace preparation can place that directory below the allocation root. > - Daytona resume returned the allocation root even when recovery needed the recorded child directory. > - This pull request lets the file-recovery owner request that exact directory. > - Ordinary resume keeps its allocation-root behavior. The recovery source checks stay strict. ## Linked Issues or Issue Description **What happened?** A stopped native run had a prepared remote workspace. The recovery service stopped and retained its allocation. The next file-recovery step failed with `source_binding_invalid` and `reopened_cwd_mismatch`. Daytona resumed the correct allocation and matched its sentinel, but returned the allocation root instead of the workspace directory recorded on the lease. **Expected behavior** The file-recovery owner must reopen its exact recorded directory without creating an allocation, an empty directory, or an agent turn. Ordinary execution resume must keep returning the allocation root for normal workspace preparation. **Steps to reproduce** 1. Prepare a task workspace below a Daytona allocation root. 2. Stop and retain the allocation with unsaved workspace files. 3. Request source-bound file recovery. 4. Before this change, the provider returns the allocation root and the strict cwd check rejects it. **Paperclip version or commit** Reproduced from `050365e09d88411960fc3bacc1c08c46c0d9b6fa`. **Deployment mode** Docker with the bundled Daytona provider. Local tests use synthetic provider boundaries and disposable PostgreSQL. Refs: #18517, #15749. Related retained-export work: #17916. That work does not change the Daytona resume handler. ## What Changed - Add an optional driver-only `requestedCwd` resume field to the SDK contract and existing host forwarders. - Set the field only after the owned file-recovery path verifies the recorded source. - Require Daytona to match the requested directory to the recorded lease directory. Require a normalized absolute path without NUL bytes. Keep the allocation and sentinel checks. - Keep ordinary resume unchanged when the field is absent. - Add provider, host, and real workspace-preparation regressions. Document the separate directory contracts. ## Verification - The original provider regression failed for reusable and ephemeral leases. The scoped regression also failed to read a real retained file from the wrong root. Ordinary resume controls passed before the fix. - Current provider handler: **345/345 passed**. This includes exact-directory, malformed, missing, foreign, sentinel, reusable, ephemeral, ordinary-root, and real-file cases. - Removing only the owned producer's cwd request makes its real PostgreSQL regression fail with the original `reopened_cwd_mismatch`; the source file was restored exactly afterward. - Current host selection: **10/10 passed** against disposable PostgreSQL 17. It covers the inspected owner, sandbox forwarding, generic plugin forwarding, normal resume without the field, same and later task workspace preparation, and ownership rejection. The same-workspace control retains saved bytes and does not append another workspace path. - Current SDK and Daytona builds, server build, module boundaries, maps, and diff checks passed. The server used its normal unchanged-Runner cache. Its canonical input digest matched the previously verified build, and the canonical assessor accepted the copied original stamp and outputs. The changed SDK and server compiled from current source. - Local package-wide validation exposed two unchanged 30-second timeouts in the 145,000-member tar-listing cases. A single-file control reproduced both timeouts. That test mocks this provider module. The failures remain recorded; no timeout changed. The earlier package run had 497 passes, two failures, and seven skips. It is historical evidence from the abandoned broader resume experiment, not final-source package acceptance. - Initial host attempts timed out during database setup before any assertion. The final host selection used the existing supported pre-migrated disposable template. No test timeout changed. - Greptile reviewed exact head `8d0e577d1234ad394887255f13707c010dee6d24`: **5/5, no actionable issues, zero review threads**. [Current review](https://github.com/paperclipai/paperclip/pull/18527#issuecomment-6555253315). - The official [Ponytail review](https://github.com/paperclipai/paperclip/pull/18527#issuecomment-6555237850) also assessed exact head `8d0e577d1234ad394887255f13707c010dee6d24`: **5/5, no findings**. Its check is terminal neutral; its completed source assessment is not a test-execution claim. - [Primary CI](https://github.com/paperclipai/paperclip/actions/runs/41286802674) initially lost its spot runner in the UI/CLI workspace lane. The log records an explicit shutdown signal and no test assertion result. UI, CLI, lane scripts and dependencies are unchanged; the adjacent identical-source baseline lane passed. The single failed-lane retry passed at the same source. The initial failure is retained. All exact-head CI gates are now green. - No live repair or image deployment is claimed by this PR. Commands: ```sh node packages/plugins/sandbox-providers/daytona/node_modules/vitest/vitest.mjs run --config packages/plugins/sandbox-providers/daytona/vitest.config.ts packages/plugins/sandbox-providers/daytona/src/plugin.test.ts node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/services/workspace-recovery.test.ts server/src/__tests__/environment-runtime.test.ts -t 'recovers a protected retained source|owned export directory|prepares the (same|later) task workspace|does not reopen a retained source|resumes only the exact stopped sandbox lifecycle|explicit export directory' pnpm -C packages/plugins/sandbox-providers/daytona build pnpm --filter @paperclipai/server build node scripts/check-module-boundaries.mjs node scripts/check-maps.mjs git diff --check ``` ## Risks - A provider that does not support the optional field can still return a different directory. Recovery keeps failing closed on that mismatch. - Ordinary resumes omit the field. The tests cover the same workspace and a later task so that normal preparation does not gain an extra path prefix. - The field grants no source ownership, stop proof, file-persistence proof, or execution authority. Existing allocation, company, task, run, claim, and cwd checks remain in force. - No schema migration, retry-policy change, model replay, or historical-row repair is included. Existing failed attempts remain historical facts. Deployed file recovery still needs its normal admission and independent persistence and physical-stop receipts. ## Model Used - OpenAI GPT-6 through Codex. Used reasoning, source inspection, tool use, and code execution. The exact serving model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
006d5691a6 |
feat(server): support volume-free web server handoff (#18518)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. **Current behavior** The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. **Proposed behavior** Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. **Reason and benefit** This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
050365e09d |
Preserve native workspace files after lifecycle Stop (#18517)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent termination must stop execution and preserve unfinished workspace files. > - A native run can acknowledge a lifecycle Stop, then finish without an accepted result. > - The file preservation worker required a board actor even when the exact native Stop receipt was valid. > - This left a prepared remote workspace allocated without a file recovery action. > - This pull request admits the exact audited receipt for file preservation only. > - The existing recovery service can then save the files and stop the same allocation without another agent turn. ## Linked Issues or Issue Description **What happened?** A cancelled native run can have a terminal, unowned coordinator and a prepared remote workspace. Agent lifecycle cancellation records a valid run-only Stop receipt without a board user. The preservation worker rejects that receipt because it uses the board Stop predicate. The source lease remains active. A resolved execution watchdog does not supply the missing file recovery action. **Steps to reproduce** Seed a cancelled native run with a terminal-failure coordinator, no result or owner, an acknowledged and audited run-only Stop, and a prepared remote workspace. Omit board attribution. Start the registered cancellation preservation worker. Before this change, it records no workspace recovery marker. **Expected behavior** Record file recovery for the exact bound allocation. Keep the cancelled result, accounting, task state and watchdog history. Use the normal recovery service to stop, save and clean up that allocation without another provider turn. **Paperclip version or commit** Reproduced on source `1e8e32290c657227857aeda2620a73db52bab4f2` with real local PostgreSQL 17. The observed deployment uses a sandbox provider and remote workspace storage. Related current workspace work: #18514. Public PR and issue searches found no duplicate repair. ## What Changed - Expose the existing exact run-only Stop receipt check. Reject empty audit identifiers. - Use that receipt only in cancelled workspace preservation. Keep board Stop and continuation checks unchanged. - Defer preservation while the run still holds a live controller lease. - Schedule that wait at the actual controller lease expiry. A lease renewal observed under the preservation lock also sets the next deadline. - Add a PostgreSQL regression for the remote-workspace-only lifecycle record, resolved watchdog history, normal recovery admission, ownership controls, rollback and idle behavior. - Document the separate file recovery obligation and its physical cleanup limits. ## Verification - Before the fix: the registered-worker PostgreSQL regression fails because the file recovery marker is absent. - After the original repair at `6fd1042efd`: 20/20 focused tests and all five sibling suites pass, 320/320 tests. The exact-head server build and typecheck pass. - The repository reviewer then found that a live controller caused a fixed 30-second retry. Both the existing-lease and real row-lock renewal cases fail against that source. The new `adb1d4eccdd48be4c86906d5bd111a6e39f20191` schedules the actual recorded deadline before locking and after the locked reread. The lock still enforces the ownership guard. - At `adb1d4eccd`: all 22 focused PostgreSQL controls pass. The directly affected retention and board receipt sibling suites also pass, 124/124 tests across three files. They prove no intermediate reads or transactions before expiry, preservation after expiry, and no extra execution. Server typecheck and the exact-head server build pass. - Map checks, module boundary checks and `git diff --check` pass. - Run `pnpm exec vitest run server/src/services/native-runtime/native-cancelled-workspace-preservation.test.ts server/src/services/native-runtime/native-failed-workspace-recovery.test.ts server/src/services/acknowledged-native-stop.test.ts` with the supported external test-database variable set to a disposable local PostgreSQL database. The earlier broader run also covered run control and workspace recovery. - Exact-head [canonical CI](https://github.com/paperclipai/paperclip/actions/runs/41274963476) completed successfully. All 59 current-head checks are terminal with zero failures; required `ci / verify` and `ci / e2e` pass. Both [Greptile](https://github.com/paperclipai/paperclip/pull/18517#issuecomment-6553528545) and the [repository assessment](https://github.com/paperclipai/paperclip/pull/18517#issuecomment-6553542139) reviewed exact `adb1d4eccd` at 5/5, with zero findings or unresolved threads. The repository assessment's check concludes `NEUTRAL`; the validated assessment itself is 5/5. Earlier reviews, including the valid 4/5 deadline finding at `6fd1042efd`, remain in review history. The [earlier CI run](https://github.com/paperclipai/paperclip/actions/runs/41273965963) is also retained. No old result substitutes for a current-head gate. - Real sandbox recovery and physical cleanup remain unverified. Earlier failed QA outcomes remain failed. This PR changes no test grade or historical row. - Repo-wide local tests and typecheck were not run. The focused server checks cover this change; canonical CI supplies the remaining repository gates. ## Risks - The newly admitted receipt permits only file preservation. It cannot authorize a continuation, accept a native result or certify provider termination. - Recovery still requires exact company, task, run, lease, allocation and workspace bindings. Accepted results and live owners remain with their existing owners. - Provider or storage failure can leave recovery pending. The source must remain retained until saved-file and physical termination receipts exist. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI GPT-6 through Codex. Used reasoning, source inspection, tool use and code execution. The exact serving model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1e8e32290c |
Give metadata project tasks private remote workspaces (#18514)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task can belong to a project without using that project's files. > - The specification gives a task without a selected or configured source its own private folder. > - The remote source planner instead rejects such a task when its project has no configured workspace. > - This stops an ordinary delegated task before native execution starts. > - This pull request selects an empty, task-owned remote source for that narrow default case. > - Explicit source choices, existing workspace authority, and repository requirements retain their current checks. ## Linked Issues or Issue Description Refs #18504 and #18351. The earlier reference-project fix covered additional project mentions. This fix covers the task's own organizational project. **What happened?** A task in a project with no configured workspace fails with `remote_workspace_source_unavailable`. Its project has no workspace policy, its four workspace selection and binding fields are null, and it requests no repository, branch, or local folder. The failure occurs before native runtime selection. A derived project folder exists in the project facade even when no source is configured; that derived path does not grant file access. **Expected behavior** Use the task's private remote workspace. Keep the project on the task for organization. Do not adopt the project folder, agent home, or another repository. Preserve explicit workspace and source requirements. **Steps to reproduce** 1. Create a project without configured workspaces or a workspace policy. 2. Assign a normal task in that project to a native sandbox agent. Leave all workspace selections and bindings unset. 3. Start the task with remote workspace storage enabled. 4. On the prior source, observe the source-unavailable error before allocation. On this source, observe a company-owned remote workspace whose `sourceIssueId` is the task and whose filesystem project and source IDs are null. **Paperclip version or commit** Reproduced against the deployed source planner at `756599e4ea069c9338c320fe8846523d3dfb92a0`. This branch starts at `987e0932bf66e703750cd04698b3c7e8993580a7`; the relevant planner and preparation inputs at that parent match the deployed source. **Deployment mode** Native sandbox execution with remote workspace storage. The regression tests use real PostgreSQL with the current migrations. ## What Changed - Allow an implicit empty source only after a company-scoped query confirms the project has no workspace rows and no explicit workspace policy. - Keep existing bindings, pending selections, explicit source choices, Git requirements, and local folder choices outside this default. - Add real PostgreSQL source controls and an assembled heartbeat preparation control through workspace reservation and the environment acquisition boundary. - Clarify the existing task-owned folder contract in the implementation specification. ## Verification - Causal RED: the new source control and the assembled allocation control both fail against the unchanged deployed planner with the observed source-unavailable error. - GREEN: 19 focused PostgreSQL controls pass across the source planner and assembled heartbeat preparation suites. The allocation control checks the persisted company/task owner, null filesystem project IDs, private remote root, original organizational project, and unchanged project and host sentinel files. - Negative controls retain configured repository sources and reject a configured source without a URL, an explicit missing source, an unknown selection, a required project policy, a required branch, an explicit local folder, a foreign project, and unsupported read-only project sources. - Current-source transitive TypeScript checks and `node scripts/check-maps.mjs` pass. The map check is structural evidence only. - Fresh exact-head Greptile is 5/5 with zero unresolved threads at `680461e180232d0f4a057528617e39b64b4f0e3a`. [Review](https://github.com/paperclipai/paperclip/pull/18514#issuecomment-6552946468). The official [Paperclip assessment](https://github.com/paperclipai/paperclip/pull/18514#issuecomment-6552931726) is 4/5 and remains recorded unchanged. A separate non-author source review found no actionable issue after the full producer/consumer trace, with manual confidence 4/5. Neither static review ran the tests or proved deployed native execution. - Full repository tests, build, and type checks passed in [canonical CI](https://github.com/paperclipai/paperclip/actions/runs/41269733360). All 59 checks are terminal: 55 success, three skipped, and the official Paperclip check neutral; no failed or pending checks. The successful build checked out merge preview `03dce9f9bc708e778dd214ca85dfa1ba58e0a806`, with parents current main `9dbbfa4fb298448db83ace946b338bff321d8b2a` and this exact head. Its four helper blobs match this head; the four added mainline document-harness paths remain unchanged. Local tests use source aliases with the current DB, shared, and Runner source, rather than borrowed generated declarations. - Actual QA staging qualification is pending deployment. The previous failed journey and cleanup evidence remain failed; these local checks do not relabel them. ### Review follow-up - Fresh workspace retry: the normal producer sets `taskDirectory:true` for this empty-source case. [Heartbeat](https://github.com/paperclipai/paperclip/blob/680461e180232d0f4a057528617e39b64b4f0e3a/server/src/services/heartbeat.ts#L2288-L2296) separates the saved filesystem project from the organizational project. [Environment selection](https://github.com/paperclipai/paperclip/blob/680461e180232d0f4a057528617e39b64b4f0e3a/server/src/services/heartbeat/environment-selection.ts#L506) selects the fresh task directory. Preparation passes that flag, and the planner's existing task-directory return runs before the new binding guard. The proposed fresh failure omits that producer flag. The existing fresh-intent and binding checks stay strict. This is source evidence, not an extra runtime test claim. - Empty policy object: the [public project policy schema](https://github.com/paperclipai/paperclip/blob/680461e180232d0f4a057528617e39b64b4f0e3a/packages/shared/src/validators/project.ts#L24-L43) requires `enabled:boolean`. The project PATCH schema makes outer fields optional and does not make nested policy fields optional. A clear uses `null`; `{}` is rejected. A defensive parser's treatment of unknown stored data does not establish a valid `{}` producer. Keep the narrow fallback closed for non-null policies. - Boolean grouping: JavaScript evaluates the existing predicate as `sourceId || (projectId && !implicitTaskDirectory)`. The explicit-source and metadata-default controls cover those branches. The optional parentheses suggestion does not identify a behavior change; this reviewed source remains frozen. - The official 4/5 assessment and its static verification limits remain visible. The independent manual review withdrew its initial fresh-retry concern after the full producer trace. Actual staging qualification still requires deployment and a new real journey result. ## Risks - The new query runs only for a source-less default admission. It reads one authorized project and uses `NOT EXISTS` for that project's source rows. It adds no recurring poll, write, provider call, or local folder access. - A project with an explicit workspace policy remains subject to the prior source requirement. This conservative rule does not reinterpret policy fields. - The allocation control uses the real planner and PostgreSQL reservation. It stops at environment acquisition, so it proves workspace ownership rather than provider filesystem execution. Live staging qualification is still required. - No API, database schema, permission, model, budget, run limit, or QA journey oracle changes are included. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. This session does not expose a more precise backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
34f8d25089 |
fix(github): expose review finding categories (#18429)
## Thinking Path > - Paperclip manages AI agents and their work. > - GitHub review bots submit structured assessments through task-bound tools. > - Each repository can set its own finding categories. > - The schema accepts a string, but the server rejects values outside the saved review policy. > - Review instructions did not list the accepted values. The error did not list them either. > - This change gives agents the values before submission and keeps server validation. ## Linked Issues or Issue Description **What happened?** Review submissions failed with an unknown finding category. The agent corrected the assessment before publication. The sampled trace has two rejections followed by a processed review. Historical tool inputs are null. Authorized diagnostic inspection found no saved raw provider trace for the representative run. The exact rejected values remain unknown. **Expected behavior** Agents can find the accepted categories before submission. Unknown values still fail validation. **Steps to reproduce** Set a custom finding category. Start a review. Submit a different category. Before this change, the instructions and error do not list the accepted value. ## What Changed - List configured categories in manual and automatic review instructions. - Return categories from the saved assessment policy in `begin_review`. - Describe the dynamic constraint in the shared discovery schema. Keep custom categories valid. - List accepted values in category errors. Keep authorization and publication checks. - Add contract and database integration tests. Update documentation and access maps. ### Hypothesis and thesis for a fix The missing category guidance can cause avoidable submission retries. A model can also ignore the policy returned by metadata reads; the historical inputs do not establish which occurred. The repair belongs to the GitHub review contract. Use the saved policy categories at review start, explain the constraint, and retain server rejection. A fixed global enum would reject valid custom categories. ## Verification - Baseline `f1d7808`: two new contract assertions failed for missing category guidance and error detail. The same assertions pass after the change. - `pnpm exec vitest run server/src/services/chat-github-review-policy.test.ts server/src/services/chat-github-tools.test.ts`: 41 passed. - `pnpm exec vitest run server/src/__tests__/chat-channels.integration.test.ts -t 'returns snapshot finding categories'`: 1 passed. Real PostgreSQL and service code; simulated GitHub HTTP. Checks policy changes, rejection before publication, accepted custom category persistence, and processed publication. - `node scripts/check-maps.mjs`: passed. Structural evidence only. - Shared TypeScript check passed. Server TypeScript checking was killed by the 8 GiB memory limit, including a separate retry after the other checks stopped. - Full typecheck and build could not complete: `cargo` is absent. Full tests were stopped after concurrent checks exhausted memory. CI must supply the full checks. - Follow-up compiler repair: the unchanged baseline reproduces TS2352 in the category discovery test. Runtime matchers replace the unsafe schema cast. They retain both guidance and custom-category assertions. The repair is commit `0dcb94c676`, with current master merged at `14d7052854`. The final head is `c46b8f35add9255fc71be294d27f5a7865bcb47f`; the remaining code-comment review finding was removed and resolved. - Final-head CI: all gates pass, with no failed or pending checks. Greptile gives this head 5/5 with no outstanding findings; the only inline thread is resolved. The workspace test shard passed on one bounded rerun after an earlier CLI child-server readiness timeout; no code change was made for that rerun. - The review host passed full recursive typecheck and build after the compiler repair and master integration. Fresh database rechecks hit the host PostgreSQL semaphore limit. The full local test command hit a pre-existing chat line-selection failure before assertions. These local limits do not replace the passing final-head CI evidence. - No live model or GitHub publication at the changed revision. No deployment or merge. ## Risks - Instructions add policy text. An agent can still ignore it. - Existing bot catalogs get the schema description when tools are synchronized. Trusted review instructions and `begin_review` also expose categories. - The service test uses simulated GitHub HTTP. It does not prove that deployed agents avoid retries. ## Model Used The original worker used OpenAI `gpt-6.1-sol` through Codex with reasoning, tool use, and code execution. A later Codex review repaired the test compiler error. That review runtime exposes GPT-6 but no exact snapshot or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Dotta <bippadotta@protonmail.com> Co-authored-by: cryppadotta <34892728+cryppadotta@users.noreply.github.com> |
||
|
DottaandPaperclip
|
e09f40888a |
fix(recovery): preserve native task access during resolution (#18489)
## Thinking Path > - Paperclip controls agent execution and task ownership. > - A native run needs its task binding until it reports its result. > - Direct task updates already reject self-status changes that remove this binding. > - Recovery resolution can also change the source task status. > - That route did not use the same protection. > - This change rejects the unsafe status change before the task or recovery receipt changes. > - The run can then use semantic completion while normal recovery permissions remain effective. ## Linked Issues or Issue Description Refs #18411. **What happened?** A running native owner can resolve an older recovery action with `sourceIssueStatus: "in_review"` or `"done"`. The request succeeds and changes its own task status. That can release the execution binding before the run reports. Later task tools fail with `paperclip_runner_tool_binding_not_authorized`. **Expected behavior** Return a conflict before the task status or action receipt changes. Tell the run to report through `paperclip_finish` or `paperclip_block`. Preserve unchanged-status resolution, Board recovery, legacy-agent behavior, and the existing conversation and handoff rules. **Steps to reproduce** 1. Give a running native owner the task execution lock and an older recovery action. 2. Call `POST /api/issues/{id}/recovery-actions/resolve` as that run. Use `outcome: "restored"` and `sourceIssueStatus: "in_review"`. 3. Before this fix, the route returns 200 and changes the status. After this fix, it returns 409 with `native_run_status_reported_by_finish` and makes no task or action mutation. ## What Changed - Reuse the existing native self-status guard inside recovery resolution's source-task transaction. - Let the guard receive the requested status from either route contract. - Add HTTP route regressions for rejected review/completion transitions and continued task editing after rejection. - Cover unchanged-status native resolution, Board recovery and legacy-agent resolution. - Update execution semantics and both recovery maps. ## Verification - Baseline `91fdf02`: the two new rejection cases fail because the real Express route returns 200. Three safety cases pass. - Fixed source: all 198 tests in `server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts` pass. - `node scripts/check-maps.mjs` passes. This is structural evidence. - Greptile and Paperclip Review both give the integrated head 5/5 with no actionable findings. - Full recursive typecheck, full build, and a repeat of all 198 route tests pass at integrated head `0fa104dfb240500acd01303cd446363ea39c6c75`. All current-head CI gates pass (55 successful checks/statuses; no failed or pending checks). - The full local `pnpm test:run` command fails in chat-test line selection with `No test found`, before assertions. This also occurs with the unchanged master chat test file. It is not a passing full local suite. - The route tests use the real route, validation and guard. They mock service and database boundaries. The observed hosted failure establishes the original mechanism. Fixed-source live provider acceptance remains unverified. No deployment or merge is part of this PR. ## Risks - A native owner now receives a conflict for a previously accepted recovery status change. It must use the existing semantic result path. - This prevents a new binding loss. It does not prove that an old remote executor stopped or recover missing provider state. - Board, legacy, conversation and handoff behavior retains existing authority. The regression includes representative allowed paths. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific snapshot ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (198 focused route cases; the full local command has the recorded selection failure) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c38844eafa |
test: observe complete workspace retention cleanup (#18507)
## Thinking Path > - Paperclip manages AI agents and preserves their work. > - Remote workspace retention removes saved objects after their last pin clears. > - The retention worker deletes each object before it commits the final snapshot state. > - The parent-revisit test treats the first object deletion as completion of all work. > - That valid intermediate state can fail the next-deadline assertion. > - This pull request observes complete storage, database, and worker settlement within the existing bound. > - The test still fails if the parent is not revisited after the child releases its pin. ## Linked Issues or Issue Description Refs: #18351 Refs: #18506 **What happened?** [The server6 CI job](https://github.com/paperclipai/paperclip/actions/runs/41263476916/job/124721633674) failed at `remote-workspace-snapshot-retention.test.ts:248`. The test expected a null retention deadline after the first parent object disappeared. The parent could still have its second object and a `delete_requested` row. The worker was not finished. **Expected behavior** The test must wait for both parent objects, both snapshot rows, and the retention worker to finish. It must retain the four-second bound and 40ms observation interval. **Steps to reproduce** 1. Use the parent/unfinished-child fixture in the retention test. 2. Hold the second parent-object deletion after the first object is removed. 3. Read the old assertion while the parent row remains `delete_requested`. 4. The old assertion receives `1793361600000` instead of null. Release the hold and observe the completed state. **Paperclip version or commit** The failed CI checkout was `ab849702a78f6bb619b1527596cd52db402d9189`. This test-only repair starts from actual mainline `b110271c9b2fd6dd5a1beb119e7f0663385df8b9`. The relevant test, retention service, coordinator, storage, test-helper, and configuration blobs match the failed checkout and mainline `756599e4ea069c9338c320fe8846523d3dfb92a0`. **Deployment mode** Built-from-source server tests. The local controls used one disposable PostgreSQL17 instance and local disk storage. They used no deployed company or model. ## What Changed - Observe both parent objects and the exact parent and child database rows in one bounded assertion. - Require deleted rows with cleared deadlines, zero active tracked work, and no coordinator wake before the test ends. - Keep the original four-second timeout and 40ms interval. Production retention and scheduling code stay unchanged. ## Verification - All17 tests in `remote-workspace-snapshot-retention.test.ts` passed on Node24 with current-source aliases, current migrations, and a disposable PostgreSQL17 database. - At immutable head `1e7eeb422be260bfaf752792596f8cdd126fff8d`, the second-deletion hold control also passed all17 file tests. Removing only the parent-revisit signal still failed the affected case within four seconds. The owned database container was stopped and its automatic removal was verified. - Normal checkout command: `pnpm --filter @paperclipai/server exec vitest run src/services/remote-workspace-snapshot-retention.test.ts --reporter=verbose`. - Four bounded controls used the exact failed CI source: unchanged baseline PASS; held second deletion with the old assertion RED; the same held state with complete-settlement observation GREEN; removed parent-revisit signal RED under the original four-second limit. - The held state proved: first parent object absent, second present, child row deleted, parent row `delete_requested`, non-null deadline, and active worker. This proves a reachable observer bug. The exact interleaving in the historical CI job was not recorded and remains unknown. - The same case passed at mainline756 in [the canonical readiness control](https://github.com/paperclipai/paperclip/actions/runs/41260612021/job/124712930107). Source identity and a passing control alone do not establish cause; the held-state control supplies that evidence. - `git diff --check` passed. The net source change is one test file, with no added comments or runtime changes. - Final exact head `1e7eeb422be260bfaf752792596f8cdd126fff8d`: [canonical CI41264965700](https://github.com/paperclipai/paperclip/actions/runs/41264965700) completed successfully. All57 checks are terminal and clean, with no pending or failed checks. This includes the aggregate verify and Snyk gates. The PR is mergeable and clean. No old failed job was retried. - [Greptile reviewed this exact full head at5/5](https://github.com/paperclipai/paperclip/pull/18507#issuecomment-6552310742), with no actionable findings and zero open review threads. Complete thread pagination was checked. - [The current Paperclip assessment is4/5](https://github.com/paperclipai/paperclip/pull/18507#issuecomment-6552292204), with a final full-head footer. Its check conclusion is neutral. It calls the fix correct and identifies one minor, non-blocking concern about a process-wide active-work counter. This score remains4/5; it is not represented as5/5. ## Risks The assertion is stricter because it checks complete deletion and worker settlement. A real missing revisit, retained object, unresolved row, or continuing worker must still fail. The bound is unchanged. No schema, API, production storage, scheduler, or deployed configuration changes are included. No user-facing command or behavior changed, so no product documentation update is needed. The Paperclip review notes that `idleWorkSnapshot().active` is process-wide. The current test configuration uses fork isolation, one worker, and serial tests, so the counter is scoped to this fixture in the present test run. We retain it because deleted rows and an empty wake queue can be observed before the coordinator finishes its active work. If test isolation or fixture concurrency changes, this assertion will need a worker-specific settlement check. This is the review's accepted non-blocking limitation; it is not waived or scored as resolved5/5. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source inspection, code execution, and tool use. The session does not expose a more specific backend model ID or context-window size; neither is inferred. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b110271c9b |
test: observe GitHub enrollment after worker settlement (#18506)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub connection events use a worker that must return to quiet after it drains an event. > - The integration test checks that work with real PostgreSQL and signed loopback requests. > - The loopback server can receive an ACK before the worker completes that request. > - The test reads the next deadline too early and can reject valid work in progress. > - This pull request waits for the observed work state without moving the fake clock. > - It keeps the exact work counts, identity checks, deadline bound and quietness checks. ## Linked Issues or Issue Description Refs #18395 and #18351. This is a test-only repair for the event-worker integration fixture. **What happened?** [CI run 41257779002](https://github.com/paperclipai/paperclip/actions/runs/41257779002/job/124704574485) failed at the replacement follow-up deadline assertion. `coordinator.nextWakeAt()` was `null` after the server received the second ACK. The worker can still be waiting for that ACK response at this point. **Expected behavior** Wait for the worker to expose a deadline or finish the exact expected follow-up. Do not drive fake time during an observation. Require the scheduled follow-up to stay within five seconds of the first observation. Keep one empty follow-up, the new identity and no I/O after the worker becomes quiet. **Steps to reproduce** 1. Run the existing enrollment integration test on the original CI merge `a1e25444a95f3698c3761cd1796bd1e11567224c`. 2. Hold only the second loopback ACK response after its server counter increments. 3. The original deadline assertion fails while the worker remains active. **Paperclip version or commit** The causal control used the exact CI merge above. This repair starts from verified master `66e8fab9447aa21350f87ddec79fcd027f3e23ad`. The fixture and relevant worker files match the earlier `b889` inputs. The original failure remains preserved. The controlled race does not prove the unique historical scheduling of that CI attempt. ## What Changed - Use bounded real I/O yields for observations. Assert that the fake clock stays fixed during each observation. - Advance fake time only to actual pending follow-up deadlines. Keep one fixed five-second reference for each drain. - Accept an already finished follow-up only when the exact expected lease count is present. Quietness must still follow. - Hold the replacement ACK on a Promise gate in the existing test. Release it during normal work and in cleanup. This keeps a deterministic regression without a sleep. - Keep the signed request, identity replacement, exact lease counts, 25-minute no-I/O checks and 30-second test budget. ## Verification - Original CI: one assertion failure at line 126; 2,619 other tests passed in that lane. It was not a test timeout or host failure. - Earlier controls: the same test and relevant worker inputs passed in two fresh CI lanes. Both used zero Turbo cache hits. Input identity alone is not the cause proof. - Causal controls on the original CI source: unchanged baseline passed; held ACK failed at the original assertion with active work; a settlement barrier passed all original assertions; a missing second hint still failed. - Final patch with real PostgreSQL 17 and current source migrations: the direct test passes, including its deterministic ACK gate. An extra 750 ms ACK delay also passes. The missing-second-hint negative still fails at the expected observation. - Focused command: `pnpm exec vitest run --project server server/src/__tests__/github-connection-enrollment-work.test.ts`. Local checks used current-source package aliases and one disposable loopback PostgreSQL instance. The test creates a unique database and no user data. - Fresh Greptile review is 5/5 for `519c3fd4a99bf77aea7b085b67c38701358fef95`. The exact head has no open review threads. The current Paperclip assessment is also 5/5 with a final response footer for that full commit. Its check conclusion is neutral. - [Canonical CI 41260540629](https://github.com/paperclipai/paperclip/actions/runs/41260540629) completed successfully for the exact head. All 57 checks are terminal with no failures or pending checks. This includes `ci / verify` and Snyk. The PR is mergeable and clean. - `git diff --check` passes. Full repository typecheck, tests and build are delegated to fresh PR CI. No QA journey, model run, deployment or CI retry was performed for this repair. ## Risks This change affects one integration fixture. Production worker logic, schedules and deadlines do not change. The observer retains a one-second real-time bound and the test retains its 30-second budget. The test still fails on missing work, extra work, a late deadline, a wrong identity or I/O after quiet. The database and HTTP server remain owned by the fixture. ## Model Used OpenAI Codex, GPT-6 family. The session does not expose a more specific backend model ID or fixed context window. Used reasoning, repository tools and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (test-only behavior; described here) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
756599e4ea |
fix(workspaces): keep project references without source folders (#18504)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can receive project references in normal task and chat messages. > - Remote workspace preparation checks those references before a native run starts. > - A project with no workspace source has no folder to attach. > - The current check rejects that project as an unsupported read-only folder. > - This pull request keeps the project reference and reports the missing source without attaching files. > - Projects with configured sources keep the current remote access restrictions. ## Linked Issues or Issue Description Refs #18403 and #18351. This is a product prerequisite for the hiring journey. It changes no journey code. **What happened?** A native chat run failed before execution when its message referenced an authorized project with zero configured workspaces. The error was `workspace_validation_failed`: “This remote runtime cannot safely provide read-only project folders.” The project had no repository or checkout. The failed QA run and original grade remain preserved. **Expected behavior** Keep the task workspace when a referenced project has no configured source. Report that no project folder was attached. Continue to reject configured cross-project sources that require an unsupported read-only mount. **Steps to reproduce** 1. Use a native remote task folder with multi-project references enabled. 2. Mention a same-company project that has no configured workspaces in a normal comment. 3. Prepare the run workspace. **Paperclip version or commit** Reproduced on `b889ac39b7fbed558c5f4cfbbc2b73c4ca916810` with the real workspace resolver and PostgreSQL migrations. The staging failure was recorded in workflow [41253428446](https://github.com/paperclipai/paperclip-cloud/actions/runs/41253428446). ## What Changed - Distinguish authorized project references with a known empty workspace list from configured sources. - Return a resolution warning and failure receipt for an absent source. Do not attach a folder or create a workspace. - Preserve project authorization, company filtering, the disabled remote-source gate, and the read-only restriction for configured sources. - Add real PostgreSQL regression cases and document the reference contract. ## Verification - Causal RED on unchanged `b889`: the metadata-only project case failed with the same read-only error. The real repository negative already passed. - GREEN: eight PostgreSQL cases pass. They cover an absent source with the remote-source switch both on and off, a saved existing workspace, a repository, a configured source without a repository URL, the disabled remote gate, mixed references, and a foreign company. - The existing assembled workspace test also passes: a real read-only project source is rejected before reservation or environment acquisition. Total: 9 selected tests pass; 119 unrelated tests were not selected. - Tests use PostgreSQL 17 and the current source migrations. The local Vitest configuration selects current DB, shared, and Runner sources because the borrowed dependency output predates the remote workspace feature. - Current-source TypeScript checking of both changed TypeScript files and their dependency graph passes. It uses the repository's real Express actor declaration. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Canonical reproduction: `pnpm exec vitest run --project server server/src/services/heartbeat/remote-workspace-source.test.ts` with the existing embedded PostgreSQL support or `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` set to a disposable test cluster. - Historical `0f2b` passed all 59 CI checks. A separate non-author manual source audit reported no actionable findings at confidence 4/5; it did not rerun tests or qualify live QA. The official 4/5 review requested a flag-scope test and saved-workspace control. This test-and-documentation update adds both, with production source bytes unchanged. - Current-head canonical [CI 41258109282](https://github.com/paperclipai/paperclip/actions/runs/41258109282) passed: all 59 checks are terminal (55 success, 3 skipped, 1 neutral), with no failures or pending checks. Build and typecheck passed. The eight-case regression file passed in the canonical Linux server 1/12 lane. Its checkout `6f69ccca70d10e78885b133fb54a80940bf175a5` has parents `b889` and this exact head; its tree equals this head. - Fresh Greptile and official Ponytail assessments each give 5/5 for exact `27660027d12ff31427d611019e90dabfb6d8503a`, with zero unresolved review threads. The separate manual audit remains 4/5. - Live QA remains pending on a reviewed app build. This regression proof does not claim that the hiring journey passed. ## Risks A project source could be added after project hydration. This change uses the same run-preparation source snapshot as the existing authorization and source check. It only permits a positively known empty workspace list. Configured sources, unknown shapes, foreign projects, and existing execution authority keep their current checks. No schema, credentials, provider, model, budget, or filesystem permissions change. ## Model Used OpenAI Codex, GPT-6 family. The session does not expose a more specific backend model ID or fixed context window. Used reasoning, repository tools, TypeScript and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
66e8fab944 |
fix: complete native chat responses and improve API discovery (#18364)
## Thinking Path > - Paperclip must perform work and report its outcome accurately. > - A chat response and its delegated or operator-owned work are separate stages. > - Native chat contracts instead required literal completion of the user's whole request. > - Agents kept searching or waiting even when a truthful handoff should finish the response. > - OpenCode also has separate acknowledgement backlog and empty-response failure paths. > - This PR aligns chat completion with its current stage and makes OpenCode completion truthful. ## Linked Issues or Issue Description Refs #18098 for retained OpenCode evidence and #18377 for separate eval setup, oracle and reporting repairs. **What happened?** Real pinned OpenCode 1.18.34 reproduced an MCP timeout behind 800 text fragments. Pilot [41222196525](https://github.com/paperclipai/paperclip/actions/runs/41222196525) separately exposed a fabricated successful reply after a reasoning-only response. Later restart attempts timed out without an answer. Their native criterion literally required restarting the instance, although the chat instructions and authority required an operator handoff. General authority guidance alone did not resolve it. **Expected behavior:** A standing chat completes its current authorized response. It performs and verifies permitted actions, verifies delegated handoffs, or explains the specific unavailable action and operator path. The worker's task remains separate. Ordinary execution tasks must still complete their requested work. Empty provider responses must not become invented success. **Version and reproduction:** Actual pinned OpenCode and native Rust Runner, with retained full-stack AWS attempts. Product baseline master `51ea624344`. This PR adds no historical catalog to master. ## What Changed - Build standing-chat completion contracts from the server-owned conversation identity. Scope each current request to the response stage; preserve request text, criterion IDs, order, human-response scope and source revisions. Require verified authorized actions and truthful handoffs. Ordinary task contracts stay strict. Existing admitted inputs retain their saved contract during recovery; new contract content uses the existing revision/hash mechanism. - Guide prospective defaults to their owning resource and preserve existing work unless backfill is requested. Guide unavailable board/operator operations to the supported handoff. These are general chat instructions with no eval identities or expected answers. - Batch adjacent assistant text for at most 50 ms and 4,000 UTF-8 bytes. Preserve text, metadata and order; flush before non-text frames and shutdown. - Give an empty OpenCode response one correction within its existing turn, model, reasoning, context and deadline. Accept actual completed answer parts with incomplete headers. Preserve actual active tool work, bound idle evidence grace and fail truthfully if correction stays empty. - Reject empty OpenCode terminals in Rust without actual prose or an accepted semantic result. Never invent successful replies or evidence. Refresh generated project-discovery contracts. - Preserve accurate public API-discovery descriptions for existing project defaults and managed instance restart operations, including their field scope and authority limits. ## Verification **Qualified: [pilot 41256592937](https://github.com/paperclipai/paperclip/actions/runs/41256592937) passes 3/3, with all 35 independent checks passing.** | Case | Checks | Full case duration | Evidence and cleanup | | --- | --- | --- | --- | | Instance restart | 9/9 | 140.1 s | complete | | Project defaults | 11/11 | 204.2 s | complete | | Project work | 15/15 | 204.0 s | complete | All three are original-attempt results from source `c51e079f7f2364305b131ffd549e263ff78c5335`, containing the exact product files from `13c2b61f31d873a475ea0c3c026b19926ff42301`. Actual saved conversation contracts were inspected, as were ordinary native worker contracts where present. Each protected archive includes the actual API state and final PNG; cleanup completes. Original queries, `openrouter/deepseek/deepseek-v4.1-flash-1017`, finite case clocks, concurrency one and zero eval retries are preserved. Earlier passes do not transfer into this qualification. Both current heads pass all checks: product **62** (56 success, one neutral, five skipped), catalog **56** (54 success, one neutral, one skipped). Both have exact-head Greptile **5/5**, zero unresolved review threads and clean mergeability. The three trusted private-evidence protocol files remain byte-identical to the dispatch workflow. Raw evidence stays protected. - **2,044 API discovery tests pass.** Red controls showed that “restart instance” selected unrelated drain/settings operations and “save project defaults” selected repository replacement. Accurate public summaries now rank the existing operations correctly; route registration preserves descriptions. No endpoint, permission or field behavior changes. - **112 real-database, authenticated HTTP, native Runner and OpenAPI consumer tests pass locally**, using disposable task-owned Docker PostgreSQL. It was removed afterward; shared IPC was untouched. These cover persisted contract selection, admitted native input and the model envelope together. The isolated CI suites also pass. - **77 focused contract/input/context tests pass.** Red controls catch the wrong conversation predicate; canonical identity requires both agent and user IDs, including the actual heartbeat projection. Ordinary tasks, agent-only issues and planner tasks remain strict. Coalesced requests preserve criterion identities, source hashes and human-response scope. - **144 OpenCode driver tests pass**; native core **441 pass, one ignored**; packaging **56 pass, six skipped**. Server typecheck and build pass. - Real pinned OpenCode/native Runner: a delayed 25-second acknowledgement completes within the original 30-second MCP deadline. The text-fragment control changes 817 notifications / 54.1 seconds / timeout to 45 / 19.0 seconds / accepted finish. Separate fast, six-minute and real authenticated API probes complete 32 tool calls plus finish. These do not reproduce every remaining historical MCP timeout, and no common cause is claimed for all such failures. Retained failures remain attributed to their original source. Earlier runs exposed fabricated empty-response success, prospective-default backfill, a literal whole-task chat criterion, fixture readiness and oracle/reporting defects. The immediately preceding [41250502608](https://github.com/paperclipai/paperclip/actions/runs/41250502608), product `c0bf249807` / source `7b1de24ce1`, is **1/3**: work 15/15; restart 4/9 with the repaired contract actually admitted but no answer; defaults timed out before settlement, then exceeded the unchanged 256 MiB original-evidence allowance. Defaults domain behavior on that attempt remains inconclusive. #18377 retains the full attempt ledger, including missing archives and pre-paid cancellations. The fresh 3/3 establishes these selected outcomes; it does not establish full-catalog reliability. ## Risks The shared contract change applies to standing native chats across providers. It changes model instructions and completion scope, while permission, approval and review gates remain enforced by their existing owners. Incorrect model claims remain possible; the product evals independently require actual mutations, native work and delivery. The correction adds at most one provider request within the existing deadline; text batching can delay adjacent text by 50 ms. Three qualified attempts do not establish full-catalog reliability. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, tool use, code execution and live eval tooling. Exact hosted model revision and context-window size are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge - [x] Final same-source three-cell pilot passes 3/3 with required evidence and cleanup --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b889ac39b7 |
fix: serialize native completion replies and bounded retries (#18502)
## Thinking Path > - Paperclip manages agents and their work. > - A delegated task reports its result to the source conversation through a durable completion outbox. > - Native execution can succeed before the presentation worker saves the accepted final reply. > - The outbox treated that short interval as a missing answer and cleared the target run. > - The presentation worker then rejected the valid answer because its delivery no longer belonged to that run. > - This change retains the existing target for one fixed presentation grace while accepted presentation is pending. > - The existing presentation owner can save the reply without another model run or more budget. ## Linked Issues or Issue Description **What happened?** A completion-delivery retry can run immediately after a native callback succeeds. It clears the callback's delivery target before the presentation worker saves the accepted answer. The reply then fails with `chat_completion_superseded`. A daily run limit correctly blocks the unnecessary replacement run, leaving the conversation active without its saved answer. **Expected behavior** The outbox must retain its current target and attempt while that exact native run has accepted presentation work. The existing presentation worker must save the answer once, acknowledge the delivery, and settle the conversation. Existing actor, session, result, digest, budget and retry checks must remain enforced. **Steps to reproduce** 1. Accept a native DONE result and current-turn final message for a completion callback. 2. Run its due completion-outbox handler after terminal projection but before reply presentation. 3. Run the existing presentation worker. Before this fix it rejects the reply because the outbox has cleared the target. **Paperclip version or commit** Reproduced on `b7f1c3769c00a107ffe4128b48b0ef8cc452cf3f` through the real PostgreSQL outbox and presentation services. The reported failure remains preserved. A new QA run is required after deployment; local tests do not qualify the deployed fix. Related: #17696 schedules accepted-response recovery. #18118 preserves canonical completion receipts. #18364 handles provider output and response-stage contracts. Those changes do not protect the completion-outbox target during this interval. ## What Changed - Defer a new execution attempt when the existing selector finds pending presentation for the same company, conversation, run and agent. - Retain the queued delivery target and attempt with an atomic compare-and-set. Await the existing scoped worker notification before that write in the same transaction. - Serialize retry with reply insertion under the existing child-task-first lock. Recheck locked task Done state and the delivery's stable completion-event version, status, target, attempt and exact lease. A delivered, retargeted or newer-claim winner prevents another wake. Admit the next attempt only after the winning transaction commits and releases its locks. - Limit presentation grace to five existing one-minute delivery leases, measured from the original run `finishedAt`. Clamp both the delivery lease and the native retry deadline to that immutable limit. Repeated sweeps cannot extend it. After expiry, resume the existing retry, exhaustion and daily-limit checks. - Keep publication authority in the existing materializer. The candidate query only defers a retry. Read the freshly reloaded delivery status, target and saved receipt after acquiring its child-task lock. - Fence both hold and retry writes with the exact claimed event version, attempt and lease. A stalled old hold cannot overwrite a newer lease. - Keep completion-event identity separate from mutable task ownership versions. Ordinary Done reassignment preserves its existing completion event; a real reopen retains normal supersession. - Add real PostgreSQL coverage for the observed ordering, outer commit and rollback, one saved reply, conversation settlement, idle SQL, retry boundaries and superseded deliveries. - Document the completion-outbox and presentation contract. ## Verification - Before the fix, the real PostgreSQL regression fails with `chat_completion_superseded` in `existingChatCompletionReply` when the outbox handles the terminal run first. After the fix, the same test passes without a new model wake. - Focused command: `vitest run --project @paperclipai/server server/src/services/native-runtime/agent-chat-response-recovery.test.ts server/src/__tests__/chat-completion-delivery.test.ts` — **95 pass** on `45f53a8c32fcca0bbdacf717286d37e30c1ba308`, using a disposable database on an owned local PostgreSQL 17 container. - Controls preserve retries for failed, cancelled, missing-final, missing-accepted-result, old-generation, foreign-company and foreign-agent targets. Reset or superseded task versions retain supersession. Digest and external-context declines never authorize a reply, never extend the fixed window, and leave it at exact expiry. An external native deadline cannot extend the limit. A valid reply at the last millisecond saves once without another model run. - The first reviewed head `b6f9c484b9567bc60af6d684c04e00058c1048ed` received Greptile 5/5 but an independent 3/5 review correctly found an unbounded hold. Three new real PostgreSQL controls failed on that source. This head adds the fixed bound and makes them pass. The earlier review, controls and CI remain retained; its review score does not qualify the new head. - The bounded predecessor `7300d2a978a5c4b3c13d85899bed3e98ae3c49fa` received an independent 4/5 review that correctly found an expiry-versus-in-flight-reply race. A real PostgreSQL advisory gate reproduced it after reply authority checked the delivery but before insertion. The retry lock and exact claim CAS now make that control pass. Both orderings, outer rollback, delivered/retargeted/superseded winners and competing lease claims pass. The next-wake test can acquire both parent and child locks, proving dispatch occurs after release. The valid 4/5 review and failing control remain retained. - At `39dd86e8f51d937cf382b2757736e0477201d21d`, both reviews found valid remaining defects: the reply helper still used its pre-lock delivery snapshot, and ordinary Done reassignment could stall a delivery by changing the task version. Four real PostgreSQL controls fail on that source and pass here: concurrent retry-first, a newly saved receipt after waiting, and reassignment within grace or at expiry. A real reopen through the issue service still supersedes the old event. - A later coordinator review found the hold CAS still lacked the attempt and lease fences already present on retry. A real PostgreSQL producer barrier reproduced an old hold shortening a newer claim from 120 seconds to 60 seconds. The same exact claim fences now prevent that overwrite. This failed control is retained; no old run was blindly retried. - The real worker saves one exact run-owned reply and acknowledges the delivery only after outer commit. An outer rollback publishes no reply. After settlement, advancing the scheduler clock by 60 seconds causes no further SQL. - Module boundaries, map structure, `git diff --check` and the canonical server build (including TypeScript and Runner vendor preparation) pass. The first local build stopped in the host package-manager setup; the installed pinned pnpm completed the build. Full repository typecheck, test and build gates pass in current-head CI; they have not all been run locally. - Final local checks also caught a test cleanup type error (TS2677) and one 94/95 readiness failure: the worker saves its comment before post-commit conversation settlement finishes. The test-only follow-up removes an invalid cleanup type predicate and waits on the existing bounded worker-quiet check before reading final state. All assertions and the same five-second bound remain. Source `45f53a8c32fcca0bbdacf717286d37e30c1ba308` passes all 95 cases and the canonical server build. The earlier failures are retained. - The final policy cleanup removes only 15 newly added narrative comment lines (seven service, eight test), as required by CONTRIBUTING.md. Both TypeScript AST signatures and `removeComments=true` emitted JavaScript are identical to tested source `45f53a8c32`. This was checked with the TypeScript 6.0.3 API and the installed native TypeScript 7.0.2 compiler. Assertions, timeouts and behavior are unchanged; the 95 tests were not repeated for identical compiled code. - The superseded `45f53a8c32` CI run recorded two 15-second timeouts in attention-retention cases. That failure remains retained. The test and its ten inspected owner or support files are identical on the tested master revisions. The canonical base run passes all 53 attention cases, and the comparison PR passes the same server lane, including those two cases in 9.3 and 9.8 seconds. This does not establish the timeout cause. Current-head CI is a new run after the required comment cleanup, not a retry of the failed source. - The optional Paperclip review bot ended neutral without a completed assessment because its execution paused. That result is not an approval. A separate [independent read-only audit](https://github.com/paperclipai/paperclip/pull/18502#issuecomment-6550216227) reports 5/5 confidence on exact head `9cfd3f74c69cc7a2da7b4ee5d1fca80f15d4af0c`, with no findings. Its manual assessment does not replace or relabel the incomplete bot result. - Current head `9cfd3f74c69cc7a2da7b4ee5d1fca80f15d4af0c` has a fresh Greptile 5/5 review with no open threads. [CI run 41249007430](https://github.com/paperclipai/paperclip/actions/runs/41249007430) passes all primary jobs and both required gates (`ci / verify` and `ci / e2e`). All 59 PR contexts are terminal, with no failed contexts. The actual CI checkout is `18db9e2750d6a799f26b5b8ba31bcccbea254227`, which combines master `681f7b77262a9df6c1b4aad55291bf960061e491` with this exact head. Fresh QA verification on the deployed application remains required before claiming the live failure is fixed. ## Risks A candidate can retain a delivery whose stricter presentation checks still prevent publication, but only within the fixed five-minute window. The existing worker keeps publication authority. If a valid reply remains unavailable at expiry, the normal callback retry may require another admitted model run; the original result and completed child work remain preserved. Existing retry, exhaustion, model-run and spending limits still apply. Failed or cancelled execution and genuinely missing accepted output retain the existing retry path. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, tool use and code execution. The runtime does not expose the exact hosted revision or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
681f7b7726 |
perf(server): retire aggregate maintenance polling (#18425)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
9ca9af544a |
Drive remaining live run and resource recovery from durable work (#18419)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
eb12cfc96b |
perf(github): activate connection work from events and deadlines (#18395)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
5e4cc01073 |
feat(workspaces): keep remote files off the webserver disk (#18403)
Keep native sandbox workspace files independent of the webserver disk. Use encrypted saved versions and Postgres ownership so a replacement server can recover files and continue work. Enable the path by default for new eligible workspaces when durable storage is configured. Preserve explicit opt-outs and existing host workspaces. Include safe snapshot repair, Files access, import, migration and retention. Bound saved-Files resources, preserve imported Git recovery, reject unfinished Git operations, publish immutable concurrent captures, and verify permanent object-version deletion. Validated 471 targeted local controls, build/typechecks, all 40 CI partitions, fresh Greptile 5/5 and security review. The PR records original failed attempts and scoped retries. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
75fa37f30f |
fix(server): expire native backups from retention deadlines (#18380)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
453f5fbb78 |
perf(external-objects): schedule scoped refresh obligations (#18384)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
c32ecf6657 |
fix(server): schedule only outstanding merge confirmations (#18382)
Reviewed and verified at
|
||
|
Nicky Leach
|
6484f96c28 | fix(workspaces): inspect storage only when needed (#18410) | ||
|
DottaandPaperclip
|
9a3f395737 |
fix(server): settle routine history without idle scans (#18399)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
91fdf02e95 |
perf(tasks): defer completed transcripts on compact opens (#18461)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task conversations must show the saved instruction and recent replies quickly. > - Compact task reads already limit the saved message window. > - The conversation still waits for completed run transcripts even when a saved reply contains the result. > - Removing that wait alone can insert activity after the reader starts reading. > - This pull request defers that output until the reader opens history and keeps required output in the initial reveal. > - The benefit is a useful conversation sooner, with a stable first layout and a working composer. ## Linked Issues or Issue Description Refs #17593, #18092, #18312, #18358, #18422. **What happened?** Opening a task with a long saved investigation can wait for native events or legacy logs before showing its existing replies. A slow transcript request can hold the conversation behind the loading screen until the 15-second fallback. **Expected behavior** Show the bounded saved conversation without reading completed output that already has a usable saved reply. Keep that output available through the existing history controls. **Steps to reproduce** 1. Create a task with many completed runs and long saved agent replies. 2. Delay its run events and logs. 3. Open the compact task directly. Before this change, the transcript read blocks the saved replies and composer. ## What Changed - Defer completed successful run transcripts on compact task opens when a usable saved reply is present. Load that output through the existing history controls. - Keep required output on the initial path: live and failed runs, yielded runs, pending interactions, placeholder/progress-only replies, explicit anchors, chats, and full history. Retain output already selected during this visit when its run completes. - Keep saved plans visible when their owning run output is deferred or still loading. - Keep the same composer and draft through history expansion. Offer **More run output** when a single saved run has deferred output. - Position same-task comment links after their target arrives. Wait only for a missing target, with a 15-second limit. Unrelated stalled metadata cannot block an available target after the reveal deadline. - Preserve the reading anchor when a queued scroll event arrives during a properties-pane width transition. - Add a 200-reply/200-run PostgreSQL browser fixture with a saved plan, covering both task views on desktop and mobile. Correct stale browser setup for current routes and the shared mobile scroller. - Remove a random company-prefix collision in the isolated privacy test fixture that failed during CI. ## Verification - Final revision `fba3f2fcbb`: 386 focused thread/scroller unit cases and all 32 production-browser journeys passed. The browser cases cover both task views on desktop/mobile, saved plans, delayed transcripts, history, comment links, Back, drafts, retry, avatar continuity, media and pane resizing. The 25 privacy fixture cases also passed against PostgreSQL. - Workspace typecheck and full build passed with Node 24. A broader local UI run passed 10,220 cases across 849 files before the final navigation/plan review fixes; the changed components were then rechecked in the 386-case focused run. - Five alternating pairs compared master `12b0cc6304` with this revision, using 200 long saved replies and 200 completed native runs. Both production UI builds used the same loopback API/database, Chromium 151.0.7922.34, 1440 × 900, fresh browser contexts, service workers blocked, no CPU throttle, and a controlled 3-second events/log delay. Builds and other suites were stopped during measurement. No page errors occurred. | Useful conversation + editable composer | Baseline median (range) | Fixed median (range) | | --- | --- | --- | | Cold direct open | 3.570s (3.538–3.616s) | 1.056s (1.022–1.141s) | | Task-list navigation | 3.373s (3.342–3.394s) | 0.899s (0.885–0.915s) | | Back | 22ms (22–25ms) | 25ms (25–28ms) | - Median paired savings: 2.519s cold, 2.470s from the list. Completed-output requests: two per baseline sample, zero per fixed sample. Recent-comments API control medians were 13.6ms for both, with ranges 12.1–18.7ms and 11.6–50.8ms. No Back improvement is claimed; an earlier revision's run also included one 1-second Back outlier, retained in the local evidence. - The full local default test command did **not** pass: it stopped on the existing chat test line-selection error. After restoring embedded PostgreSQL library symlinks omitted by an install with scripts disabled, Node 24 passed 672 cases in the first two chat shards, then failed to select the third. This is separate from the passing focused suites and remote CI. - Earlier CI failures were addressed: the privacy prefix collision is fixed; the delegated task-completion reply case passed locally after merging master's reply-recovery fix (#18465). Browser fixture corrections retain the original behavior assertions. Red tests reproduced the loading, navigation, resize and saved-plan regressions before their fixes. - Map, module-boundary, token-gate and added-line token checks passed. The full-tree token scan still finds pre-existing active-username fixtures; this PR adds zero forbidden token lines. - Greptile: **5/5**, final revision, all four review threads resolved. The final-revision PR CI workflow passed, including build, typecheck, all server/chat/workspace test shards, all eight E2E shards, and every Runner lane. GitHub reports 55 successful checks and three skipped checks. The separate **Paperclip Review** check remains queued with **Waiting for the assigned agent**; it has not produced an assessment and has not been bypassed. The branch has no merge conflicts. ## Risks - The default compact task omits completed transcript details when the saved result is sufficient. The reader must open history to inspect them. - Required structural data remains inside the initial reveal boundary. Slow server responses for that data remain visible latency. - Link positioning falls back after 15 seconds when its target never arrives. The conversation remains readable. - The measurements use a controlled local delay. They do not establish a deployed staging speedup. - No API, schema, authorization, or migration changes. ## Model Used OpenAI Codex, GPT-6, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (focused suites pass; the full default wrapper has the existing selection error documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (code CI passed; the separate assigned-agent Paperclip Review remains queued) - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
dce383c733 |
perf(server): stop recurring attention retention scans (#18388)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
b6b57079b9 |
fix: evaluate status cards on committed changes and deadlines (#18401)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
e40de0c9e8 |
perf(ci): refresh test weights and narrow graph dependencies (#18464)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors need fast, reliable CI feedback. > - Diff-aware CI selects test files and partitions them into runner jobs. > - Stale cost estimates put several slow suites on the same runner. Broad imports also couple small tests to large modules. > - This PR refreshes test costs, runs independent graph discovery in parallel, and narrows three dependency boundaries. > - Selection safeguards and CODEOWNER-only AWS routing stay in force. ## Linked Issues or Issue Description Refs #18398. This follows the audit of [PR #18427's verification run](https://github.com/paperclipai/paperclip/actions/runs/41169978784). The audit found 403 selected general-server files without timing weights. Three suites measured at 232s, 122s and 116s shared one shard because each received a roughly 2s fallback weight. Local graph discovery across four projects took 43.7s serially and 21.7s with two workers. A reverse-order repeat took 37.7s serially and 22.6s with two workers. Both comparisons produced identical selection results. This is a local measurement, not a claim about production CI speed. ## What Changed - Refresh general-server costs for 1,285 of 1,309 current files and serialized costs for all 163 files from two successful protected-master Cloud readiness runs. Retain prior costs for unmeasured files, including dedicated chat and native-runner suites. - Record passing server file costs, including collection, setup and environment preparation. A weekly hosted workflow validates trusted source runs and opens a timing-only PR. It tests shard coverage, pauses while a refresh PR is open, and never merges automatically. - Discover at most two project graphs concurrently in isolated, heap-limited Node processes. Cache repeated inventory reads within one selection. Worker errors and timeouts retain the full task plan. - Fetch only the merge commit and parents for the selector. A real shallow-clone regression proves the complete diff still includes more than 300 files, renames and deletions. - Share the identical log viewer and run details between both AgentDetail pages. The viewer test imports the shared implementation and runs each of its 13 distinct cases once. - Extract the pure API error class while preserving the API client's public exports. Import delivery work signals through their existing narrow DB export. ## Verification - PASS: `pnpm -r typecheck` and `pnpm build`. - PASS: 284 CI selection, timing, shallow-diff, sharding and routing tests. - PASS: 74 further cache, routing and graph-proof tests. - PASS: 94 targeted UI assertions across the viewer, run invocation, API errors/client and progress helpers. All 76 assertions in the 12 server suites made independently selectable also pass. - PASS: `pnpm check:token-gates`, module-boundary checks, YAML parsing and `git diff --check`. - PASS: [Full PR CI and required `ci / verify` aggregate](https://github.com/paperclipai/paperclip/actions/runs/41206683345), including all 12 general-server shards, all nine serialized shards, workspace groups, chat suites, Runner lanes and build checks. - PASS: [GitHub hosted real-workspace selector and cache/routing qualification](https://github.com/paperclipai/paperclip/actions/runs/41206683205). - PASS: Both Greptile and Paperclip reviews are 5/5 with no actionable findings or inline comments on `7dc65c51b8`. The PR has no merge conflicts. - PASS: The production refresh parser accepted all 107 structured file-cost records from a real successful server CI job, including setup/loading cost. - Local `pnpm test:run` completed all six chat shards successfully. The duplicate monolithic local run was stopped after the complete remote suite passed; it was not completed locally. The focused log-viewer command improved from 18.84s (26 duplicate variant cases) to 8.49s (all 13 distinct behavior cases). The dependency audit confirms 12 server files and the API-error test no longer need conservative always-run retention. Repartitioning the audited PR with the new costs changes the slowest shard from 810s to 591s of summed historical test execution (1,124 of 1,142 files measured). This is a workload simulation, not measured new CI wall time. ## Risks - Two graph processes use more peak memory. Each has a 2 GiB heap limit and a three-minute timeout. Any failure preserves the original task plan. Real workspace qualification runs in GitHub CI. - Historical logs contain test/hook costs only. New reporter records add loading/setup costs after merge. The manifests identify measurement coverage and source runs. These weights change placement, never permission to skip tests. - Runner load adds timing noise. The refresh uses medians and requires at least 90% measurement coverage in each lane. - GitHub may require approval to run checks on a PR created with `GITHUB_TOKEN`. Refreshes remain reviewable and do not merge themselves. - The shared viewer is a mechanical extraction. Both pages retain their existing exports, and the behavior assertions remain covered. AWS runner authorization is unchanged. ## Model Used OpenAI Codex, GPT-6, with repository inspection, reasoning, code execution and GitHub tools. The exact deployment ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks and six chat shards; complete suite passed in CI as detailed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b7f1c3769c |
feat(native): record verified provider-pack session admission (#18469)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote native sessions can use an installed pack or a retained earlier pack. > - The current app revision does not identify the pack that a session admitted. > - Core already verifies the selected pack and binds its launch artifacts. > - Normal run records lack a closed receipt for that admitted selection. > - This pull request records that receipt after session admission and attachment. > - Operators can compare the actual Core admission identity with their expected pack. ## Linked Issues or Issue Description Refs #17749 for protected provider-pack selection and #18343 for retained controller recovery. The related OpenCode producer repair is #18456. Those changes do not publish the closed run receipt added here. A search found no existing `nativeProviderPackAdmission` PR. **What happened?** Core can admit a compatible installed provider pack with a source revision that differs from the current app. A retained session can also keep its original pack. The normal run record does not show the admitted pack digest, proxy digest, and source revision. An expected launch path or current app revision cannot fill that evidence gap. **Expected behavior** The run has a closed, server-owned receipt after actual session admission and attachment. The first receipt remains immutable history. A separate current-proof state verifies only the actual current admission epoch. Retained recovery needs positive original provenance, and missing legacy evidence stays unknown. **Steps to reproduce** 1. Select and verify an installed pack whose digest differs from the current app pack. 2. Open the native session successfully. 3. Inspect the normal run profile for a bound admission receipt. **Paperclip version or commit** The original selected base was `8bcc5a01edd5d87540401521e6f30d3df43e1848`. The branch now includes exact actual prerequisite parent `3ee4c5b6138da82d0b1f51aa010540561bf43c22`; the PR diff against it is the five receipt, executor, test and documentation files. **Deployment mode** Core remote native execution with provider-pack selection and retained session attachment. ## What Changed - Retain the verified selected pack identity only after a backend returns the matching admitted session. - Publish `runnerProfileJson.nativeProviderPackAdmission` from the session callback after attachment succeeds. - Mark the receipt with `stage: session_admitted` and `authority: core_verified_selection`. - Store bound company, agent, run, session, runner, logical `runnerEnvironmentLeaseId`, pack digest, OpenCode proxy digest, and source revision fields. Retained attachments identify `inheritedFromRunId`. The logical Runner scope is distinct from the physical environment lease. - Lock the current coordinator claim before the profile write. Check owner, attempt, generation, controller boot, lease expiry, phase, and absence of an accepted result. Merge JSON fields without replacing other profile data. - Restore retained metadata only from the exact prior closed Core receipt, positively verified prior state and matching retained binding. Capture same-run prior provenance from the locked claim row before invalidation; cross-run recovery reads only the exact source. Legacy absence remains unknown while recovery continues. - Preserve the first receipt as history. Atomically mark a new opaque admission epoch pending in the existing mandatory claim transaction. Publish current verification only for the matching claim and epoch after actual attachment; changed or unsupported admission becomes unknown without blocking authorized execution. - Preserve synchronous session publication and quarantine ordering before the best-effort receipt write. Fresh replacement selects its newly verified pack; a captured retained-process flag cannot relabel the replacement with an older pack. Preserve the existing private pack authority even when a local dirty revision is ineligible for the closed receipt. - Add focused admission, recovery, identity, PostgreSQL ownership, lock, and JSON-preservation tests. Document the receipt contract. ## Verification - RED: the actual managed-backend path verified and admitted an installed pack but had no closed metadata. Failed open published no evidence. Additional controls fail when retained recovery invents missing history or dirty-revision eligibility gates the existing private authority. - RED calibration: removing mandatory pending invalidation fails the actual pre-admission control; reusing the retained flag for fresh replacement falsely restores the old pack. Removing the epoch fence fails the real PostgreSQL stale-callback case; reintroducing a receipt-specific execution error fails both authorized Runner/logical-lease replacement controls. - GREEN: 49 focused Core tests, all 33 admission identity/transaction tests (20 pure and 13 PostgreSQL), and all 23 existing private-authority tests passed on exact integrated head `165a9096cf3e0d34d2db1e6b52960ad4ac0ad132`. No all-skipped run is counted. - Final test-only head `11ac34baf00552a5d322b5f5f1a8e491859cacec` adds the positive executor-callback SQL-writer wiring control, exact session/identity/claim/epoch assertions, profile preservation, and null-session zero-write check. Removing only the production writer call makes it fail on a missing receipt. All70 focused identity/Core tests pass with this closure; production code, the13 PostgreSQL tests and the23 private-authority tests are byte-identical to the qualified165 source (106 distinct passing controls across the combined proof). - The PostgreSQL tests used an owned loopback-only cached Postgres 17 container with the supported `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` helper and aliases to current DB/shared source. The container was removed afterward. They cover immutable history, JSON preservation, stale claims and epochs, expired leases, terminal runs, changed/unsupported replacements, and a failed write retaining pending proof. - Historical pushed `f7f12fe` CI executed all 24 earlier admission tests, including nine PostgreSQL cases, in server5. Its warm-quarantine failure exposed the new hook-ordering issue fixed here. The same run also encountered a six-character UUID fixture-prefix collision and the independently repaired cached issue-link browser selector; those historical failures are not claimed as green. - Node 24.22.2 and Vitest 5.0.3. The credential-free normal QA runtime import preflight passes with networking disabled. Map contracts, token gates and `git diff --check` pass. - Local TypeScript validation still sees stale borrowed Runner/shared declaration mismatches outside the changed ranges. No diagnostics remain in the new helper or changed test/hook ranges; clean current-head CI is required. - Repository-wide typecheck, full tests and build were not completed locally. A full local executor attempt reached this host's unsupported native PostgreSQL listener and was interrupted; its result is not counted. Canonical run `41214512239`, attempt 2, is successful on exact final head `11ac34baf00552a5d322b5f5f1a8e491859cacec`. All 59 checks are terminal: 55 success, 3 skipped and 1 neutral; none failed or pending. The actual Linux suites execute all 33 admission tests (including 13 PostgreSQL cases) and all 1,022 executor tests successfully. The [original five failures and bounded unchanged-source controls](https://github.com/paperclipai/paperclip/pull/18469#issuecomment-6545593767) remain preserved. One failed-only request was made; GitHub expanded the reusable workflow call and dependents. No second request was made. Fresh exact-head Greptile review is 5/5, all review threads are resolved, and the PR is mergeable. - These local results claim no live provider admission, rollout or recovery. ## Risks - The authority is Core's verified selection plus successful session admission. It is not an independent provider statement or continuous runtime measurement. - Receipt fields cannot grant execution or replace the existing provider, process, lease, workspace, or recovery checks. - Legacy missing receipts and unavailable database reads remain unknown. The current release cannot label an older admitted pack. - The closed receipt requires a clean, 40-character hexadecimal source revision. Verified local `-dirty` packs still execute and retain their existing private recovery authority; their closed admission proof stays `unknown`, so it cannot certify an immutable source revision. - The first receipt is historical. Consumers must require current state `verified`; pending or unknown cannot certify the current pack. Changed Runner scope or pack evidence does not add execution policy. - A receipt save failure leaves the atomically invalidated state pending. The existing artifact authority, cancellation, workspace and process checks remain responsible for execution safety. - The receipt adds no manifest, environment, credential, or filesystem path to the run record. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, code editing, and tool use. The session does not expose an exact model build ID or context-window capacity. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
51ea624344 |
fix(server): reject a native run's own task status change (#18411)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs (the Paperclip Runner) report their outcome with
`paperclip_finish` or `paperclip_block`, and the status arbiter then
sets the task status
> - Every runner tool, `paperclip_finish` included, requires that the
task's `executionRunId` is the current run
> - When the run sends its own status change through `PATCH
/api/issues/{id}` (for example with `call_api`), the issue update clears
`executionRunId` and `checkoutRunId`, so the run removes its own
execution lock
> - After that, every tool call fails, the run cannot report, same-run
recovery spends its retries, and the task goes to `blocked`
> - This pull request rejects that status change with a 409 before it
commits, and the message tells the run to report with `paperclip_finish`
or `paperclip_block`
> - The benefit is that the run corrects itself in the same turn, and
the task does not get stuck
## Linked Issues or Issue Description
No public issue exists. This is the bug description.
**What happened?**
An agent on the Paperclip Runner followed task instructions that said
"set the task to `in_review` and keep it assigned to yourself" while it
waited on a scheduled monitor. It sent `call_api` with `PATCH
/api/issues/{id}` and `{ "status": "in_review" }`. The server applied
the change and cleared the execution lock. Then:
1. `get_task`, `set_task_monitor`, and `paperclip_finish` all failed the
run binding check. The provider saw only "Paperclip semantic operation
failed", or the monitor-wait message from `paperclip_finish`.
2. The `call_api` receipt for the PATCH showed `status: null, outcome:
unknown`, because the authority check after the response failed when the
lock was gone.
3. The session ended with no semantic result. Exact-checkpoint resume
ran out of retries (`native_session_retry_exhausted`), and recovery
would not replay the run because the mutation outcome was unknown. The
task went to `blocked`.
This occurred on each check where the agent tried the status change.
Runs that did not try it completed normally.
**Expected behavior**
A native run that holds the task's execution lock cannot remove its own
lock. It gets a clear, actionable error and reports its outcome through
`paperclip_finish` or `paperclip_block`.
**Steps to reproduce**
1. Assign a task to an agent that uses the Paperclip Runner (native
runtime).
2. While the run executes, have it send `call_api` with `PATCH
/api/issues/{id}` and `{ "status": "in_review" }` for its own task.
3. Look at the next tool calls of the run. They fail, `paperclip_finish`
cannot complete, and recovery blocks the task.
**Paperclip version or commit**
`master` at
|
||
|
DottaandPaperclip
|
3ee4c5b613 |
Make saved execution receipt recovery react to durable work (#18409)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
42fd568cf8 |
Drive issue recovery from committed changes and exact deadlines (#18402)
Reviewed and verified at
|
||
|
Dotta
|
12b0cc6304 |
fix(chat): recover committed native Agent Chat replies (#18465)
## Thinking Path > - Paperclip manages agent work and keeps its results available to users. > - Agent Chat needs a visible reply before the conversation can wait for more input. > - Native result commitment and reply presentation are separate obligations. > - A completed conversation can retain a valid final message without a saved reply. > - Existing reply recovery excludes these completed conversation turns. > - This pull request repairs that persistence gap through the existing recovery worker. ## Linked Issues or Issue Description **What happened?** A native Agent Chat turn completed successfully and saved its workspace. Its accepted result and final assistant message were durable, but no reply appeared in the conversation. The chat stayed active. The deployment hiring journey in #18351 then reached its deadline while waiting for the first reply. **Expected behavior** Recover the exact accepted turn's final reply once, then use normal conversation settlement. Keep later user input active. Do not run the agent again or replace a selected or deleted reply. **Steps to reproduce** 1. Create an active conversation with one admitted native turn. 2. Commit an accepted DONE result and a final-channel assistant message for that exact run, session and turn. 3. Finish workspace finalization before live heartbeat reply presentation. Leave no saved reply. 4. Run normal response recovery. Before this fix it finds no conversation candidate and leaves the chat active. **Paperclip version or commit** Reproduced with actual services at frozen master `8bcc5a01edd5d87540401521e6f30d3df43e1848`. The production predicate also exists in the deployed revision that exposed the failure. Related work: #18351. The broader receipt and recovery work in #18118 and #18409 does not touch these source files. ## What Changed - Discover unanswered committed Agent Chat turns through the existing native response presentation worker. - Recheck company, agent, responsible user, current execution and conversation generation under the normal transaction locks. - Require the accepted result digest and an exact final message from the same native run, session and accepted turn. Keep external publication authority separate. - Commit the reply and presentation receipt before normal conversation settlement. Later input stays active. Recovery does not repeat agent work. - Preserve skip decisions and selected or deleted replies. Add database controls for foreign identity, stale sessions, partial commitment, duplicates and ordinary internal task behavior. - Document the recovery boundary in the developer guide. ## Verification - Real-service RED/GREEN uses all 418 canonical migrations in isolated PGlite and the exact new regression fixture. Before the fix: zero candidates, no reply, recovery returns false and the chat stays active. After the fix: one candidate, one exact reply, recovery returns true and the chat waits. A repeat returns false and creates no second reply. - All 20 canonical regression cases pass locally through a fixture-only PGlite bridge. Only local database provisioning and `createDb` selection differ from the normal embedded PostgreSQL fixture. Production services, SQL queries, transactions, protocol validation and assertions remain real. Linux embedded PostgreSQL CI is required before merge. - Focused current-source no-emit typing passes for both production modules and the new regression file, with normal Express augmentation and source aliases. The broader local presentation/ownership and conversation-settlement PGlite bridge run was interrupted before it reported results; it is not counted as a pass. No repository-wide local build is claimed from borrowed workspace dependencies. Clean-build CI and canonical PostgreSQL consumer tests remain required. - `git diff --check` and the scoped source/test secret scan pass. No captured run data, credentials or private target identities are committed. - No live journey retry, deployment, configuration change or test-plan admission was performed. The original failed journey remains failed; this production fix still needs deployed qualification through the unchanged journey. - Current-head [canonical Linux CI](https://github.com/paperclipai/paperclip/actions/runs/41207018345) passed all 57 terminal checks with no failure or pending check. The new 20-case database regression actually passed in [server shard 8](https://github.com/paperclipai/paperclip/actions/runs/41207018345/job/124555006090), at 13:59:25 UTC (2,643 ms). Typecheck, build and the existing consumer lanes passed. Fresh [Greptile review](https://github.com/paperclipai/paperclip/pull/18465#issuecomment-6544208002) on exact `f32a432e10b377ae6277d6f728995d5fedcf9d7b` is 5/5; the review-thread inventory is empty with no pagination. No live deployment or journey pass is claimed by these checks. ## Risks Eligible active conversations can gain a previously missing reply. Recovery accepts only the exact saved final message under its accepted result and current conversation identity. It does not run a model, repeat hiring or change a prior test grade. Reply persistence precedes waiting-state settlement; existing conversation recovery retains ownership of a later interrupted settlement. No schema or public API changes are included. ## Model Used OpenAI Codex, GPT-6 family. The session does not expose a more specific backend model ID or fixed context window. Used reasoning, repository tools, TypeScript and real database test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
baf651c8d0 |
perf(heartbeat): drive retry promotion from durable deadlines (#18387)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
12d8c3043e |
fix(native): fence retries when replaying an audited Stop (#18457)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs keep Stop intent and retry state in durable records. > - A provider error can leave a failed retry with an earlier acknowledged Stop. > - The replay path returned the existing receipt before it closed that retry. > - This pull request applies the existing retry fence to the validated replay. > - The same Stop can settle without another provider cancellation or a new receipt. ## Linked Issues or Issue Description Refs #14733 for the related caller-bound Stop contract. This change covers a later retry that already has a Stop acknowledgement. The older CI-only #14743 is not a production delivery PR. **What happened?** A failed native run can have a retryable coordinator and an earlier audited Stop. A repeat Stop returns the old acknowledgement without moving the coordinator to `terminal_failure`. The final failed-to-cancelled transition requires that fence, so the Stop cannot settle through that path. **Expected behavior** A valid replay closes the same retry and keeps the original intent and acknowledgement. It does not send a second provider cancellation. Physical stop and workspace cleanup still require their own evidence. **Steps to reproduce** 1. Store a failed native run with a `retryable_failure` coordinator. 2. Store a matching run-scoped Stop with a dispatched acknowledgement and audit IDs. 3. Call `cancelNativeSession` for that same run and inspect the coordinator. **Paperclip version or commit** Reproduced on public master `f2db909393de124caf6a5380a52d4e0e1d621661`. **Deployment mode** Core native execution. The regression uses the real cancellation executor with controlled persistence and provider boundaries. ## What Changed - Reuse the existing guarded coordinator fence for a validated Stop replay. - Test repeated replay, unchanged receipts, one fence, and no second provider cancellation. - Reject foreign company, run, issue, scope, effect, and missing-audit receipts before any write. - Check that an acknowledged Stop with an unconfirmed remote lease still fails the physical-stop guard. - Document the separate retry fence and physical-cleanup requirements. ## Verification - RED on the selected public base: the new executor case returned success but left the coordinator in `retryable_failure` with a future retry. One case executed and failed for that assertion. - GREEN: 62 focused tests passed across `native-session-executor.test.ts` and `native-cancellation-request.test.ts`. These include ownership, caller conflict, changed-phase, and receipt controls. - Command from `server/`: `node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-cancellation-request.test.ts -t 'native session cancellation|caller-bound native cancellation|failed native retry caller admission'`. - Canonical Linux CI passed all 1,018 executor cases and all 32 caller-cancellation cases. The exact file completion lines are in [server shard 2](https://github.com/paperclipai/paperclip/actions/runs/41203989386/job/124545366271) and [server shard 10](https://github.com/paperclipai/paperclip/actions/runs/41203989386/job/124545366219). - Node 24.22.2 and Vitest 5.0.3. Map contract checks and `git diff --check` pass. - Local dependencies are linked from an existing checkout. Full local TypeScript validation reports older Runner/shared declarations that lack current Kimi and continuity APIs. The changed ranges have no diagnostics. The canonical clean CI Typecheck + Release Registry gate passed. - A broader two-file run was stopped when an unchanged integration case tried to start embedded Postgres and hit the host listen restriction. It is not a passing full-suite result. No host settings or other services were changed. - Repository-wide `pnpm -r typecheck`, `pnpm test:run`, and `pnpm build` were not completed locally. Current-head CI passed against clean source and dependencies. - Fresh [Greptile review](https://github.com/paperclipai/paperclip/pull/18457#issuecomment-6543772487) on `dbc1c1a5b71450194d568725592969660864cb5a`: 5/5 with no actionable findings. The review-thread inventory has no threads and no pagination. Full [current-head CI](https://github.com/paperclipai/paperclip/actions/runs/41203989386) passed: 59 terminal checks, 55 successful, 3 skipped, and 1 neutral. No failure or pending check remains. Merge status is CLEAN. No live recovery, rollout, or physical termination is claimed. ## Risks - A wrong or stale Stop must not close another retry. Existing run/coordinator locks, caller checks, receipt binding, and the final compare-and-set remain in place. - This fix changes logical retry settlement only. Provider termination, workspace preservation, lease cleanup, and continuation gates remain separate. - The regression controls persistence and provider boundaries. It does not establish the physical state of a remote provider. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, code editing, and tool use. The session does not expose an exact model build ID or context-window capacity. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ab51cabbbd |
test: await async outcomes and drive saved deadlines deterministically (#18416)
Reviewed and verified at
|
||
|
DottaandPaperclip
|
c2971badde |
perf(server): recover independent native replies concurrently (#18421)
## Thinking Path > - Paperclip manages work for AI agents. > - A native run can commit its result before its reply is saved. > - The response recovery worker saves replies from those committed results. > - It checks each candidate in series, so one task can delay unrelated tasks. > - This pull request checks up to four independent tasks at once. > - Each task keeps serial checks, and the existing coordinator owns retries and shutdown. ## Linked Issues or Issue Description **What existing behavior does this improve?** Recovery of committed native task replies. **Current behavior** The worker checks one candidate at a time in a batch of up to 25. A database lock or a slow read for one task delays replies for other tasks. **Proposed behavior** Check up to four company-and-task groups at once. Check candidates within each group in their existing order. Keep the current batch limit and company selection. **Reason and benefit** A ready task reply can be saved while another task waits for a database lock. The change reduces waiting under a reply backlog. **Breaking changes** Replies for different tasks can be saved in a different order. There are no API or schema changes. Other deadline workers keep serial dispatch. Related work: #17696 adds the recovery worker. #17501 reduces its read payloads. #18367 adds its timing spans. These changes are already merged. The PR search found no open change that adds this concurrency. ## What Changed - Add optional keyed concurrency to the shared scoped deadline worker. - Enable a limit of four for native response recovery, keyed by company and task. - Await all active checks before the pass ends. Retain per-candidate failure backoff, drain behavior and pending work. - Test concurrency limits, task order, company separation, drain/resume, shutdown, retries and idle wake behavior. - Add a PostgreSQL regression that holds one task's finalization row lock while a second task saves its exact accepted reply. - Document the worker's concurrency and sleep contract. ## Verification - The PostgreSQL lock regression fails with serial dispatch. It passes with this change. Both accepted replies are saved once after the lock is released. - `pnpm exec vitest run server/src/__tests__/scoped-deadline-work.test.ts server/src/__tests__/delivery-work-coordinator.test.ts server/src/__tests__/company-work-batch.test.ts server/src/__tests__/deadline-work.test.ts server/src/services/execution-preparation-timing.test.ts` — 55 tests pass. - `pnpm exec vitest run server/src/services/presentation-ownership-read.test.ts` with its documented dedicated PostgreSQL URL — 21 tests pass. The disposable database was dropped. - `pnpm -r typecheck` — passes. - `pnpm build` — passes. - One matched local replay with 25 older pending replies and one fresh reply used the real worker, a 10-connection PostgreSQL pool and 8 ms delay in each network direction. The saved fresh reply was first observed after 29.193 s with serial dispatch and 9.798 s with this change. All 26 replies were saved exactly once in both runs, with no worker errors. Both disposable databases were dropped. This is one synthetic workload pair, not a staging measurement. - `pnpm test:run` was attempted. Embedded PostgreSQL could not initialize its template after five attempts, so the broad run was stopped. The changed PostgreSQL tests pass on the separate local server. Remote CI is green. This PR does not claim a measured staging speedup. - The [first CI browser shard](https://github.com/paperclipai/paperclip/actions/runs/41162257463/job/124422204506) failed during company cleanup after its task assertions and empty-run check passed. Company deletion hit the `activity_log_company_id_companies_id_fk` constraint. The fixture uses paused process agents and reported no runs. The one failed-jobs rerun passed all 45 browser tests. All CI checks are now green. ## Risks - Four checks can use more database connections than serial recovery. The limit applies to this queue only; the batch limit stays at 25. - The shared helper has more dispatch logic. Its default remains serial. Tests cover task order, retry deadlines, drain and waiting for active checks during shutdown. - This improves reply recovery under backlog or lock contention. It does not establish a target for total agent-turn latency. ## Model Used OpenAI Codex, based on GPT-6, with code editing, tool use and local test execution. This session does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d5766e4074 |
perf(chat): scope task questions and batch inbox queue checks (#18422)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Standing chats show recent messages and current task questions. > - Recent-message pagination bounds history work, but it does not bound question work. > - The question read built the full company attention feed before it selected the chat's tasks. > - Each feed read also checked seeded queue membership with one insert per item. > - This pull request scopes question collection in the attention owner and batches queue membership checks. > - Chats keep their current question cards and initial layout while doing less work. ## Linked Issues or Issue Description Follow-up to #18358 and #18323. Related work: #18312. **What happened?** A hosted standing chat remained slow after recent-message pagination shipped. On commit `4eaa435bf4`, an embedded-browser reload made the conversation usable at about 8.6 seconds. Recent comments finished at about 1.9 seconds. The pending-question request held the initial reveal until about 8.4 seconds. That request called the complete company attention feed and then filtered its results. The separate badge summary also did broad work. **Expected behavior** Chat question cost should depend on the relevant tasks. Reading an existing queue must not require a separate insert for every member. Current access and decision policy must remain in force. **Steps to reproduce** 1. Create a standing chat with questions on tasks assigned to its agent. 2. Add many unrelated tasks and pending questions to the company. 3. Open the chat from a fresh application cache. 4. Compare recent-comment arrival with the pending-question response and content reveal. **Paperclip version or commit** Baseline: `909e29dd071b1aeeb24c2933af049da90a6066d3`. Its affected services match the observed hosted revision `4eaa435bf4`. ## What Changed - Add an internal task-question projection to the attention service. Filter pending interactions by company, task IDs and question kind before card enrichment. - Keep the same access, audience, dismissal, queue, snooze, retention, archive and training-example logic. Ranks are local to the returned question list. - Use the projection from the conversation receipt service. Keep current chat audience and session checks there. - Batch seeded membership inserts in groups of 500. Keep the queue lock, conflict protection, audit events and activity logging for each new membership. Existing members produce no new audit records. - Add real PostgreSQL regression cases and a reproducible paired benchmark. No schema migration or UI reveal change is needed. ## Verification - `pnpm -r typecheck` passed. The final server typecheck also passed after the queue change. - `pnpm build` passed. - Real PostgreSQL service coverage: 57 attention and chat-handoff tests passed. Seven queue-route tests passed, including concurrent reads, mixed existing/new items, duplicates, disabled seeds and audit preservation. - Red/green proof: the unrelated-attention regression fails with the baseline conversation implementation and passes with the scoped read. - `node scripts/check-module-boundaries.mjs` and `git diff --check` passed. - Full `pnpm test:run` was attempted and stopped with exit 130 after embedded PostgreSQL startup failures and 17 failures in the untouched native-session executor suite. It is not a local pass. The focused Docker/PostgreSQL results above completed before Docker Desktop became unavailable. A direct native PostgreSQL probe identified the host semaphore limit (`could not create semaphores: No space left on device`). No shared kernel settings or other test clusters were changed. [Full CI](https://github.com/paperclipai/paperclip/actions/runs/41162314995) passed on `2809f096b3fb2c01b44a96a1b82a27809c9359df`, including all eight browser shards. Final status: 55 successful checks/statuses, three skipped and one neutral. Greptile is 5/5 with no findings or unresolved threads. The fresh local browser retest is blocked because Docker Desktop cannot start and native PostgreSQL cannot allocate semaphores. The original hosted reproduction above is complete; no candidate browser pass is claimed. Local service benchmark: PostgreSQL 17.11, two relevant tasks, five alternating pairs per scenario, after warmup. These are local service timings, not a claim about deployed latency. | Fixture | Read | Baseline median (range) | Candidate median (range) | SQL calls | | --- | --- | --- | --- | --- | | 1,000 unrelated tasks/questions | Chat questions | 670 ms (632–1,905) | 12 ms (11–15) | 1,076 → 22 | | 1,000 unrelated tasks/questions | Company badge summary | 641 ms (622–717) | 167 ms (161–171) | 1,068 → 69 | | Small company | Chat questions | 38 ms (37–52) | 13 ms (12–16) | 52 → 22 | | Small company | Company badge summary | 34 ms (33–39) | 34 ms (32–40) | 44 → 43 | All question fields and ordering matched, except rank is relative to the scoped list. Summary fields matched except the generation timestamp. The interaction plan changed from 1,002 selected rows to two rows through the existing company/issue/status index. The benchmark loads baseline attention, conversation and queue modules together so the queue change cannot contaminate the baseline. ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-questions.ts \ --baseline-ref=909e29dd07 --unrelated=1000 ``` ## Risks - Scoped and complete feeds must share policy. This change reuses the same attention collection and finalization path and tests dismissals, named audiences, snoozing, archives and private access. - Batched inserts must audit only new members. The concurrent PostgreSQL test checks unique membership and one event/activity record per new item across batch boundaries. - Full inbox summaries still collect the company feed. The small-company summary timing is unchanged. Other chat requests and transcript hydration can still contribute to hosted latency. - The first reveal still waits for question cards to preserve layout. This follow-up has not been deployed, so its hosted effect remains unverified. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, code execution and browser profiling. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
492d537340 |
fix(native): recover controllers across webserver replacement (#18343)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native execution runs in a remote Runner while the webserver owns its controller. > - Encrypted PRP checkpoints and session snapshots already survive a webserver restart in Postgres. > - Ordinary warm and per-turn admission still needs local markers and supervisor metadata to find that state. > - Replacing the webserver with an empty native controller directory can therefore lose a usable retained session. > - This change adds bounded recovery metadata to the existing snapshot transaction and reconstructs local projections from the selected durable source. > - The benefit is native execution continuity across webserver replacement, with the existing ownership and agent lifecycle policies. ## Linked Issues or Issue Description **What existing behavior does this improve?** Native remote controller recovery and retained-session continuation during a webserver upgrade. **Subsystem affected** server/ — native runtime, checkpoint persistence, recovery scheduling and internal upgrade reporting. Product E2E qualification also changes. **Current behavior** Without both the local PRP marker and supervisor file, normal continuation can return before discovering the saved Postgres checkpoint. The snapshot already exists in Postgres, but its local reuse compatibility envelope cannot be reconstructed. **Proposed behavior** Select the existing durable source, validate its identity and ownership, restore its PRP state and supervisor projection, then use normal authenticated admission. Missing or inconsistent evidence holds recovery. It does not authorize fresh provider work. **Reason and benefit** A replacement webserver must recover active execution and later continue its retained session without the old server's native files. This is one component of deployment continuity. HTTP traffic switching, workspace storage migration and Railway rollout remain separate. **Breaking changes** No new public endpoint, schema migration or database pool. The private run-profile envelope is additive. Disk-independent rollback also requires a reader for the staged controller format and compact checkpoint part; the first production release containing these readers is the floor. Older file-backed sessions need an owner checkpoint or the bounded internal importer. **Additional context** Related foundations: #17401, #17916, #17254, #14621, #14622. The implementation contract and open acceptance gates are in [the dated plan](doc/plans/2026-10-30-native-controller-checkpoints.md). ## What Changed - Persist a versioned recovery envelope, at most 4 KiB, beside the redacted session snapshot in its existing transaction. Keep it out of public run projections. - Compare frozen admission separately from prepared attachment instructions during staged recovery, preserving exact template/identity/authority checks. - Bound the existing lazy checkpoint pool at three connections after measured two-slot burst backlog; keep request/renewal capacities and SQL operation counts unchanged. The tuning is a separate commit. - Discover and restore the exact selected retained source without local native markers. Preserve the 32-candidate window, progress barriers and current authority checks. - Fence snapshot and result writes. Acquire coordinator, issue and run locks in order and check expiry after lock waits. Combine snapshot ownership/read work into one statement. - Reuse existing recovery deadlines and drain/resume scheduling. Retained-owner waits preserve the original unstarted run and actual lease deadline. Stop, ownership and provider-progress fences precede deferral. Add bounded native evidence reporting, retained-source import and retention protection. - Admit a settled warm source with an authenticated idle goal-read suffix while retaining active-work and mixed-authority holds. - Permit lifecycle `turn.stop` checkpoints after accepted results under existing ownership fences. Claim the existing successor coordinator before retiring the old warm session, and stage shared attachments only after verified retirement. Preserve omitted isolated-home defaults and existing fresh skill-selection order during recovery without hiding real configuration changes. - Carry the exact durable six-field PRP binding during active recovery after a healthy warm handoff. Keep the Runner mismatch guard and preserve opaque attachment IDs. - Preserve an already verified active or idle warm process during remote preparation; revalidate its proof/liveness and retain later authenticated PRP attachment. Ordinary resumed leases retain suspension checks. - Drain retained source events under their original authority, reconcile immutable source accounting and block unknown source tool dispatch before target work. - Persist compact staged handoff intent through authenticated adoption and recover direct/listener command-loss windows without another Runner or duplicate attachment. - Rebuild suspended lifecycle receipts for resumed prepared tasks, and retry missing lifecycle reads after partial PRP materialization without overwriting contradictory evidence. - Add normal warm/per-turn diskless continuation coverage, real Postgres races and SQL-cost checks. Add a reproducible paired load workload. - Give only the explicit outage fixture a 15-minute provider idle window, with a red/green catalog regression preserving all production timing and lifecycle policies. - Publish the goal-less task-session pointer in the existing post-acceptance terminal checkpoint transaction, with current ownership, accepted-result, reset and newer-progress fences. Keep a private bound receipt so late completion cannot duplicate or resurrect publication. - Use a retained warm sandbox in the dedicated continuity fixture, and accept an exact output token followed by an explanation. Keep the strict identity oracle. - Extend the explicit Daytona cell to a five-minute active outage and another replacement after completion. Require retained provider identity, a new tool receipt and a useful follow-up result. Add an owned loopback Postgres E2E fixture for hosts that cannot start embedded Postgres. ## Verification **Merge recommendation — `fc68584ce6231a02e47ecd743f55b036ccea034f`: ready for the qualified native-controller component.** Full CI, fresh review, local application paths, controlled performance qualification and the strict live Linux/Daytona case pass. Independent resource cleanup is complete. Broader rollout, browser reconnection and the unexecuted transport/platform combinations remain integration gates. Integrated 484 passed full local recursive typecheck/build and ordinary active→completion→cache-loss→C component qualification. Fresh review then found a valid attached-wake failure: preparation appends attachment instructions before the verifier compares execution to the frozen admission. 9f4 fixes that by separately checking the pre-preparation admission and exact prepared execution. Both hashes are captured privately per admission; the full template, identity, configuration and authority checks remain. Its regression fails before the fix; 36 focused cases, 42 staged source/target cases and server typecheck pass afterward. The review thread is resolved. Exact-head full CI passes ([run 41168228822](https://github.com/paperclipai/paperclip/actions/runs/41168228822): 48 successful jobs, one intended skip); the complete `pnpm test:run` partitions are forced, with the five native suites in the Runner lane. This is not a claim that an unsharded local command passed. The fresh exact-head Greptile check succeeds with 5/5 and all six threads resolved. The original two-connection database comparison closes **steady PASS and burst FAIL**. Every relative CPU/WAL/request-p99 gate passes and all operations/ownership remain correct, but candidate arm 3 drains in 1,015.99 ms and baseline arm 4 in 1,059.67 ms against the unchanged 1,000 ms absolute gate. Those results remain failed. A separate 2→3→4→2 diagnostic reproduces drain times 1,051.57 / 956.79 / 882.02 / 1,065.72 ms: three is the smallest tested passing cap, and the repeated two-cap failure survives run-order/cache effects. Separately reviewable `fc68584ce6` changes the existing lazy checkpoint pool default to three connections. Request capacity stays ten and liveness two; idle release, shutdown fencing, transaction counts and pool count are unchanged. Thirteen DB lifecycle/cap tests and DB typecheck pass. Full recursive typecheck/build pass at exact |
||
|
DottaandPaperclip
|
f1d7808382 |
fix(tools): stabilize tied profile entry order (#18427)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Advanced YAML editor reads shared tool profiles through their owning service. > - The editor must preserve hidden settings and reject real concurrent changes. > - Batch-created entries share a creation time. Queries sorted only by that time can return the same entries in a different order. > - The editor can then report a false concurrent tool change before it sends a profile binding request. > - This pull request adds an ID tie-breaker to profile entry reads. > - The benefit is a stable saved configuration without removing the concurrent-change guard. ## Linked Issues or Issue Description Refs #18356. Related: #18168. **What happened?** The compact YAML browser case failed at its second save. Save stayed enabled and the profile binding write did not occur. The original CI artifact has no response bodies. It does not establish the exact entry order from that run. A controlled browser test reproduced this failure. It changed only the order of 30 equal-time entries in the second preflight response. The client reported `tools changed while you were editing. Reload saved before saving.` No binding request followed. A separate first-save response barrier showed that the pending-save guard worked. **Expected behavior** Reads of unchanged entries must have a stable order. Real entry changes must still trigger the existing configuration checks. Entry order does not set policy precedence; exclusions still win over matching includes. **Steps to reproduce** Create a profile with entries that share one creation time. Read the profile, change the physical row layout in a disposable PostgreSQL database, and read it again. The old service has no tie-breaker for these entries. The new regression fails against that service. ## What Changed - Sort list and detail entry reads by creation time, then ID, in the profile service. These reads supply saved configuration snapshots. Other reads retain their existing behavior. - Add one real PostgreSQL regression. It changes heap layout in both directions, checks stable list and detail values, preserves chronological ordering, checks company boundaries, and checks include/exclude results and a real entry change. - Include the exact test-only recovery prerequisite from commit `36c8d9ddd76d0b7ddb28a9a6c674d9e538f6df88`. Its file blob is unchanged. It removes a one-sweep fixture assumption while keeping bounded recovery assertions. No document UI or journey code is imported. ## Verification - Base: one frozen public master snapshot, `8c9dd1e4445e58cc9ca08076dfe9c0b27038d6ee`. The affected service, schema, projection, resolver, and save guard match the failed CI merge `94d98d5c75c8f331aad4bfab2111ee587b0c63fe`. - RED: the new real PostgreSQL test fails on the old service. Both physical layouts were verified before the ordering assertion failed. - GREEN: `cd server && pnpm exec vitest run src/__tests__/tool-profile-entry-order.test.ts src/__tests__/tool-profile-entry-validation.test.ts` — 18 tests pass with real PostgreSQL 17. - GREEN: `cd ui && pnpm exec vitest run src/lib/agent-configuration/save.test.ts src/lib/agent-configuration-yaml.test.ts --maxWorkers 1` — 51 tests pass. - GREEN: one focused Chromium treatment of the original compact YAML case passes. The treatment changes the actual disposable database heap after the first save. It does not transform HTTP responses. The 30 equal-time entries stay identical through the API, the second bind returns 201, and the original hidden-setting and disabled-Save assertions pass. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Local tests use Node 24, a macOS source server, Chromium, and an isolated loopback PostgreSQL 17 container. The server and container are stopped. Private traces and control source are retained outside Git. These controls do not prove a deployed Cloud result. - A review follow-up removes two unnecessary query changes. Fresh real PostgreSQL verification still passes all 18 tests. The list/detail queries and browser path are unchanged from the passing heap-reorder treatment. - Prior head `59edcade203d6a05923955f5a0118e62c7e549c5` had a separate browser failure in run `41169427774`: `disconnect-write-recovery.spec.ts:204` kept the original title after a mocked session lookup failure. Its relevant test and UI source are unchanged from the base. The cause is unresolved. The two removed query changes do not repair it. The original log and artifact are retained; no unchanged job retry was issued. On the current head, this case passes in 14.3 seconds. The compact YAML case also passes in 11.2 seconds. The later success does not establish the prior failure cause. - Final exact head: `81c584aeac116b9d7a126b55dc99d5f6c70c195f`. [CI run 41169978784](https://github.com/paperclipai/paperclip/actions/runs/41169978784) completes with success: all 47 workflow jobs are terminal, and all 57 PR check/status entries are green or accepted neutral/skipped states. Full typecheck, build, Runner checks, security checks, and `ci / verify` pass. They were not repeated as broad local commands because the shared host is under load. - The new PostgreSQL regression also passes in Linux CI in 911 ms. Server shard 12 has 1,839 passing tests. Browser shard 3 has 45 passing tests, including the original compact YAML case. - Fresh Greptile and Paperclip reviews both give this exact head 5/5 with no findings. All review threads are resolved. The branch is mergeable and clean. ## Risks - Entries with the same creation time now use ID order. Their prior relative order was undefined. Entries with different creation times keep their existing order. - The query still sorts the same rows and adds one key to the sort. It adds no query, polling, schema change, or migration. - The configuration equality and concurrent-change checks are unchanged. This fix does not ignore real tool changes. - The original CI response order is unknown. The regression proves a permitted ordering defect and the matching failure mechanism, not the unique historical cause. ## Model Used - OpenAI Codex, based on GPT-6, with reasoning, source inspection, terminal tools, and browser test execution. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
59da15066a |
feat(runner): deliver native Kimi Code (#18391)
Deliver native Kimi Code setup, exact models, scoped credentials, verified packages and runtime packs. Preserve current upstream release attestations and restore verified same-run remote retry dispatch without weakening process ownership. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
100b364264 |
Preserve selected Google services when setup resumes (#18048)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connections control which external services an agent can use. > - Google Workspace setup lets a person choose services and read or write access. > - A draft lost that choice after a declined or expired OAuth handoff. > - Finish setup could then show all services with write access instead of the saved narrow choice. > - This pull request saves the exact draft choice without treating it as granted access. > - The benefit is safe setup recovery that preserves the person's choice and existing account limits. ## Linked Issues or Issue Description Refs #15545. The duplicate search found no open PR for Workspace draft consent recovery. Related #18000 concerns satisfied gateway access requests, not saved OAuth setup intent. **What happened?** A cold Workspace draft resumed with the new-account service defaults. The selected request lived only in short-lived OAuth state. Denial consumed that state. The picker ignored draft connections. **Expected behavior** Finish setup restores the same person's saved service choice. It does not use another person's choice, a broad profile hint, or a credential grant as draft intent. An old draft with no reliable choice asks the person to choose services. **Steps to reproduce** 1. Start a new Google Workspace connection. Select Gmail read-only. Turn off Drive and Calendar. 2. Continue to Google, then decline authorization. 3. Close the setup page. Return through Finish setup. 4. Open the service picker. Before this fix, all three services can be enabled with write access. **Paperclip version or commit** Reproduced in tests on master `820c458465fa40137c81fe510f3e6dc573b4f895`. **Deployment mode** Authenticated and local source builds. The source regression uses real PostgreSQL fixtures and the production React setup components. Live provider acceptance of this repair is still pending deployment. ## What Changed - Save validated, non-secret draft intent before managed or customer-owned OAuth. Bind it to company, connection, actor, grant kind, and subject. - Restore that intent after denial, state expiry, and cold Finish setup. Keep active reconnect bound to its saved grant. Do not change new-account defaults. - Preserve agent-to-human setup handoffs. Consume only the addressed person's reviewed proposal. Keep other people's proposals. - Lock and merge the current draft before saving intent. Return 409 if its credential identity or method changed during setup. - Preserve the latest saved choice across overlapping customer-owned starts. Merge both current configuration records under short locks. Reject stale client-binding changes. Keep each OAuth state's own requested scopes. - Add shared, service, handoff, picker, and cold-page regression tests. Update the connector contract and existing feature plan. - Route the default ungrouped test/release paths through six sequential exact-cover chat partitions. Preserve remaining server/native coverage, explicit grouped CI, selection-first execution and shared-database paths. - Correct the retention test driver so pending owned real I/O does not advance logical deadlines. Pump zero-duration connection timers and add a held-I/O regression; keep production scheduling and original timing assertions unchanged. The shared helper is needed by the server and picker to validate the same narrow binding. It does not resolve credentials or grant access. No production schema migration, new route, provider scope or deployment configuration is added. The test-harness changes do not change production timers. ## Verification ### Current candidate: source qualification passed; live acceptance pending - Current published candidate: `82d13d4d698a5579378b5d506cb7a1f9cb64b5c8`, tree `78d6a821ce53f481fe3e9991edce56f0acffb22f`. It combines the Workspace repair with launcher commit `bf29c9` and retention-test commit `e25987`, then integrates master `493e229d93ba7798e80152f4549e2e9177ce50ae`. The integration brings 40 upstream commits across 713 files. One launcher conflict keeps upstream validated CI selection and its selection-first path, then applies six default chat partitions only without a selection. The authored PR diff has 16 files, 833 additions and 100 deletions; it adds no authored lockfile or `.github` changes. The public head is confirmed at this revision. Fresh CI and review passed as recorded below. Source qualification is complete using exact-tree full CI and the local results below. The separate local whole command was interrupted by host disk protection and is not called a pass. Live repaired-draft acceptance remains pending deployment. - Exact clean `82d13d4` passed one Node 24.19.0 builtin-only run: 41/41 cases (37 launcher and four CI-workflow cases), zero failures/skips. HEAD, tree and relevant file hashes stayed unchanged and clean before, at start and after. This proves launcher fixtures, not real chat execution, current typecheck/build, the dependency-backed selection graph, or full feature acceptance. - The new default ungrouped whole command uses six sequential exact-cover chat partitions, then the remaining server suites, workspaces and serialized suites: 28 expected invocations. It preserves all native suites, including upstream usage coverage. The plain ungrouped release launch path also changes. Explicit grouped CI, validated selection and shared-database execution stay unchanged. Launcher hard RED had three expected body failures and two unchanged passes; the repaired focused five and final 37 passed. A later 41-case staged-merge pass is retained separately; the clean 82 run above is the frozen proof. - Fresh [CI run 41153113944](https://github.com/paperclipai/paperclip/actions/runs/41153113944) completed successfully on October 31 at 01:14:53 UTC. Current head has 52 successful check runs, four conditional skips and no pending/failing checks; legacy Snyk also passed. The skips are test-file selection, the optional paired benchmark and two Storybook checks. The full server/chat/workspace/serialized matrix, all eight e2e shards, normal build, build-gap typecheck, Runner checks and final aggregate gates passed. Build and all three chat job logs identify checkout `e5e831fd3a8534d0d0f1e892118c0f8fbe12dd92`, merging this head into `493e229d93`. Its Git tree is exactly `78d6a821ce53f481fe3e9991edce56f0acffb22f`, identical to the clean candidate. The six real chat partitions passed 314, 358, 273, 239, 321 and 311 cases: 1,816 total. Other cases shown as skipped in each partition were selection exclusions, not extra passed cases. This is current tree-equivalent CI proof, not a pass of the separate default ungrouped local whole command. - Fresh [Greptile summary](https://github.com/paperclipai/paperclip/pull/18048#issuecomment-6473856992), edited October 31 at 01:05:04 UTC, explicitly reviews exact head `82d13d4` at 5/5 with no actionable code issue. Its exact-head check passed. The prior overlapping-start discussion is resolved. This review is source-only, not provider or live acceptance. - Exact clean `82d13d4` passed full recursive typecheck and normal `pnpm build` on the independent darwin/arm64 snapshot. Typecheck finished on October 31 at 01:27:33 UTC; build finished at 01:28:33 UTC. Both exited 0. Normal native preparation and code signing passed. Source HEAD, tree and tracked/untracked state stayed unchanged and clean. `pnpm check:token-gates` also passed all token, palette and ClipLab checks on this candidate. The earlier forced-clone `ENOSYS` preflight remains recorded as setup failure, not an assertion result. - The unchanged default `pnpm test:run` started at 01:51:33.796 UTC on clean exact `82d13d4`. All six real chat partitions passed: 358, 314, 273, 239, 311 and 321 cases, or 1,816 actual cases. The original FIFO and ambiguous-retry cases both passed under their unchanged 15-second limits. General-server then reported 37 completed file rows with 5,438 passed cases and five default remote-continuity skips, including all 498 tool-access service cases and all 83 connection-route cases. At 02:39:44 UTC, the unchanged 5-GiB host-disk guard terminated the owned run. The child ended with SIGTERM and the container/harness exited 143, with no OOM or recorded assertion failure. Only seven of 28 RUN invocations started. Retention had no completed local row. Later invocations did not run. This whole local command is interrupted/incomplete, not passed. - A coverage check found no default suite missing from current full PR CI. The maintained launcher tests prove exact coverage for all 12 PR server shards, chat and five native suites; all configured workspace projects; and all nine serialized shards. The final Runner CI shard executes the five native server suites. The 41 local launcher/workflow cases cover six sequential chat children, per-child isolation, native retention, fail-fast propagation and unchanged explicit selection/shared-database paths. The actual six-chat run above exercises that changed default path. The remaining unqualified local difference is uninterrupted sequential whole-command composition and resource accumulation, not omitted test selection. Current exact-tree full CI supplies broad source-test qualification; no local whole-pass claim or provider acceptance is inferred. - A task-owned remote runner was considered to complete that separate composition check. Its provider rejected the requested allocation with per-sandbox limits of 8 GiB RAM and 10 GiB disk. No sandbox was admitted and no remote tests ran. No tests, timeouts, assertions, database synchronization, source or provider settings were changed to avoid the capacity limit. Historical a0 failures remain below. No deployment or live repaired-draft acceptance has run. ### Bounded follow-up on the prior source - On a0, one finite phase-traced chat diagnostic executed exactly the two failed cases. Both passed (1,374 and 1,767 ms); 1,814 other cases were filtered, not passed. This did not reproduce a blocked await or establish the original monolithic timeout cause. No assertion or 15-second limit changed. - The retention diagnosis preserved an initial zero-body selector failure, then corrected only the selector: one original case passed in 783 ms, with 42 filtered. Trace showed the old settling helper advancing logical time while real I/O was active. A new held-I/O regression then failed hard because the old helper advanced 1,000 ms. This proves a test-driver defect, not the precise cause of the old 5,340 ms full-run gap. - The first manual-clock repair failed the complete 44-case file: 43 passed, one body hit 15 seconds and cleanup hit 30 seconds. Its planned mutant did not run. Source then identified a fresh PostgreSQL connection's zero-duration timer. The corrected driver pumps zero-duration timers while initial readiness or tracked I/O is pending, without advancing logical Date. That pre-integration file passed 44/44 with no skips. A separate scratch-only 5,000 ms normal-continuation mutant failed the original hard `5000 < 3000` assertion; its expected failure is not a test pass. The original page-count, fairness, expiry and quiet assertions stayed intact. The authored repair does not change the production worker. The merged master changes an older retention case and dependencies; these pre-integration results are not a complete 82 retention or whole-suite pass. ### Historical qualification: SHA-bound, not 82 acceptance - Historical a0 candidate: `a0f9089fc1ff466bf4e1eeddc5b1a910d09e380b`, tree `b9d73713319dcd52b9eaa6053008b5190e43fdf1`. It adds the overlapping-start repair to parent `6c18`, which already merged pinned master `1511c92965325249ed6b52dd5be52eb6d736fbfa`. That candidate changed 12 files. Its CI and review passed; its local full qualification remained failed/incomplete. - Greptile's previous `6c18` review found P1 [Latest service choice gets lost](https://github.com/paperclipai/paperclip/pull/18048#discussion_r4425965828), confidence 4/5. Two hard-assert regressions reproduced it on untouched `6c18`: the already-bound path overwrote the later choice, and real first-use client stamping also restored the old choice. Existing OAuth states/grants were not widened, but a later setup could restore an older request. The new repair covers both write boundaries without holding locks over external work or copying another attempt's scopes. - The complete affected service file passed 498 tests with no failures or skips. All 31 required controls ran, including both unchanged RED cases, 14 new client-drift cases, seven earlier identity guards, reconnect/cold-resume cases and upstream transaction tests. The tested three-file patch is byte-identical to this commit; the connector contract update is separate. Independent scoped review found no additional actionable issue. Fresh Greptile reviewed exact `a0f` at 5/5 with no actionable findings. The previous P1 thread is resolved. - On exact `a0f`, the five shared/UI files passed 312 tests. An initial complete connection-management run then failed: 74 passed, eight timed out at the unchanged 15-second limit, and one AI-account verification returned 422 instead of 201. The Google addressed-user handoff passed. The container exited 1 without OOM and stopped before chat/token/whole tests. This material failure remains retained and its cause unclassified; the later controlled recovery below is not a causal explanation. - One diagnostic executed the first failed gateway case and the two AI cases with temporary phase-only tracing. All three passed at 8.516, 5.511 and 3.523 seconds; 80 other cases were filtered. Original hard assertions and timeouts stayed unchanged. No timeout or suspected late provider cleanup reproduced, so the complete-file failure remains unresolved. Vitest exited 0; the harness exited 1 because its trace parser missed ANSI prefixes. All 203 events were recovered from retained logs without rerunning tests. Instrumentation was removed. This is not acceptance. - Historical a0 CI [41097596364](https://github.com/paperclipai/paperclip/actions/runs/41097596364) completed successfully: 52 successful checks, two expected Storybook skips and no failed/pending checks. Legacy Snyk passed. All 48 CI jobs passed, including all 1,816 chat cases and explicit passing lines for the original 20 local failures. [Server 11/12](https://github.com/paperclipai/paperclip/actions/runs/41097596364/job/124207469276) explicitly passed the complete 83-case connection file in 129.49 seconds, including all nine local failure titles. The actual checkout was synthetic merge `06851da6eb969d63e1c7dc9f7c95f08d09fa4ff1`, combining `a0f` with `aa58fa107389e8b9ae5949f7ad815943dfd94292`. This is not standalone local `a0f` qualification and does not erase the nine local route failures. - Linux admission uses Node 24.22.2, pnpm 9.15.4 and Rust 1.97.1. Frozen offline install, exact native hashes and real migrated PostgreSQL 18.1 passed. Native/protocol/build/lock inputs are identical between `6c18` and `a0f`; validated cache reuse retains the actual `6c18` native-build provenance. The selected packaged release binary is not called a fresh `a0f` build. Database synchronization settings stayed on. This admission alone does not explain the route failures. - A fresh Mac snapshot of exact `a0f` passed frozen install, workspace prerequisites, full sequential recursive typecheck and normal `pnpm build`, both on their first attempt. Typecheck completed at 17:43:05 UTC, build at 17:44:43 UTC; both exited 0 with clean tracked/untracked source. Pinned Node 24.19, pnpm 9.15.4, Rust 1.97.1, package concurrency one, Cargo jobs one and a 12 GiB V8 heap ceiling were retained. Largest-process maximum RSS was 7,668,154,368 bytes for typecheck and 10,038,837,248 bytes for build, not aggregate tree peaks. The first build correctly withheld its freshness stamp because 125 ignored, pinned Copilot assets appeared during preparation; exact path/hash analysis explained this. The normal build then recorded a genuine stamp and reused Cargo outputs in 1.18 seconds. No manual stamp, source workaround or gate retry. This is darwin/arm64 static/build proof, not Linux packaging/runtime or full tests. - The historical a0 candidate's token checker passed all four gates across 1,410 files, run directly with pinned Node 24 against the script behind `pnpm check:token-gates`. - The single controlled complete-file comparison passed: baseline `6c18` 83/83 in 287.07 seconds, then candidate `a0f` 83/83 in 293.03 seconds, with zero skips. All nine earlier failure titles passed on both. Both child/container exits were 0, no OOM/interruption; source and native/resolver/PostgreSQL guards matched. This ran only after all owned Mac compilation ended, with fresh processes/databases and identical four-CPU-quota/6-GiB/overlay inputs. No filtering, instrumentation, assertion/timeout/source changes or retry loop. This establishes standalone complete-file recovery, not the cause of the earlier eight timeouts/AI 422. - The first unchanged full `pnpm test:run` on exact clean `a0f` ran from October 30 at 18:01:43 UTC to 21:06:19 UTC. It failed naturally with test and container exit 1, without OOM. General-server reported 1,257 passed, one failed and seven skipped files; 28,633 passed, one failed and 103 skipped tests. Only the first of 22 invocations ran. The later four groups and 21 invocations did not run. The sole failure was the unchanged chat-shard CLI meta-test: its nested projectless pnpm lookup hit the offline network guard. This was a failed full gate, not Google provider acceptance. - A finite environment pair reproduced that one meta-test failure, then passed the same unchanged case after supported `corepack prepare pnpm@9.15.4 --activate` as the existing non-root node user. The cached pnpm payload was identical. Only the missing node-user last-known-good record changed. Core source, dependencies, lockfile, fixtures, assertions, timers and offline guards stayed unchanged. This isolates the nested collection defect; it does not explain the earlier route failures or complete the full gate. - The corrected unchanged full command ran from October 30 at 21:29:03.500 UTC to October 31 at 00:02:44.207 UTC on the same exact clean `a0f`. It failed naturally with exit 1, signal null and no OOM. Source stayed clean before, at start and after. It retained two visible CPUs/quota, 6 GiB, private IPC, normal overlay, matching native inputs, synchronized real PostgreSQL 18.1, and the default five groups/22 invocations. Only the first general-server invocation ran; the other 21 did not run. General-server reported 1,265 files: 1,256 passed, two failed and seven skipped. It reported 28,737 tests: 28,631 passed, three failed and 103 skipped. The two chat cases emitted `Test timed out in 15000ms` at `chat-channels.integration.test.ts:77310` and `:81790`, with durations of 15,045 and 15,058 ms. These were the durable pre-run FIFO notice and ambiguous GitHub text restart/retry cases. The retention case emitted `expected 5340 to be less than 3000` at `skill-version-retention.test.ts:593`. Its real duration was 1,727 ms; 5,340 ms was the recorded fake-time directory-page gap, not a deadline timeout. The timeout stacks do not identify the blocked internal await. The cause of these failures remains unclassified. Inside this same failed run, the complete 83-case connection file and 498-case service file passed. The former Corepack meta-case also passed in 6,396 ms. These component passes do not waive the failed/incomplete full gate. No rerun, cancellation, timeout/assertion change or full-pass claim. A later fresh two-case phase trace passed both chat cases; it did not reproduce or establish the original timeout cause. At that checkpoint, the PR stayed draft. - Prior `6c18` targeted checks passed 930 tests across eight files and token gates. Its selected queue group passed all 53 cases, including the original 20 failures and 12 new named controls. Backlog cases took 10.49 and 8.45 seconds under the unchanged 15-second limit. Another 1,763 chat cases were intentionally filtered. This is historical targeted recovery, not a full pass or proof of the new repair. No timeout or assertion was relaxed. - Prior `6c18` full local qualification failed during serial typecheck: server TypeScript was killed with code 137. Docker confirmed OOM in the 6 GiB container; full tests and build never started. The failure is retained. The new Mac resource plan avoids repeating that undersized run and does not change the shared Docker VM. - Prior CI [41089793558](https://github.com/paperclipai/paperclip/actions/runs/41089793558) completed successfully for synthetic merge `a73bffcfa3ede051961bcabcfa576c703d10c932` (`6c18` plus `abc270ee4c03e13b4d6eaad156a41b4ebed4bb0a`). All 48 CI jobs passed; all 1,816 chat cases passed, including explicit log lines for the original 20. Check runs were 51 success, two expected skips and one Greptile failure; legacy Snyk passed. These results do not qualify `a0f` or erase its local route failures. - Red proof on the unmodified master baseline: cold picker and Finish setup restore broad defaults; managed and customer-owned drafts lack saved intent. The stale setup tests also fail before the lock guard. - Live staging baseline `b25f3e2af8e77d882cd22d36ca55640e6468e46f` also reproduces the ordinary failure: select Gmail read-only only, cancel at Google, reload the connector catalog, then Finish setup. All three groups return enabled at read/write. Authorization was declined; no new Google access was granted. This is baseline failure evidence, not a deployed-fix pass. - Historical affected suites: 800 tests passed before the final lock guard. The final focused service and handoff cases passed 126 tests after that guard. These are not results for the integrated candidate. - Historical `pnpm -r typecheck` and final lock-guard server typecheck passed. The historical a0 candidate passed the full recursive gate separately, as recorded above. - `pnpm build`: passed again on the final lock-guard source in commit `1c31cde0ed199fbf4e41b7649590554f7886dfc2`. - Historical `pnpm check:token-gates` passed; the historical a0 candidate's direct checker pass is recorded above. - `pnpm test:run` on macOS failed on the frozen commit. It exited 1 after 9,432.41 seconds. General server results: 1,195 files passed, 10 failed, eight skipped; 26,706 tests passed, one timed out, 157 skipped. Nine setup hooks timed out while starting embedded PostgreSQL. The retained log records host-wide semaphore exhaustion (`semget`), not disk exhaustion. One unchanged publication-text performance case also timed out. Its cause is not established. Later workspace and serialized groups did not run. This is a failed and incomplete local gate. - The independent full-source Linux run also failed and was incomplete on the same frozen commit. Native and real-database admission passed. Chat integration reported 20 failures in 1,593 tests. Tool-access passed all 479 tests. Native-session-executor passed all 965 tests. Docker interrupted the run after about 48 minutes on October 29. The container exited 143 and was not OOM-killed. The run produced no final group totals or detailed assertion summary. Shell trap metadata reports zero, but this is not a valid test-pass result. Source and evidence remain unchanged. The focused diagnosis below retains those failures; their cause is not assumed from the interruption. Neither historical local full run passed. - A fresh focused replay on the frozen source then executed exactly those 20 case bodies. All 20 failed; this was not a setup-hook failure. The two backlog cases reached the unchanged 15-second timeout. The 18 FIFO cases failed on missing notices or receipt rows. All 18 FIFO cases passed when run alone in a fresh process and database. This supports an order interaction, not a complete cause or baseline conclusion. A separate two-backlog control used RAM-backed temporary storage and real PostgreSQL with synchronization settings on. One backlog case passed in 10.65 seconds; the other still timed out at 15 seconds. No source, assertion or timeout was changed. The full local gate remains open. - Earlier mixed-revision run: it began before the final guard and imported older source before collecting later regression cases. It failed those new cases with that older cached module. This run is retained, not counted as candidate acceptance. The fresh focused run passed all seven stale-setup cases. No case was skipped to hide a failure. - Historical remote gates for `1c31` were terminal and clean: 53 successful contexts and two expected Storybook skips. Main CI [40720536514](https://github.com/paperclipai/paperclip/actions/runs/40720536514) passed. Greptile reviewed exact head `1c31cde0ed199fbf4e41b7649590554f7886dfc2` at 5/5 with no actionable findings; the separate review job also passed, and there were no inline review comments. These results do not qualify `6c18`. - CI source qualification: the chat jobs checked out synthetic merge `ef36746b3d2ec52ba5727dcbe4da265a03421fe4`, which merges the PR into `7d75ae0569e41786640cf09647ae8849411d2f22`. All 20 locally failing case names have explicit passing CI log lines. These passes do not establish standalone local acceptance of `1c31`. The merge adds two upstream Teams tests, so it collects 1,595 chat cases instead of 1,593. Neither backlog/FIFO assertions nor their timeouts changed. - Independent source review found an earlier credential-identity race. The guard and seven deterministic cases resolve that finding. A later integration review missed the separate overlapping-start race. Both RED failures and that review miss are retained. The narrow follow-up repair passed 498 service cases, independent source review and fresh remote CI/5-of-5 review; full local qualification remains incomplete. - Source qualification is ready for merge. Live repaired-draft recovery, delivery and Google approval are not established by these results. The local whole command remains explicitly incomplete. ## Risks - Historical drafts without a matching intent now need an explicit service choice. This is deliberate and prevents silent broadening. - A concurrent credential or method edit now returns 409. Reloading lets the person review the changed setup. - Overlapping starts keep separate requested scope sets. Updating draft intent does not relabel an earlier OAuth state's consent request. - Saved intent is not consent. Provider-reported scopes and current action policies still control discovery and dispatch. - No migration is needed. No existing token or grant is rewritten by saving draft intent. - Default ungrouped launch paths now execute six chat children in sequence and 28 whole-command invocations. Exact coverage, local launcher contracts and all six real default chat partitions passed. A complete uninterrupted local whole-command run remains unqualified because of host capacity. - The retention test-clock repair depends on owned-I/O and zero-duration connection-timer progress. The controlled regression, sensitivity mutant and current exact-tree CI protect the original assertion. It did not reach a completed row in the interrupted local whole run. ## Model Used OpenAI Codex, described by the runtime as GPT-5 based. The exact runtime model ID and context-window size are not exposed to this task. Used reasoning, code execution, browser verification, and independent delegated review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, including the runtime metadata limit - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket or instance identifier - [x] I have run tests locally and they pass — exact clean 82 passed 41 launcher/workflow cases and all 1,816 real default chat cases, full recursive typecheck, normal build and token gates. Full exact-tree PR CI passed. The separate local whole `pnpm test:run` was interrupted by disk protection and remains incomplete; it is not claimed as a local whole pass. - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green — current exact-head checks passed; the verified CI checkout tree equals clean 82. The separate interrupted local whole command is disclosed above. - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — fresh exact-82 summary and check passed; the earlier finding is resolved. This is source review, not acceptance. - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8ff812cdde |
feat(chat): deliver proactive task messages to agent chats (#18374)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent Chat gives each person a persistent conversation with each
agent.
> - Tasks and routines produce results outside that conversation.
> - Those runs need a safe way to bring a requested result back to the
person.
> - The server must select the agent's own chat and check the source
task's access rules.
> - This pull request adds proactive chat messages and unread
indicators.
> - People can see the result, open its source task, and reply in the
same chat.
## Linked Issues or Issue Description
**Subsystem affected**
Server, native runner tools, shared contracts, database, and UI.
**Problem or motivation**
An agent can report progress on a task or comment on an authorized task.
It cannot send a routine result to its own standing chat with the
responsible person. A morning review routine therefore cannot leave a
visible digest in that person's chat.
**Proposed solution**
Add `send_chat_message({body, idempotencyKey})` and `POST
/api/agents/me/chat/messages`. The server selects the company, agent,
and responsible person from the active run. It saves an attributed
comment and audit record in one transaction. The chat shows a
source-task link and per-person unread state.
**Alternatives considered**
A destination parameter on a generic comment tool would expose
unnecessary routing authority. A new delivery worker would add another
durable delivery path. This change uses a dedicated tool and the
existing comment and live-event paths.
**Roadmap alignment**
This extends the experimental Agent Chat capability in `ROADMAP.md`. The
existing `enableAgentChat` setting controls it. Related public work:
Refs #13282, Refs #13283, and Refs #14408. Searches for proactive chat
and unread chat found no duplicate PR.
## What Changed
- Add a focused messaging service shared by the native tool and HTTP
route. It checks current run ownership, execution mode, recipient
membership, task access, and task privacy.
- Reuse the canonical company/agent/person conversation. Save the
comment and audit record together. Keep task completion and chat
execution separate from delivery.
- Add source-task provenance and source-scoped retry keys. Identical
retries return the saved receipt. Changed content returns a conflict.
- Enforce the conversation's fixed agent at the shared comment-write
boundary. Keep normal reply cancellation and session checks.
- Add chat unread state, accessible agent dots, aggregate Chat
indicators, and visible-message read cutoffs. Refresh through existing
live events.
- Add the request-key constraint, native tool contracts, documentation,
capability maps, stories, and regression coverage.
## Verification
- Local delivery/security proof: all 38 proactive-message cases pass.
Eight credential-scope regressions and four quota regressions fail
against the prior source. The repaired delivery and canonical
cross-issue cap suites pass all 49 cases; all 31 OpenAPI tests, server
typecheck, and capability-map checks pass. The shared per-run cap covers
native and HTTP sends, mixed ordinary mutations, concurrent last-slot
contention, and retry/recovery without double charging.
- Credential tests preserve the authenticated person and authorizer;
reject bridge/test credentials and persisted skill-test runs after
task-mode changes; and recheck API-key revocation, scope, and
responsible person after destination locking. The earlier
delivery/native-authority suites passed all 74 cases before the four
quota additions.
- Four PostgreSQL contention cases cover native and HTTP sends plus
retries during a native chat mutation. All four deadlock against the
earlier source and pass with destination-before-source/run locking.
Recovery and human-reply delivery enqueue only the human comment; a
proactive digest never wakes the chat agent.
- Passed 108 companion conversation, native-authority, and OpenAPI
tests; 112 runner catalog/workflow-report tests; 47 completion-delivery
tests; and 25 project-handoff tests. Focused runner tests cover native
tool availability without generic API tools.
- All 327 affected UI tests, UI typecheck, token gates, and catalog
checks pass. The read hook retries at most three times, respects server
retry delay, coalesces only seen cutoffs, recovers on reconnect, and
cancels on scope changes.
- Real browser journeys with a deterministic process agent verify
routine-to-chat delivery, first-chat creation, unread state across
reload, the source link, secondary chat navigation, and a follow-up
reply referring to the digest. This does not qualify live GitHub
retrieval or model output quality.
- The Storybook showcase uses production components with deterministic
network fixtures. Rendered interactions pass for the landing, ordinary
read success, transient read recovery, and persistent failure after
bounded retries. Both Chat navigation variants update from two unread
chats to one while only Kodi's dot clears. Mobile and light variants
were reviewed. The existing Coordinator Recents label contrast finding
is unchanged; no Linux visual baseline archive exists.
- Current master's independent sandbox migration retains 0418.
Regenerated the feature migration as 0419 from the combined schema and
retained exactly the newest five snapshots; numbering/safety, replay,
and schema-drift checks pass.
- Workspace typecheck and build passed before PR preparation; canonical
server/UI typechecks passed again on the combined tree. Migration
`0419_slippery_impossible_man.sql` passes safety and schema drift checks
on disposable PostgreSQL, including interrupted index repair, concurrent
migrators, one-time owner read-state backfill, preserved receipts, and
two SQL replays. A default five-second local snapshot comparison timed
out; a diagnostic 30-second run passed without drift.
- Earlier failures are retained in the record: full local `pnpm
test:run` and broad shards encountered embedded-PostgreSQL startup,
timing, and provider-recovery failures. Thirteen selected
external-channel cases pass in isolation on the original and changed
sources; an unordered comment-row assertion now explicitly orders both
reads. The earlier route registry, native tool count, runner catalog,
and accessible Chat badge-name failures were fixed and passed full
remote CI on `920b3bb7db`.
- Repeated managed-skill server8 failures came from CI's newer synthetic
merge: its Cloud guard correctly rejected the old local-disk fixture.
Master independently repaired the fixture in #18385; this branch keeps
that repair unchanged. All 57 managed-skill and Cloud prerequisite cases
pass on the combined tree, and that fixture has no remaining PR diff.
- Current reviewed head `dc16f314739a6d977bed21cb1446d0e80a541578`:
Greptile 5/5 with no findings, Superagent security assessment passed,
advisory Paperclip Review 5/5 with no findings, and zero open review
threads. All 55 current-head check runs pass, with five expected skips
and the advisory review neutral at 5/5; Snyk also passes. No checks are
pending or failing. Protected squash auto-merge is enabled and is
waiting for required code-owner approval. A normal exact-head merge was
refused by the base-branch policy; no admin bypass was used. The
previous head's Superagent quota finding is fixed and covered above.
## Risks
- Chat remains company-visible. Delivery rejects a source task with
private-task, project, or ancestor restrictions.
- Read state uses a comment timestamp cutoff. The server retains
database precision and never moves a read receipt backward.
- The new unique constraint changes comment query planning. An existing
regression now uses an explicit row order. The migration builds the
unique index concurrently, then takes a brief table lock to attach it.
It uses the existing top-level migration and interrupted-index recovery
path.
- The upgrade backfills missing chat-owner read-state rows at the latest
existing agent comment. Existing read receipts remain intact. New chats
can still be unread before the human participates.
- Quota observation commits atomically with delivery. A rejected send
rolls back its attempted denial audit row and any first-chat creation;
the canonical warning remains in server logs, and previously spent quota
remains enforced. Identical/conflicting retries and receipt recovery
spend no extra slot.
- This version delivers in-app only. It adds no external-channel, push,
or email delivery.
- Local full-suite acceptance remains limited by the failures recorded
above. The feature stays behind `enableAgentChat`.
## Model Used
OpenAI Codex. PR preparation and CI work used `gpt-6.1-sol` with tool
use and code execution. The earlier implementation also used Codex, but
its exact model ID was not recorded in the visible session. The
context-window size was not provided.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run focused tests locally and they pass; the broad local
failures are recorded above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
Devin FoleyandPaperclip
|
87ae20ae65 |
fix(server): reconcile failed sandbox cleanup with signed absence (#18407)
Reconcile failed ephemeral Cloud leases through a signed cleanup-only absence observation when local historical inspection is unavailable. Preserve failure outcomes, exact resource bindings, retention and row-version fences; grant no run-continuation authority. All 54 current-head checks/statuses passed. Greptile is 5/5, Medium, with no findings or unresolved threads. The 38 focused tests pass; both new reconciliation cases fail without the fallback. Isolated baseline and branch idle suites each pass all 225 cases. The original broader local teardown timeout is retained in the PR, and the full local run remains in progress. Cloud protocol support is already serving in staging. Exact candidate composition and hosted provider sleep/wake qualification follow this merge. Production has not been changed. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
9e904ee519 | perf(server): overlap independent heartbeat preparation reads (#18372) | ||
|
DottaandPaperclip
|
91261d5a03 |
fix(server): preserve budget history for deleted resources (#18389)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Budget views show saved policies, incidents and spend for each company. > - A policy can remain after its agent or project has been deleted. > - The overview currently stops when it cannot resolve that old resource. > - A current agent's budget then becomes unreadable during normal setup. > - This pull request preserves the historical row in display summaries. > - Admission and policy changes still require a real company-owned resource. ## Linked Issues or Issue Description **What happened?** The budget overview throws `Agent not found` when a saved policy references a deleted agent. One old resource prevents the company from reading its current budgets. The same problem affects historical project policies and incident summaries. **Expected behavior** The overview should retain historical policy and incident IDs and spend. It should display `Deleted agent` or `Deleted project` for a missing resource. Read access must not grant permission to execute work or change a budget for that resource. **Steps to reproduce** 1. Create an agent and a budget policy for it. 2. Permanently delete the agent after its other dependencies have been cleared. 3. Read the company's budget overview. The existing service throws instead of returning its remaining current and historical rows. **Paperclip version or commit** The isolated repair starts from merged commit `d9354bbfc35ef74333e90e596ce21dc8e59149b9` and integrates the exact merged baseline test fix `8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57`. It also integrates the actual merged Antigravity auth fix `18b1637b6371a6a6adade857e2a73654ee626fef` (#18254). The diff against the selected merged parent remains two budget files. **Related work** Earlier reports and implementations: #11058 and #4333. This extracts the product prerequisite from #18351 so it can qualify independently of that journey's other dependencies. The change contains only the budget service repair and four database regression cases. ## What Changed - Resolve deleted agent and project scopes only when building display summaries. Preserve their original IDs, policies, incidents and accounting records. - Reject a saved policy or incident that resolves to another company's resource. Propagate other read failures. - Keep missing-company resolution, work admission and policy changes strict. Validate company, policy and incident scope bindings before either incident action. A budget raise requires a live company-owned resource. Historical dismissal may proceed only for a genuinely deleted owned agent/project, and must reject live foreign or mismatched scopes. - Add four canonical PostgreSQL regression cases: deleted agent history, deleted project history, missing-scope admission and budget raises with safe historical dismissal, and foreign/mismatched references. The history cases also verify current ledger spend and unchanged stored rows. ## Verification - Real-service RED/GREEN against official PGlite with every canonical migration: the original overview throws for deleted history; the repair returns the historical IDs, incidents and spend while retaining current budgets. - Mutation regression RED/GREEN: missing-resource budget raises must reject before writes; owned deleted-scope dismissal must reject its linked pending approval, retain policy/agent/ledger data and incident history, and remove the incident from the active list. Live foreign and mismatched scopes reject both actions without mutations. - Current integrated head `b3434f5ad05fd3067379352700c36bfe826179c4`: all four canonical PostgreSQL regressions pass in [server 2/12](https://github.com/paperclipai/paperclip/actions/runs/41142588583/job/124361194077), and all five existing budget-service tests pass in [server 7/12](https://github.com/paperclipai/paperclip/actions/runs/41142588583/job/124361194080). Both use the CI merge against selected parent `18b1637`. - Clean-build declaration typing passes in the current CI typecheck gate. The earlier local source-alias no-emit check also passed, including the canonical actor argument. The local worktree borrows compiled dependencies from a sibling checkout; a later real-service import was blocked by its stale `sandboxCreationOperations` export, so clean-build CI is authoritative for the final integration. No production workaround was made. - `git diff --check` passes. Broad repository typecheck, tests and build run in current-head CI rather than repeated locally. Native local PostgreSQL could not start because this host exhausted its semaphore pool; no global IPC settings or unrelated processes were changed. - One unchanged Runner lane retry was authorized after a probable intermittent close-settlement fixture failure. The [original failure](https://github.com/paperclipai/paperclip/actions/runs/41142588583/job/124360234111) occurred in `omitted-context` cleanup. A [fresh passing Linux lane](https://github.com/paperclipai/paperclip/actions/runs/41142552195/job/124360110386) passed the same case and all 1,998 tests with byte-identical entire Runner and adapter-utils trees, lockfile, root package, scripts and workflows, against the same merged base. The fixture has a five-second close allowance with a 2.5-second suspension reserve; a host stall is not proven. No budget path or changed close input was found. The [single targeted retry](https://github.com/paperclipai/paperclip/actions/runs/41142588583/job/124365812609) passed the same case in 1,034 ms and all 1,998 lane tests; dependent verification also passed. All 57 current PR checks are terminal and green. - The product helper requires all current CI and clean review before merge. Actual QA budget-overview proof will run on the combined final app before hiring qualification. No deployed success is claimed. The older `b4863dd` artifact was never deployed; selected parent `18b1637` adds sandbox/lifecycle changes and migration 0418, so whole-app identity is not claimed. The budget module and relevant budget contracts remain unchanged, and final hiring requires the combined final app with explicit source identities. ## Risks Budget summaries now return historical rows whose resources have been deleted. These rows use explicit display names and preserve their accounting data. They do not create a resource or relax work admission, authorization or budget grants. Historical incident dismissal changes only that incident and its linked approval; it does not resume work or raise a budget. Cross-company references still fail closed. There is no schema migration and no new background work. ## Model Used OpenAI Codex, GPT-6 family. The session does not expose a more specific backend model ID or fixed context window. Used reasoning, repository tools, TypeScript and database test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (the PR description documents the display-only history and strict-mutation contract) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
45a3a92e6d |
fix(environments): persist sandbox ownership before creation (#18357)
## Thinking Path > - Paperclip manages companies and the agents that do their work. > - Remote execution creates provider resources that need a durable owner. > - The server could create a sandbox before it saved its lease. > - A failed lease write could leave cleanup data only on the old web server disk. > - This change saves a company-scoped operation before creation and transfers ownership atomically. > - Recovery finds the original Daytona resource or records an explicit manual reconciliation outcome. ## Linked Issues or Issue Description **What happened?** A sandbox could exist without a lease after a database failure or company rebind. Failed cleanup recording could then create a local fallback file. Replacing that web server could lose the cleanup reference. **Expected behavior** A failed admission write makes no provider call. An interrupted creation retains a committed operation. Recovery must not create a second sandbox or destroy an attached lease. **Steps to reproduce** 1. Run the disposable PostgreSQL cases in `server/src/__tests__/sandbox-creation-operations.test.ts`. 2. Reject the lease insert after the controlled provider creates its resource. 3. Start a new receiver process with a different empty web disk. It must find the same resource and create one cleanup owner. Related ownership report: Refs #14672. This change addresses new allocations before lease persistence. It does not replace existing lease recovery. ## What Changed - Add an idempotent migration and a small creation-operation table. Retain original company, environment and run identities across deletion. - Commit creation intent before run, ad-hoc lease, and plugin task replacement calls. Fence duplicate requests and ambiguous lease commits. - Retain a known returned task resource ID in the existing operation before attachment. Preserve manual reconciliation and never delete an ambiguously attached resource. - Keep company and run authority checks and atomic lease transfer with the lease owner. - Reuse Daytona names, labels, read-only inspection and claimed cleanup. Keep provider payload handling in the provider. - Retain nonsecret credential identity hashes. Refuse recovery lookup under a changed credential value. - Record manual reconciliation for missing/deleted or explicitly inactive saved credentials, versions, and provider vaults. Keep temporary resolution failures retryable. Never reactivate a disabled credential. - Use transaction signals and concrete deadlines. Failed records back off independently. Manual reconciliation stays durable and quiet. - Remove new orphan-file fallback writes from the covered paths. Keep legacy file replay and its tests for cutover. - Replace obsolete fallback-producer assertions with normal-entrypoint durability tests. Keep cleanup, diagnostics and warm handoff coverage. ## Verification - [Independent current-head verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534744320) passed 190/190 combined tests, 40/40 incoming TestOps tests, full workspace typecheck, and build. PR head `958af986130f25e136065ad30d1fdd843c46255c` has tree `e6c3e5b02182d5941a70ccf5bf55638df3c2e28d`, identical to tested combined commit `b9d7a9c2e9b30f0153685fb24a5f70b56132201f`. Real PostgreSQL, compiled Rust Runner and controlled object providers were used; external-S3 integration remains outside this local proof. - Current head `958af986130f25e136065ad30d1fdd843c46255c` normally merges master `8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57`. The sole conflict was the maintenance test: a parameterized case now preserves both legacy-local `local_storage` and valid shared-S3 `provider_mismatch` checks, including the parent local receipt, exact version identity, missing-snapshot/no-seed refusal, and unchanged persisted row. The three master TestOps files remain intact. No production, schema, migration, or package changes were made after the fully qualified prior head `0b4d05505e`. Independent review approved the resolution; the full managed-skill-content file and Cloud guard controls pass 58/58 (105.11 seconds). [Required PR CI passed on this exact merge head, first attempt](https://github.com/paperclipai/paperclip/actions/runs/41136987412): 54 successful checks and three intended skips. The additional Paperclip review is neutral at 4/5; two external Superagent checks (Contributor trust and Security Scan) remain in progress with zero annotations and no assessment, so they are not passing scan evidence. Greptile reviewed exact head `958af986130f25e136065ad30d1fdd843c46255c` at 5/5; the final thread audit has zero unresolved threads. All completed evidence below retains its recorded revision scope. The additional Paperclip Review on `958af98613` is neutral at 4/5 with a Ship verdict and no must-fix or should-fix item. Its optional observation is that a generic create rejection remains pending/manual even when an operator believes allocation did not occur. This change preserves uncertainty because the generic create-error contract does not prove provider absence; no new error classification or provider-specific compensation was added. - [Independent final-source verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534328247) passed all 189 real-PostgreSQL/Runner and controlled-fixture cases, full workspace typecheck, and build. The integration commit `c8411a8528386f4824b340ec5fdaaed1efb9ea69` has the same source tree as final PR head `0b4d05505e` (tree `bda65efc14b4c904d0f589154db77bb160641c8c`). Controlled object-provider coverage passed; the separately guarded external-S3 suite was not run because the bounded local MinIO pull was unavailable. This includes the two final compatibility fixes. - The first final-head CI attempt exposed one E2E timing failure in `disconnect-write-recovery.spec.ts:181`: the title stayed unchanged through the 15-second session-recovery poll. The same original case passed in 13.8 seconds on immediately prior head `7ec06f77b7`, and all UI/session/E2E source is identical across those heads. The failure log and CI artifacts are retained; no cause is claimed. One failed-job-only retry is authorized on this evidence, without changing a timeout, assertion, or production behavior. [The single failed-job retry completed successfully](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2): all 35 E2E shard cases passed (6.6 minutes), including the original session-recovery case in 14.0 seconds. The initial failure remains disclosed and its cause remains unproven. - Final head `0b4d05505ee0aa72f2981cdff74761406960ec73` adds two independently reviewed integration fixes after the complete 7ec CI matrix exposed two deterministic failures. Server 9/12 failed legacy local-storage maintenance because the CLI reached the new Cloud server startup guard; server 5/12 found the Daytona package version did not match its bumped manifest. The failed run is retained at https://github.com/paperclipai/paperclip/actions/runs/41131204203. It is not passing evidence and was not blindly retried. - Maintenance now constructs the existing explicit storage dependency after loading the required persisted Cloud snapshot, and passes it to both maintenance owners. It can verify legacy local storage before provider cutover while normal Cloud server startup remains guarded. The original case reproduced red locally, then the complete managed-skill-content and managed-cloud-prerequisites files passed 57/57 (32.05 seconds), including missing-snapshot refusal, unchanged stored configuration, and existing server guard controls. Daytona package version now matches manifest 0.1.10; the complete bundled package identity file passed 14/14 (3.98 seconds), and explicit manifest/lifecycle controls passed 2/2 (328 intentionally filtered; 3.42 seconds). An initial workspace-filter command matched no project and is not counted as pass evidence. The final diff changes no sandbox runtime or schema behavior. [Full CI is green on this exact final head](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2): 56 successful checks, three intended skips, one additional incomplete/neutral Paperclip Review, and no pending or failed checks. The neutral review is not a passing assessment. [Greptile reviewed the exact final head at 5/5](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6529396969), with no new actionable issues and all review threads resolved. - Final combined head `7ec06f77b7c14a745c569d18401f16df3d26b626` includes master managed-skills migration 0417 and shared-storage prerequisites from `198218bf55` through normal merges. Sandbox creation moves to generated `0418_curved_lady_mastermind.sql`; its idempotent SQL is byte-for-byte unchanged. The regenerated 0418 snapshot contains both sandbox creation operations and the master skill mutation receipts, and points to the master 0417 snapshot. Master SQL and journal history remain intact. Exactly five snapshots remain, 0414 through 0418. The only sandbox-specific changes since the approved credential policy are migration metadata/number and the matching test/document references; the sandbox runtime files are unchanged. - [Independent final combined verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6533960026) passed on integration revision `e6f05b4cf0cd`, containing final sandbox head `7ec06f77b7` and actual master `198218bf55`: migration/declaration checks, all 158 real-PostgreSQL/Runner and controlled-fixture cases (251 seconds), full workspace typecheck (81 seconds), and build (35 seconds). - Final combined migration proof: fresh real PostgreSQL has both tables. A separate original-0417 sandbox table with a retained pending row survives two applications of the renamed 0418 SQL unchanged, with the expected primary/index and foreign-key structure. Migration numbering/safety checks pass. The normal runtime migration reapply case passes 1/1 (20 intentionally filtered cases; 46.69 seconds). The first temporary qualification harness stopped before any database assertion because its external `.ts` file selected CommonJS; the `.mts` harness corrected that setup and passed. Independent review approved the exact metadata and reference diff. Earlier full suite and live-provider results below retain their stated revision scope. - Credential-policy head `c606d473b03709f27577a55a1dfc0b0d3b4359fd` handles the remaining inactive-credential review warning. The existing boundary accepts only concrete secret-owner 422 codes and exact no-details disabled/coming-soon vault messages. Unknown 422, authorization errors, provider wrappers, and temporary database/provider failures still propagate. Independent source review approved this three-file correction. No generic resolver, schema, provider protocol, retry cap, or activation behavior changed. - Creation/cleanup qualification at `c606d473b0`: all 39 cases pass against real PostgreSQL (81.27 seconds). The new cases use actual disabled/archived saved secrets, a valid disabled version, and provider vaults created ready before normal API updates disable them or mark them coming soon. Each remains manual through restart/repeated scans, with zero provider lookup and no retry deadline. The independent plugin idle blocker remains intact. The temporary 503 control retains retry ownership and later recovers one cleanup owner. Before this correction, all five inactive-state cases remained pending; that failed evidence is retained. The prior 18-case known-ID task proof below belongs to `237144fcf6`, whose helper and task path are unchanged by this correction. - Known-ID head `237144fcf69a249c0e0310563428c0f55ed6d55c` also addresses the independent task-replacement reliability warning. The existing creation operation now saves only the known nonsecret returned resource ID before expiry validation and attachment. The write is scoped to operation/company/original environment and allows only a missing or matching ID. It changes no ownership state and copies no resolved configuration, credentials, or provider payload. Independent source review approved this four-file fix. - The complete real-PostgreSQL task suite passes 18/18 (9.52 seconds). Rejected attachments retain the ID through manual reconciliation and repeated scans. A real committed attachment followed by an injected lost response retains the same active lease and attached operation, with no release or second create. Same-ID retries work; conflicting IDs and foreign companies cannot change it. The initial three regression cases failed because the ID was absent, before the fix. - The shipped Daytona manifest does not declare task support, so it cannot enter this generic replacement path. The dispatcher requires both declared task support and an environmentTask worker method. Generic drivers without creation lookup still use manual reconciliation, with a known returned ID retained when the database write can commit. No automatic lookup capability or unconditional compensation was added. - Previous boundary-fix head `8ba77def151ea85eeae2db7693254db7f87673b0` fixes the fresh review finding about missing credentials. Only known secret-owner 404 results at the recovery secret-resolution boundary select the existing manual outcome. Provider inspection and transient database/provider errors remain retryable. Independent source review approved this change. - Secret-boundary focused acceptance at `8ba77def15`: all 34 cases pass against isolated real PostgreSQL. The complete creation suite passes 16/16 (100.75 seconds); the complete cleanup coordinator suite passes 18/18 (99.83 seconds). The new cases use normal saved-secret creation and deletion, confirm zero provider lookup, restart/repeated scans, no automatic deadline, preserved independent plugin idle blocking, and no additional idle blocker after the controlled worker supplies its quiet proof. A temporary 503 resolution failure preserves pending ownership and later recovers one cleanup owner. No additional live provider resource was created. - Failed attempts remain part of the evidence. The initial red run reproduced the pending-operation bug and also exposed a numeric-deadline test harness mistake, which was corrected. A later combined run reported an existing replacement-receiver failure and a new idle-fixture assertion failure; it was interrupted while the next test file remained in imports. An isolated diagnostic confirmed that the manual state was correct but the installed fixture plugin correctly blocked sleep without its independent quiet proof. The corrected test retains that blocker assertion before supplying the controlled worker witness. The unchanged replacement-receiver case passes in the final complete 16-case run; its earlier failure is not counted as a pass or assigned a proven cause. No timeout or production idle guard was relaxed. - Greptile reviewed previous combined head `7ec06f77b7c14a745c569d18401f16df3d26b626` at 5/5, with no unresolved review threads. Its CI produced the two concrete integration failures addressed above. The earlier CI and live evidence below remain historical evidence, not a claim that the current full gates have completed. - Master integration update: merged `dbff3747b8` and then latest master `76fde891cc` without rewriting branch history. Only `doc/DATABASE.md` conflicted; the sandbox ownership, object-backed run evidence, and cloud configuration sections all remain. The master merges changed no sandbox implementation, migration, or provider code. Migration numbering/safety checks pass and exactly five schema snapshots remain. At merge head `f5d23d4a59828835d01c150f83fa2645fb2d3bec`, the two focused creation/coordinator suites pass all 32 cases against isolated real PostgreSQL (64.57 seconds). Those merge-head checks precede the missing-credential fix below. The earlier live Daytona proof still covers the unchanged normal creation/lookup/teardown path; the new missing-credential branch is qualified with real PostgreSQL and controlled provider hooks. The optional Superagent Security Scan was inconclusive on prior head `7ec06f77b7`: its pinned PR merge-base checkout failed or timed out, with a neutral conclusion and zero annotations. An earlier scan could not start within 120 seconds; a bounded rerun request returned HTTP 404. These scanner failures are not passing security-scan evidence. On final head `0b4d05505e`, the additional Paperclip Review check is neutral: its assigned review execution did not complete and produced no validated assessment. Fresh Greptile 5/5 and independent source review are separate evidence. Previous-head required CI, independent review, and the live Daytona qualification below passed. - [Complete previous-head CI](https://github.com/paperclipai/paperclip/actions/runs/41105726052) passed at `0b5cbb763ab8faa25aa0ff1d8a405cdc0d3db409`: 53 successful checks and three intended skips, including typecheck, full tests, build and E2E. Greptile is 5/5 on this head and both original findings are resolved. - Initial head: 200 PostgreSQL runtime, creation and coordinator tests passed. The corrected head also passes all 32 creation and coordinator tests, including deletion before/during lookup, a failed lookup after deletion, restart, settled polling and actual idle-safety checks. Cases cover failed admission, failed lease writes, concurrent recovery, lost provider replies, ambiguous transfer retries, company rebind, environment deletion, stopped runs, lookup backoff, restart, drain and settled SQL quiet. - Separate receiver processes: passed. A controlled HTTP provider retains the resource while three independent receiver processes use two empty web disks. Recovery creates one cleanup owner and never repeats creation. - Fresh PostgreSQL migration and two additional applications: passed. The pending creation row survives unchanged. - Server TypeScript and SDK/shared build checks pass after the corrections. Migration numbering/safety checks passed. Exactly five schema snapshots remain. - Daytona adapter: all 330 tests passed, including stable-name lookup, ownership checks, deletion and the updated manifest version. An earlier run exposed the version expectation and it was corrected. Previous-head CI passed. - Live Daytona acceptance passed on `0b5cbb763a`: isolated real PostgreSQL, normal runtime acquisition through independent receiver processes, actual Daytona plugin/SDK 0.203.0, one disposable Ubuntu 24.04 sandbox (1 CPU / 1 GiB RAM / 1 GiB disk). A forced lease insert failure retained the committed operation; a fresh receiver with empty disk found the same sandbox, repeated recovery retained one cleanup owner, and the actual claimed cleanup sweep expired the lease and resolved the operation. Independent SDK lookup returned typed not-found; ordinary recorded-ID inspection returned destroyed. This successful attempt made one create, one recovery lookup, and one teardown. Two earlier failed harness/provider-confirmation attempts are disclosed in the [live acceptance record](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6530313054). The coordinator independently queried all three exact resource IDs through the SDK and confirmed typed not-found for each. - Complete CI-affected heartbeat/plugin warm-task suites passed 21/21 after test corrections. An unchanged workspace exposure case passed in isolation after the prior CI shard reported an unhandled readiness rejection. Complete previous-head CI and refreshed review passed. ## Risks - Deleted companies retain an explicit manual reconciliation outcome. Recovery does not retry missing credentials or insert a lease for a deleted company. - Only Daytona declares automatic creation lookup. Other uncertain providers retain `needs_reconciliation`. A missing creation name alone cannot prove that a late allocation will never appear. - Pending operations use the existing process-local signal and startup recovery model. This does not add cross-replica wake delivery or prove deployed sleep. - Legacy direct provider probes and custom-image setup use separate lifecycles. They are outside this replacement of the run-lease orphan-file fallback. The normal Test path that acquires a runtime lease is covered. - Existing orphan files still require the separate cutover inventory and settlement. This change does not delete them. - The schema change is additive. The new SQL can be reapplied. Historical SQL and journal entries are preserved; only the oldest generated snapshot is pruned. ## Model Used OpenAI `gpt-6.1-sol`, with tool use, code execution and repository review. The harness did not report the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge The required branch status checks (`ci / verify`, `ci / e2e`) and all test/build jobs passed on this head. The non-required Superagent Contributor trust and Security Scan checks are still running without findings; they are not claimed as passes. No required check or branch rule is overridden. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
81ecaf0654 |
fix(server): bound failed lifecycle verification (#18383)
Failed lifecycle verification could allocate another test environment every minute indefinitely. Count failed host verification tests separately from claims, persist a repair wait for confirmed configuration failures or three generic failed tests, and clear it through authorized retry or execution configuration changes. Preserve typed temporary execution-target failures through adapter checks so timeouts, transport failures and interrupted probes keep automatic recovery. Retain existing readiness, authorization, termination and cleanup gates. Document the policy and add real PostgreSQL persistence and actual OpenCode adapter regressions. Verified 398 focused tests (one Windows-only test skipped on macOS), repository-wide typecheck and build, structural checks, fully green current-head CI and Greptile 5/5 with no unresolved comments. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8595efc20b |
fix(evals): support official TestOps job metadata (#18385)
## Thinking Path > - Paperclip manages AI agents and verifies real work through Product E2E tests. > - The fixed Chat feature workflow uses TestOps to select and report one backlog case. > - Official registration exposes a workflow-query URL and a Branch parameter. > - The existing reader accepts only workflow-file or numeric URLs, and the admin launcher requires an operation parameter. > - These assumptions reject valid official metadata before the feature can run. > - This pull request supports the exact observed form while preserving source, selection, and once-only execution checks. ## Linked Issues or Issue Description Refs: #18322, #18375. **What happened?** The second zero-model bootstrap stopped with a generic job-mismatch code. Ordinary TestOps UI inspection then showed an official workflow-query URL outside the accepted literals and a Branch-only job. The exact first failed predicate in that bootstrap remains unknown. **Expected behavior** Accept that exact fixed-workflow URL. Preserve the existing master Branch value when the job has no operation parameter. Keep rejecting foreign URLs, unknown parameters, wrong branches, and conflicting operation IDs before mutation. **Steps to reproduce** Pass the official query URL and Branch-only metadata through the existing bootstrap, selection, and admin-launch owners. The new pure controls exercise those actual helpers without a CLI, backend, or model call. **Paperclip version or commit** This change starts from `0e15c92073a09e50df99190a37f69a86286b989b` after #18375. **Deployment mode** GitHub Actions and the protected TestOps reporting environment. ## What Changed - Share one exact URL predicate across bootstrap, evaluation selection, and admin preparation. Add only the observed official workflow-query literal. - Support Branch-only, operation-only, and Branch-plus-operation jobs with strict definition IDs, names, values, and duplicate checks. - Preserve the exact master Branch value. Set an existing operation parameter to `evaluate`; otherwise use the fixed workflow's existing default. - Use the same checked parameter vector for job and plan writes and their readbacks. - Report distinct fixed bootstrap predicate codes without response values. - Document when the protected operation-parameter ID is required. - Align one existing Cloud-maintenance test with the inherited managed-Cloud S3 policy. Preserve the same-version local-storage proof in the parent, use valid synthetic Cloud settings in the child, and check the expected provider mismatch without S3 requests. Retain the no-seed and unchanged persisted-row assertions. ## Verification - URL controls: three failures on the parent; all pass with the correction. - Branch controls: two failures and one correct rejection on the predecessor; all pass with the correction. - `node --test tests/runner-e2e/chat-feature-testops-api.test.mjs`: 40 owning pure/mock controls pass on Node 26.4.0. - Two MJS syntax checks and `git diff --check` pass. TypeScript noEmit was not run for this MJS/documentation change. - The first [normal CI run](https://github.com/paperclipai/paperclip/actions/runs/41131094974) failed only in the managed-skill Cloud fixture and its dependent verification gate: 46 jobs passed, two failed, and one skipped. The tested merge included a new upstream policy that rejects the test's local-disk Cloud configuration. The exact caught runtime exception remains unknown. - Three pure controls using the actual policy validator pass: the old invalid Cloud fixture rejects, the new synthetic S3 fixture admits, and an invalid master key rejects. An initial tsx wrapper IPC permission error executed zero controls and remains a separate setup failure. - The PostgreSQL/maintenance-child test was not run locally. On new-head AWS CI, `managed-skill-content.test.ts` passed all 17 tests in server shard 9. - [Normal CI](https://github.com/paperclipai/paperclip/actions/runs/41133876636) is green on exact head `b76d5dd40d7711f1b5f6b6316b1114c0aae02d57`: 48 jobs passed and one skipped. Both required `ci / e2e` and `ci / verify` pass. - Greptile reviewed this exact head at 5/5 with no actionable findings or unresolved review threads. - No actual CLI, backend, model, provider, app, database, build, or bootstrap was run to validate this correction locally. ## Risks The exact original bootstrap predicate is still unknown. This fixes a demonstrated metadata false negative and preserves the prior failed run. Actual backend parameter types and integration dispatch still need verification. Unknown metadata stops before the once marker and writes. This PR changes no workflow, credential, case, model, grader, clock, or private-evidence policy. It does not claim a passing feature result. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, code editing, tool use, and pure test review. The runtime does not expose the exact context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
198218bf55 |
fix(config): require shared storage for managed cloud (#18350)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Managed Cloud instances must retain shared objects and secrets across server replacement. > - Implicit storage defaults can select local disk or an unintended bucket. > - Cloud runtime settings now come from a canonical database snapshot. > - This pull request validates bootstrap prerequisites before database setup and effective S3 settings after snapshot resolution. > - The benefit is an actionable refusal before local signing state or HTTP setup, while valid canonical Cloud settings work. ## Linked Issues or Issue Description **What happened?** A managed Cloud instance could start with local disk storage, an implicit bucket, or a local key file. After canonical Cloud configuration landed, checking only environment or file values would also reject a valid database bucket. Shared schema defaults could make an omitted raw bucket appear explicit. **Expected behavior** Managed Cloud requires external PostgreSQL, a stable environment master key, and effective S3 settings with a nonblank bucket. The canonical snapshot remains the configuration authority. Invalid configuration refuses startup before signing or HTTP effects. Self-hosted defaults remain compatible. **Steps to reproduce** 1. Select managed Cloud with its tenant token or managed configuration document. 2. Leave storage as local disk, or omit the S3 bucket from a raw canonical snapshot. 3. Start the server. Startup must refuse. An explicit valid bucket in the canonical snapshot must work without a file or environment bucket. **Paperclip version or commit** Current base: `4edd10804e2b42941aa2038e90c362c13a0b98dc`. **Deployment mode** Managed Cloud. Ordinary self-hosted defaults remain unchanged. Related PRs: #18335 owns canonical Cloud runtime configuration. #18327 owns S3 run logs. #18352 owns credential persistence. #18347 owns cutover checks. This PR adds configuration prerequisites to those existing boundaries. ## What Changed - Validate external PostgreSQL and a stable environment master key at the Cloud bootstrap boundary, before database setup. - Validate effective S3 settings after canonical snapshot publication and before Cloud signing, runtime consumers, or HTTP setup. - Require a nonblank bucket in the Cloud snapshot schema. Reject omitted or blank buckets in every legacy Cloud file before one-time import normalization, even when an environment bucket is supplied. Keep shared self-hosted schema defaults. - Preserve typed bucket provenance at the existing file-read boundary. Accept explicit canonical bucket names, including `paperclip`, and environment settings when the snapshot is null. - Validate effective legacy storage with the existing resolver before seeding. A rejected candidate leaves no canonical rows, so a corrected source file can be imported on retry. Environment overlays are checked but never persisted. - Add real PostgreSQL snapshot and migration-retry tests, startup refusal tests, and environment/file compatibility controls. Document that Cloud always requires its stable environment key, even with an external default secrets provider. ## Verification - Latest focused acceptance passed 199/199 in six files (54.29 seconds), using the command below plus `server/src/__tests__/cloud-configuration-migration-postgres.test.ts`. Real PostgreSQL proves rejected storage leaves all three migration authority rows absent, then a corrected legacy file imports on retry. An external-default-secrets control proves the deliberate unconditional Cloud master-key contract. - Before the peer correction, the frozen product diff passed 178 tests in five files: managed-cloud-prerequisites, cloud-runtime-config, server-startup-feedback-export, config-file, and cloud-configuration-readers. Command: `pnpm exec vitest run server/src/__tests__/{managed-cloud-prerequisites,cloud-runtime-config,server-startup-feedback-export,config-file,cloud-configuration-readers}.test.ts --maxWorkers=1 --no-file-parallelism`. Real PostgreSQL used a qualified POSIX test-only PG18.6 override because the host cannot supply the stock runtime's SysV semaphore allocation. - Coverage includes raw database and legacy omitted/blank bucket rejection, valid database and environment buckets, null snapshots, explicit default-named buckets, self-hosted compatibility, and refusal before local signing mutation. - The real AWS SDK configured-provider control uses a simulated loopback S3 service. It writes, reads from a fresh provider, and deletes an object. It also reads an encrypted credential with the stable environment key. This does not qualify production cloud permissions. - `git diff --check` passed. Final merges add unrelated master changes from #18241, #18363, and #18370; the focused storage product diff is unchanged. - Peer review reproduced a fresh legacy local-disk import that gained the default bucket before an S3 environment overlay. The importer now requires a raw explicit bucket for every legacy Cloud file. Six controls cover omission/blank values with and without an environment bucket; the full focused rerun passed 184/184 across all five files in 194.89 seconds. - Initial revised startup testing passed 167/168: one fixture omitted the database URL and failed before its intended key assertion. The corrected final run passed 178/178. An initial direct server typecheck failed because Runner declarations were not built; this is a setup failure, not a compile pass. Integration verification rebuilds dependencies before full checks. - Prior head `fae1e4aa` passed complete CI with 53 successes, two intended skips, and Greptile 5/5. `de71345f379cecb71dbfb219f35e14f942c10a3f` then passed complete CI (55 successes, three workflow skips, and a neutral external review), with Greptile 5/5 and zero unresolved threads. The external review identified the preseed rejection problem; its correction is committed at `75e04d0d77e19dcef0a8cb070b6b518f0f173b14`. The final merge head is `19b809ea2823a6a4e5ceb48ef2c2bac4721733d6`; [Fresh complete CI](https://github.com/paperclipai/paperclip/actions/runs/41127309762) passed on that head: 55 successful checks, three workflow skips, and one neutral external review. Greptile reviewed this exact head at 5/5 with no actionable issues. Both earlier external review threads are resolved with the tested fixes. Earlier infrastructure retries remain documented in [the first diagnosis](https://github.com/paperclipai/paperclip/pull/18350#issuecomment-6529964690) and [the runner shutdown evidence](https://github.com/paperclipai/paperclip/pull/18350#issuecomment-6530364769). Earlier missing SDK build and fixture metadata failures were repaired before those passes. - Current frozen-head CI first attempt failed one unchanged Antigravity authentication success case in Runner Vitest 1/2 (`outcome=failure`, 1994 tests passed). The same file passed 11/11 locally in 2.33 seconds, and identical Runner source passed the prior complete CI run. Attempt 2 retried only failed jobs (the Runner shard and dependent aggregate). Both passed without a source change. No second retry was used. The exact timing cause is not established. [Failed job](https://github.com/paperclipai/paperclip/actions/runs/41127309762/job/124309181778). - No production data, buckets, or live credentials were used. Complete local repository suites are not duplicated in this worktree; current-head required CI supplies the full suite gate. ## Risks - Managed deployments with local disk, omitted buckets, file-only keys, or missing external database URLs now refuse startup. Operators must supply valid prerequisites before rollout. - Required deployment environment overlays must remain stable across replacements. For example, a saved local-disk provider with an explicit bucket can use an environment S3 provider; removing that required overlay correctly refuses startup until the environment or canonical configuration is corrected. Environment settings are never silently copied into the canonical row. - Effective legacy storage must also pass before first seeding. A bad candidate does not create authority rows. Existing saved rows remain authoritative and can require environment overrides or a canonical configuration change if they are invalid. - A present legacy Cloud file needs an explicit raw bucket even if its saved provider is local disk or its environment supplies a bucket. Correct the file before one-time import, or use the existing absent-file/null-snapshot environment-only setup. Environment overlays are not persisted as operator choices. - A previously stored canonical nonblank `paperclip` bucket is authoritative. Prior normalization erased whether that value was originally explicit. This change cannot recover that provenance and does not reject existing nonblank canonical values. - Validation proves configuration shape, not cloud permissions, database reachability, or key compatibility with existing encrypted data. Keep the master key stable across replacements. - Cloud snapshot JSON parsing is stricter; omitted or blank raw S3 bucket values need correction. No database migration or provider operation changes are added. These prerequisites do not prove that every server persistence path is disposable. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`. Context window: unknown. The model used reasoning, repository tools, code execution, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |